Ai Dreams Forum

Member's Experiments & Projects => General Project Discussion => Topic started by: Freddy on September 05, 2014, 05:11:04 pm

Title: 3D Avatar Project
Post by: Freddy on September 05, 2014, 05:11:04 pm: Here's where I am at with my animated head. I've added blinking and eye movement now. The blinking is automatic and the eyes are currently controlled via a slider. Eventually I will work out some noise routine so that the eyes wander realistically.

I'm not completely happy with the hair. Getting the transparency right for that has been the bugbear so far. I mean the layering is not quite right, but I have one idea to try still.

This is a new voice too - 'Jess' from Cereproc.

Most of the recent work has been 'under the skin', quite literally. As I mentioned before somewhere I am now working with a lower poly model. This has meant I can do much more. There's also a bit of optimisation done to the mesh, so that only the parts that need animating are processed. If it doesn't move then I simply display it, but then forget about it.

Animated head - blinking and eye movement (http://www.youtube.com/watch?v=Q9feCLs-_7I#)
Title: Re: 3D Avatar Project - Animated Head
Post by: Data on September 05, 2014, 08:31:52 pm: She already sings better than some I've seen on xfactor ;D

Coming together nicely O0
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 05, 2014, 08:42:56 pm: Thanks mate :)

I find the bad singers on xfactor more entertaining than the ones who are supposed to be good >:D
Title: Re: 3D Avatar Project - Animated Head
Post by: squarebear on September 05, 2014, 10:58:35 pm: Nice work Freddy. That voice is amazing.
I would love to have something like that to represent Mitsuku instead of a flat 2d drawing.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 05, 2014, 11:47:15 pm: Thanks Steve. :)

I'd like to do some web stuff like this but at the moment I don't know how to approach it. I guess WebGL would be one way, but the voices...not sure what to do about that.

There was someone posting some 3D Haptek-like demos on the Knytetrypper forum, I keep meaning to invite them over here.
Title: Re: 3D Avatar Project - Animated Head
Post by: 0JL on September 06, 2014, 03:15:05 pm: Don't know about the voice-on-web issue, but yes, very nice work! (...and you have good taste)
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 06, 2014, 03:25:58 pm: Thank you :)

There is a web solution using Google Translate. But it's not just the voice, it's the phonemes I need too. I do have another option using this open source TTS voice I found called MaryTTS. I can run that on my Linux box. It think it does generate phonemes, but I have yet to look into it that much.
Title: Re: 3D Avatar Project - Animated Head
Post by: mendicott on September 07, 2014, 01:09:16 am: > http://www.quora.com/Are-there-any-software-tools-or-APIs-that-will-allow-you-to-create-a-3D-talking-head-that-will-generate-correct-lip-movements-when-given-a-text-stream (http://www.quora.com/Are-there-any-software-tools-or-APIs-that-will-allow-you-to-create-a-3D-talking-head-that-will-generate-correct-lip-movements-when-given-a-text-stream)

See above link for my recent Quora answer to "Are there any software tools or APIs that will allow you to create a 3D talking head that will generate correct lip movements when given a text stream?"

> http://stallios.com/ (http://stallios.com/)

Since that Quora post I've come across a new player in the space, Stallios Mayca, apparently targeting the high-end => https://www.youtube.com/user/MaycaVideos/ (https://www.youtube.com/user/MaycaVideos/)
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 07, 2014, 02:26:02 pm: Nice but she can't hold a candle to Guile3D's Denise as far as a beautiful, lifelike character. But then again, beauty is rather subjective.

The lip-sync issue has plagued many would be developers in this digital avatar industry but I feel that soon, one day, they'll get it right and it will will work nicely on all platforms. There...that should help brighten everyone's day! Or something like that.... ::)
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 07, 2014, 04:09:29 pm: Thanks Marcus, but I wasn't really looking for an end product - this is something I am doing for fun and to learn things. Which means I assemble or code it all by myself.

I agree with Art, that Stallios avatar is nice, but the lip-sync is a bit iffy in my humble opinion.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 08, 2014, 01:14:45 am: I know the reason for publishing a trial run is to get some useful feedback. It seems as if the audio track may have been rendered slightly behind the video track by a few tenths of a second. By looking very closely, the lips may continue to move a fraction of a second after the speech stops.
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 08, 2014, 10:05:46 am: Quote from: 8pla.net on September 08, 2014, 01:14:45 am
... By looking very closely, the lips may continue to move a fraction of a second after the speech stops.

Yeah...I know of some politicians who exhibit similar behavior! :2funny:
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 08, 2014, 01:26:36 pm: Thanks for the observation 8pla. I just tried talking and my lips might still be moving for that fraction of time ;)

But I know what you mean, I am at the mercy of the tools I play with.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 10, 2014, 10:55:20 pm: Goes without saying, your trail-run render looks fantastic! Good point! Perhaps fine tuning audio skew in a 3D avatar may be more an art than a science. Please share with me your reasoning. In other words, and in your opinion, why do the phoneme sounds (the voice) reach the listener before the physical lip postures are complete (mouth finishes moving) ? Maybe the speed of sound is a factor, or something?
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 10, 2014, 11:19:36 pm: I don't think it's the speed of sound really over such a short distance. It's probably because my PC can only process the graphical changes at that speed.

I use a decay setting, or in other words I cause it to slowly taper off a morph (lip shape). Perhaps I need to shorten that decay so that the mouth shapes finish more quickly.

Like you observed; it's a trial run, I ain't even remotely finished yet...
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 16, 2014, 03:16:51 am: Here is a (temporary) video of a test run I just did with my Lip Sync hobby
project now being written in PHP. Warning: This is unfinished work. ;)

https://www.youtube.com/v/E2BMywBxjbE&rel=0&info=0 (https://www.youtube.com/v/E2BMywBxjbE&rel=0&info=0)

It uses the GD and Image Functions to draw the avatar in PHP and the
Festival speech synthesizer.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 16, 2014, 10:39:18 am: Nice :) How many visemes are you using at the moment ?

I have a 2D project too here : http://aidreams.co.uk/forum/index.php?topic=5045 (http://aidreams.co.uk/forum/index.php?topic=5045)

That's all in Windows at the moment, but I have had thoughts about converting it to the web.

Festival is good, have you tried MaryTTS yet ? It's very nice too...

http://lnv-90208.sb.dfki.de/ (http://lnv-90208.sb.dfki.de/)
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 21, 2014, 12:56:03 am: Freddy asked, "Nice :) How many visemes are you using at the moment ?"

Oh, I don't even have a final viseme set yet to be honest. I am still trying to reduce the wobbliness in the visemes. So, that count goes up and down depending on which frames per second combination animates the smoothest.

Freddy said, "I have a 2D project too here : http://aidreams.co.uk/forum/index.php?topic=5045 (http://aidreams.co.uk/forum/index.php?topic=5045)"

That is a very cool 2D project you got there. May I suggest removing the meshes for the body, leaving everything from the neck up?

Freddy said, "That's all in Windows at the moment, but I have had thoughts about converting it to the web."

If you don't mind another suggestion, perhaps you may get started with testing on a localhost running on a Microsoft IIS webserver, and then make a demo video of it running to show its potential.

Freddy asked, "Festival is good, have you tried MaryTTS yet ? It's very nice too..."

Indeed, I have briefly. Yet, I like to have the project looking like a work in progress during early testing.

Here is an update to this work in progress: https://www.youtube.com/v/vEFk94LAjQ8?rel=0&info=0 (https://www.youtube.com/v/vEFk94LAjQ8?rel=0&info=0)
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 21, 2014, 02:22:12 am: No offense intended but all I can see are the lips pretty much constantly moving without any pauses between words.

Does or will it pause the lip movement if speech is slowed down or is that not an option at present?

Please keep us posted in this interesting approach.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 21, 2014, 04:01:15 pm: Please let me clarify that the binary black and white 2D design process is the beginning step. And the blue hair and red mouth are subsequent steps in 3D glasses support.

Art, no offense taken at all. On the contrary, thank you for actually helping me with your valuable feedback. Before we continue our discussion, please let me check with a moderator that 3D glasses are considered on topic for this thread.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 21, 2014, 09:07:35 pm: Quote
That is a very cool 2D project you got there. May I suggest removing the meshes for the body, leaving everything from the neck up?

Thanks :)

If I were to move this to a web based animation then I probably would do that yes. It's just when I am developing something at first I may take a few short cuts - like not removing the body in this case.

Quote
If you don't mind another suggestion, perhaps you may get started with testing on a localhost running on a Microsoft IIS webserver, and then make a demo video of it running to show its potential.

I don't mind suggestions. I'm more at home with Apache to be honest I use it a fair bit with web stuff. Recently I built a Linux box and put a server on it that will some time in the future be used for various things. One of them as my voice server (MaryTTS) as well as any animated characters I might demonstrate.

Quote
Here is an update to this work in progress...

She looks cute :) Did you find a way to time visemes firing in Festival ?

Quote
...please let me check with a moderator that 3D glasses are considered on topic for this thread.

That's fine by me. Art is a moderator as well. We are not too strict about it here. If something goes way off at a tangent we can just spilt the topic off.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 22, 2014, 01:40:11 am: Art asked, "Does or will it pause the lip movement if speech is slowed down or is that not an option at present?"

Freddy mentioned, "She looks cute :) Did you find a way to time visemes firing in Festival ?"

Bingo! The wobbliness in the lip movement is due to the frames per second (fps) being too fast. There may be a noticeable improvement here... https://www.youtube.com/v/jH80QJr3ddg?rel=0&info=0 (https://www.youtube.com/v/jH80QJr3ddg?rel=0&info=0) Yet, it is still unfinished, even for a trial run. Now, believe it or not, the optimizations I attempted on the visemes may no longer be needed, so I may have to undo them. So, the viseme count may be going back up again.

The reason I chose this funny poem for her to read was that it is supposed to contain every phoneme sound.
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 22, 2014, 10:01:01 am: Agreed, coming along with improvements. I had my 3D glasses on too but I really don't get much out of those (just mho).
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 24, 2014, 11:40:38 pm: Thanks, Art, for testing with your 3D glasses. I really appreciate that very much! I still have to order my pair of 3D (red/blue) glasses. So this time I had to guess the two different perspectives in this experiment. So, I am still doing that math now. However, the first part of the experiment was rendering the color palette, which I think was at least a partial success. The support of 3D glasses is still in very early stages of development.
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 25, 2014, 02:45:01 am: I also have a pair with clear lenses that operate on an electronic shutter principle where each eyeglass lens opens and closes alternating with Left, Right and so on at a rather fast, almost undetectable rate.
It was used on a variety of 3D projects, games, VR, etc., several years ago. Mine works on tiny batteries instead of being plugged in.

Nice concept but honestly, we live in a 3D world but aside from optical correction like myopia or astigmatism those 3D glasses are really nothing more than a curiosity for the general population instead of a really useful, everyday item. Just MHO and please, don't sway your project if my comments seem to be raining on your parade. I mean no harm, only a general observation from my lifetime. I just don't think people want to have to wear a pair of glasses to see 3D like they did in the theaters in the 1950's and 60's.

Then there's those wonderfully stylish and ever accessible, Apple iGlass(es). Go figure! Maybe Brin will come up with a different type of Google Goggles! (instead of the app). O0

Good luck with the rest of your project!
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 25, 2014, 03:54:36 pm: @Freddy:

I was curious as to whether you may be working with some type of alpha channel for a partial transparency on the hair for your model?

Also there appears to be an audio skew correction now. Perhaps it was You Tube doing maintenance or simply my Internet connection before?
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 25, 2014, 04:14:09 pm: The tool kit I was using (Helix 3D Toolkit for WPF (https://github.com/helix-toolkit)) has limited support for transparency. At the moment or at least in the last few months I was only able to use transparent PNG textures.

As you can see in the video it does not render that well where strands of hair overlap. Although the eyelashes are fine, I think mainly because there is nothing else in front of them.

I tried a few things, like depth sorting the polys but to be honest I was out of my own depth there and I could not get anything to work.

Apart from that Helix tool kit is a great bit of kit and is one of the easier ways into 3D on Windows and .NET that I have found. You can do a lot with it and it helped me learn more about how 3D works.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 25, 2014, 04:15:18 pm: Quote from: 8pla.net on September 25, 2014, 03:54:36 pm
Also there appears to be an audio skew correction now. Perhaps it was You Tube doing maintenance or simply my Internet connection before?

No idea sorry - I can say I have not changed anything though.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 25, 2014, 04:25:30 pm: @Art:

Well, I just ordered brand new (never used) 3D glasses from Ebay for 99 cents with free shipping from Hong Kong. They are plastic and look like Clark Kent Superman style glasses, except they have one red lens and one blue lens. If this experiment is successful, then I plan to make 3D glasses optional using a radio button on the website.

I am also researching the other 3D glasses options for the long term. From what I have read so far, it seems advanced 3D glasses, such as those you mentioned, may require specialized theater equipment.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 25, 2014, 04:37:32 pm: @Freddy,
I have to review the technical terms, but in simple terms, I remember there was a pair of inverted black and white monochrome images that are the exact opposite of each other used in this rendering. The point being that they may not be modified separately since they have to match black to white and vice versa. So if you modify one, then make a copy, invert it and regenerate the second one.

I am not sure if this applies to your workflow. I am just brainstorming. If this sounds familiar then I will research it in more detail. In any case, I like this idea, and I think the results may turn out to be efficient and realistic.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 25, 2014, 06:09:38 pm: Quote from: 8pla.net on September 25, 2014, 04:37:32 pm
@Freddy,
I have to review the technical terms, but in simple terms, I remember there was a pair of inverted black and white monochrome images that are the exact opposite of each other used in this rendering. The point being that they may not be modified separately since they have to match black to white and vice versa. So if you modify one, then make a copy, invert it and regenerate the second one.

I am not sure if this applies to your workflow. I am just brainstorming. If this sounds familiar then I will research it in more detail. In any case, I like this idea, and I think the results may turn out to be efficient and realistic.

What you are saying does light a bulb in my head - I think you mean alpha maps ?

I don't think Helix allows the use of that technique at the moment. I was using OBJ format models with their associated material file. Only some materials/texture features are supported at the moment.

I may leave Helix for now as I have tried the hair model in Unity (http://unity3d.com/) and it looks much better - more like the Daz Studio render and the transparency works fine.

I took a screen shot of her in the Unity editor, it's an improvement I think, also the skin tones come out better :)

EDIT: Actually that's a different hair model, but already you can see the transparency works well. This hair was the same one as I used in that static render you posted about today.
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 27, 2014, 12:10:05 am: Borrowing from Yoda,
Pretty she is!!
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 28, 2014, 04:48:57 am: UPDATE:

Here is an updated video. I've rolled back some of the optimizations
which did not apply to the solution I went with. So there are more
visemes for the time being. I tested, for the first time, the YouTube
option to make it a 3D video.

https://www.youtube.com/v/QTSmDhOKgw4&rel=0 (https://www.youtube.com/v/QTSmDhOKgw4&rel=0)

Do you think the optimizations should have been left in the video?
(Both versions of the video are still up there for comparison.)

Thanks for your feedback.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 28, 2014, 05:03:48 am: Close-up

EDIT OUT 1411911247: Originally, there was a transformation of a geometric figure here, which dilated the lengths of the sides by the same factor. However, this dilation was considered unpleasant (See next post). So, it was edited out.

Image is quoted from Freddy's post above. This is the same image (same filesize) enlarged using bbcode tags.
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on September 28, 2014, 12:58:19 pm: And the reason you just made me think something had gone awry with my eyesight?! :o

Just as in photography, the image gets blurry, fuzzy, distorted when zoomed or enlarged.
But you knew that?

Just wondering...
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 28, 2014, 02:38:29 pm: @Art:

The reasoning was to visualize the future of Freddy's avatar...

A life sized android. :idiot2:

Apologies! :'(
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on September 28, 2014, 03:45:24 pm: Sorry you lost me there...

But I think the added visemes are better, not sure.

Don't know if this is useful to you :

Visemes - for Character Animation (http://aidreams.co.uk/forum/index.php?page=Visemes_-_for_Character_Animation)
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on September 29, 2014, 01:41:17 am: Quote from: Freddy on September 28, 2014, 03:45:24 pm
But I think the added visemes are better, not sure.

This comment in particular was helpful. It led me to suspect an error was present.
Well, there was. I found a pair of visemes that were in there backwards.

https://www.youtube.com/v/TYpB_RVdv2s&rel=0 (https://www.youtube.com/v/TYpB_RVdv2s&rel=0)

So I swapped them and rendered a new animation with batting eyelashes.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 01, 2014, 02:36:29 pm: I think it's better, perhaps less blinking may be more effective though :)

I made some progress on my 3D model in Unity. I spent 2-3 days getting back into Blender and learnt how to rig a model for use in Unity. I was able to get some free motion capture animations and now she can walk 8)

It's very basic at the moment, this is just me in learning mode, but the effect is pretty good.

Simple walk animation (http://www.youtube.com/watch?v=EPAazysqaCI#)
Title: Re: 3D Avatar Project - Animated Head
Post by: Data on October 01, 2014, 04:15:53 pm: The motion capture animations work well 8)
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on October 01, 2014, 10:11:51 pm: Agreed, mocap appears smooth and fluid with a very lifelike body lean toward each side as each step is taken...nice.

One very minor pick. her feet slide to the left ever so slightly when she walks but this is not your fault as I have seen such effects much worse in many popular games and visualizations. Most of it has to do with the internal physics routines and not having collision detection within the engine.

Again, I wouldn't sweat it at this early stage in your development but my honest, unbiased, professional appraisal cost you absolutely nothing.
Hey! Free is good right?!

Then again, the Pilgrims thought so and look where that got us! ;) :2funny:
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on October 01, 2014, 10:43:40 pm: Can someone please explain to me
how to embed a video (http://www.youtube.com/watch?v=uKSJ1kYAOOY#)
how to embed a video here on this site?
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 02, 2014, 12:51:35 am: Thanks chaps :)

@Art, yes there is a bit of drift - these are freebies and not perfect - I have my eye on some more professional ones that look near perfect.

I can't decide if I should be able to walk her around or have her as someone to 'meet' if you get me. I have a first person controller that works well. So say she was in a house, you could go in the house and talk to her. I think I will do that as moving her around via the keyboard in third person may break any illusion of her being in control of herself. Dunno, not sure. :idiot2:

@8pla - it looks like you worked it out. You just paste the YouTube url from it's address bar. But for it to work you need to remover the 's' in https that YouTube uses.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on October 02, 2014, 02:11:08 am: @Art: I suspect it may be fast render settings, which are so convenient to work with. Only a few seconds to finish the rendering. Sometimes I may let high quality videos render overnight while I sleep, with a fan blowing on my computer to prevent it from overheating.
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on October 02, 2014, 09:51:06 am: And thus was born another great idea and business opportunity...

the wonderful, Render Farms. Where a group of computers are tasked with rendering a large project together, usually with each doing a piece or part of the final product / render. Especially useful for large scale animations, movies, etc.

I imagine it akin to SETI where thousands of individual personal computer "examine" packets of information thus equaling the computing power of several super computers working for quite a long time.

Just rambling...which old people often do.... ;D
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 02, 2014, 11:23:31 am: Guys, this is real time animation. There's no over night renders here, it's all done live.
Title: Re: 3D Avatar Project - Animated Head
Post by: Data on October 02, 2014, 01:03:47 pm: Quote from: Freddy on October 02, 2014, 11:23:31 am
Guys, this is real time animation. There's no over night renders here, it's all done live.

With your PC Freddy you might Haswell (http://en.wikipedia.org/wiki/Haswell_(microarchitecture)) ;D

Sorry, couldn't resist. :P
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 02, 2014, 01:13:38 pm: ;D It is performing well I have to say.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 02, 2014, 09:43:39 pm: Unity keeps amazing me. I bought a terrain generator and a few things to use in the landscape like trees, bushes, grass and plants etc. I'm going to give my AI character somewhere to live :)

I have been playing with this little world for a couple of days, here's what my world is looking like so far...

The media section reduced the quality and I compressed it a bit so it isn't huge. It's gives a good idea I hope.

(https://aidreams.co.uk/forum/MGalleryItem.php?id=407)
Title: Re: 3D Avatar Project - Animated Head
Post by: Data on October 02, 2014, 10:25:54 pm: That is gorgeous 8)

The lighting, textures, detail, all really nice indeed. O0
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 02, 2014, 10:39:09 pm: Thanks, cool isn't it. :)

The terrain generator cost me a bit, but I'm really enjoying playing with it. You can use loads of settings to make a terrain, you can control mountain height, textures, tree/plant/grass density and so on. There's loads of plants and stuff in the Unity store, so I bought a few of them which you can see in the screen shot.

A lot of fun :)
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on October 03, 2014, 02:36:09 am: Freddy,

Lovely scene!

Some time back Bryce X.X (maybe 7.X or so), was given away free and it's a great landscape program as is Terragen
and Vue. There was one other called Natural Scene Designer but I think it was a pay program.
Bryce used to have a very large following with a pretty accomplished Forum for help and sharing.
Terragen used to be quite capable and was free but can't say for sure now.

I use past tense as I am no longer very active in the 3D world as I once was.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on October 03, 2014, 01:01:14 pm: Here's an idea...

You could make your A.I. a god, with its face, the sun, shining over the earth.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 03, 2014, 01:29:54 pm: ;D I like it - reminds me of Teletubbies.

(https://aidreams.co.uk/forum/proxy.php?request=http%3A%2F%2Fi136.photobucket.com%2Falbums%2Fq186%2Fdarraghmc80%2Fteletubbies-sun.png&hash=83122e04271e7ea7ae1e0c5604a0a3925a9a173b)

But I think I would prefer her to be wandering around the landscape, or be able to move her around. Perhaps a combination of both.
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 03, 2014, 01:39:21 pm: @Art, yes I played with Bryce a few times a very long time ago.

It's still being offered by Daz3D for $20.00 : http://www.daz3d.com/bryce-7-pro/ (http://www.daz3d.com/bryce-7-pro/)

I did pick it up when it was free. I'm not sure if they do much work on it now, it seems unloved.

And I remember Terragen too, that was nice.

What I like about the one I bought for Unity is that it is integrated into the editor, so it's easy to use. Instead of the often troublesome ways of getting models to work between different programs.

On the avatar front I think I have worked out my work flow now. It took weeks of experimenting to get things playing nicely with each other. I wrote some tools myself to help me with all the importing and making models compatible.

Over the next few days I am going to try to get her talking again, this time in Unity. It might be simpler than what I have done already as things like morphs I can incorporate into the model of her. Well touch wood. I did get TTS to work in Unity so I am part way there.
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on October 05, 2014, 03:57:31 am: Blender Festival Lipsync (100% Linux)

Blender Festival lipsync (http://www.youtube.com/watch?v=kcNjHwHu8u0#)

This is an early lipsync test run of Blender 3D model to Festival text to speech engine. The visemes are custom for testing. So, they are not part of the original blender model, which is so well-done...

This Mechanical Face Rig was created by
Gord Goodwin / Feel-Good Comics
(http://gord-goodwin.blogspot.com/ (http://gord-goodwin.blogspot.com/))

Created for the Blender Artists Rigging Thread:
http://blenderartists.org/forum/showthread.php?t=158111 (http://blenderartists.org/forum/showthread.php?t=158111)
Title: Re: 3D Avatar Project - Animated Head
Post by: Data on October 06, 2014, 12:58:19 pm: Lets not forget Nvidia FaceWorks, capable of doing this in "real time" already.

Nvidia FaceWorks Human Head Demo 1080p (http://www.youtube.com/watch?v=zKr4W_IVRT0#ws)
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 06, 2014, 01:54:28 pm: @8pla, that's looking good. With a bit of tweening I think that will work well O0

@Data, I had forgotten that, thanks for the reminder 8)
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 09, 2014, 10:43:04 pm: As I mentioned I switched my project over to Unity as it does a much better job of the graphics.

I'm using my own MaryTTS server which I also mentioned somewhere. The server generates a WAV file and I also request a set of timings for the occurrence of phonemes and I use that to trigger the mouth shapes.

The voice is quite good. I'm keen to try making a custom voice at some point.

I think she animates quite smoothly, I probably need to tweak a few things. Aside from having to write my own timing system, using the morphs (different shapes) was pleasant to do. It's written in C# still but thanks to the way Unity uses models the amount of code has been reduced significantly.

The next stage will be to try the same thing but with a full figure character.

If you can't quite work out what she says :

Quote
Hello everyone at Dreams. My name is Jess and I have been created by Freddy. I hope to explore a wide world some day, but for now I only exist in a box. Well it was nice to meet you. So you all take care and I will see you soon.

Unity and MaryTTS Animated Head (http://www.youtube.com/watch?v=GvO-Jw-I8HE#)
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on October 10, 2014, 01:19:49 am: OK!
First, that was very nice Freddy! You're a man of many talents! She's lovely as you do have good taste!

Now for the critique, you knew it was coming....

At first I couldn't quite put my finger on it then after two more views, I got it. Not lip-sync or mouth movement not
expression at all. In fact those seemed fine.

The one thing that was bothering me until I figured it out was that the inside of her mouth is or appears to be lighted instead of somewhat dark like a real person's mouth when they talk.

Is there any way to tone down this light effect inside her mouth? If you can then I think you'll have a winner for sure!

It might be something related to the manner in which Unity processes the frames or something and I'll also bet you're aware of it and already digging about.

Keep plugging, my friend! You've certainly come a long way and I'm proud of your accomplishments over the years! Go on!! O0
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 10, 2014, 01:41:18 pm: Thanks for the kind words Art :)

I know what you mean - I did think her teeth looked a bit bright. I'm using two light sources in the scene and one of them is a spot light off to her left. It's possible this is too bright so I could adjust that. Also I could look at shadows, at this moment I am not sure if I turned them on.

So far it's really been mainly about the lip-sync. I think it's actually a bit better than using Windows SAPI - this might be due to the fact that MaryTTS recognises a lot more visemes than SAPI does, so there's more going on there than before.

Thanks for the input :)
Title: Re: 3D Avatar Project - Animated Head
Post by: 8pla.net on October 11, 2014, 03:53:42 am: Nice job Freddy!

Here is a low poly model I am working on in Blender...
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 12, 2014, 05:59:53 pm: Looking good. :)

Spent the last few days trying to get my character animated and into Unity. No end of problems, still some things that need to be fixed.

The main problem is I can get an animated figure in okay, but I'm having difficulty getting the model to work with the mouth shape animation.

I may have to do it so when you talk to her it does a cutaway scene with just her face as a compromise.

Anyway I have a few feelers out.

Instead I went back to the mocap and have her all kitted out and walking now, which worked relatively smoothly for a welcome change.

Walking (http://www.youtube.com/watch?v=qD1YV7t2E6k#)
Title: Re: 3D Avatar Project - Animated Head
Post by: Art on October 12, 2014, 06:49:11 pm: Very nice. She does walk like she's on a mission! Nice outfit BTW!!
Is collision detection available or even possible at this early stage?

Good work so far!! O0
Title: Re: 3D Avatar Project - Animated Head
Post by: Freddy on October 12, 2014, 09:08:23 pm: Thanks, this was an outfit I had for a long time which I used on some renders of Tyler. :)

Collision detection is possible yes - I've glanced at it so far - seems you can set up a sort of cage around things to detect collisions. That's for later though. First I need to figure out this morph mess I'm in :uglystupid2:
Title: Re: 3D Avatar Project
Post by: Freddy on October 17, 2014, 05:52:03 pm: Been busy on this. I've now moved on to full body characters. All week I was experimenting with things, many times hitting a brick wall. At times frustrating but at others great fun. I had to change my whole process, but I've now found a much more convenient way to get the models into Unity.

I bought a few more tools to help the process. A really useful one was a decimator that drastically reduces polygon count in models without too much loss of detail.

I had to do this because the models exported from Daz Studio once converted to tris added up to over 500,000 polygons :o

Even so my PC handled it surprisingly well and was animating the full figure smoothly. But that's a lot of polygons for just one character, so I've now got it down to about 60,000 polys. It's good that I found out that I can really push this machine to places I did not think it could go though.

So yep, here's a new character talking and animated. I haven't connected her to a chatbot yet - that's actually one of the easier things. I want to work on the character a bit more. Her chest area got a bit mangled but I think I can make that look smoother and more feminine.

Also I want to work on the voice. At one point it took a little time for my server to create the audio - which in turn made the animation stick. I'll have to investigate if I can do the audio stuff in the background so it does not affect the visuals.

Anyway stop rambling Freddy ;)

Full body real time talking animation (http://www.youtube.com/watch?v=qglAoS-foeI#)
Title: Re: 3D Avatar Project
Post by: claude2 on October 17, 2014, 07:02:11 pm: It's a nice lady! Quality Daz we recognize, in the finish of character.
Title: Re: 3D Avatar Project
Post by: claude2 on October 17, 2014, 07:25:48 pm: behold, the skin of Daz character. They are several format each type is different. You can added, makeup, ECCT.
Title: Re: 3D Avatar Project
Post by: claude2 on October 17, 2014, 08:07:37 pm: Here the code of the joint, the face, it can be accessed using 3DExchange. By iclone.
Title: Re: 3D Avatar Project
Post by: Data on October 17, 2014, 08:33:23 pm: Freddy, it's good to see your progress, still coming alone nicely I see 8)
Title: Re: 3D Avatar Project
Post by: Art on October 17, 2014, 10:35:01 pm: Quote from: Data on October 17, 2014, 08:33:23 pm
Freddy, it's good to see your progress, still coming alone nicely I see 8)

Right on Freddy! I second Data's observation and general consensus!! Nice work!! O0
Title: Re: 3D Avatar Project
Post by: Freddy on October 18, 2014, 12:45:52 pm: Thanks all :)
Title: Re: 3D Avatar Project
Post by: GT40 on October 18, 2014, 04:45:30 pm: As always, wonderful work in progress. O0 O0 O0
Title: Re: 3D Avatar Project
Post by: 8pla.net on October 18, 2014, 05:09:23 pm: Congratulations Freddy on reaching a milestone in your project! In my opinion, your results are neat and clean and aesthetically pleasing.

3D animation is a topic I enjoy discussing. I think animating skin tight outfits on 3D models, usually is not to achieve a trashy effect by the animator. Rather it is a shortcut, or compromise, exchanging the default nude model with "shrink wrapped" or "sprayed on" dressed model.

I have seen this technique enhanced further, by removing polygons, by remodeling the cleavage area on a 3D female model, or otherwise removing polygon detail with low polygon dress skirt or shorts built-in to the final model... Neat trick, I think, to customize a final 3D model.
Title: Re: 3D Avatar Project
Post by: Freddy on October 19, 2014, 02:11:08 pm: Thanks you two :)

8lpa, yes this really is a milestone for me, I didn't think I would be able to get to the stage of making a talking full body character that is also animated.

The choice of costume was decided by how cheap and how low poly it was - or could be made so. There are some costumes which are like spray on clothing, but there aren't many, just a couple I found. I have another outfit that I quite like and which can be reduced.

I couldn't use my favourite character as that's not available for the newer model I am using, but I found one I may pick up today.

I've got a few free or cheap buildings to use to make an environment. I think I am going for medieval period, kind of like in RPG games. Most chatbots are present day, so I thought I would go for something different.

I had thought about turning her into a desktop assistant, but I'm not convinced that desktop assistants make life easier. Sure I could ask her to play a song, but isn't it easier just to click on an icon ? Anyone have any thoughts about that ?
Title: Re: 3D Avatar Project
Post by: Art on October 19, 2014, 03:57:15 pm: Your remark about, "I'm going for medieval period..." made me think of a comedian named, Steven Wright. He has a wry, dry, deadpan yet hilarious sense of humor and delivery. He said, "I went to a restaurant that serves "Breakfast Anytime". So I ordered french toast during the Renaissance."
Title: Re: 3D Avatar Project
Post by: 8pla.net on October 19, 2014, 09:40:36 pm: Quote from: Freddy on October 19, 2014, 02:11:08 pm
Anyone have any thoughts about that ?

If you program Visual C++... You could build a First Person Shooter game.
Perhaps get a used book with the source code on a CD for cheap:

Programming Mutliplayer FPS Direct X, 1st Edition
ISBN: ISBN10: 1-58450-363-7

( But on second thought, you are already using Unity now. )

Another idea...

You might try some different backgrounds... Without adding geometry.
Title: Re: 3D Avatar Project
Post by: Freddy on October 20, 2014, 01:03:44 am: Ahh the summer breeze...the lively sounds...but who is this... ?

(Switch to 720p or click through to Youtube for better quality)

A walk in the woods (http://www.youtube.com/watch?v=reSdHa3sCyU#ws)
Title: Re: 3D Avatar Project
Post by: Art on October 20, 2014, 01:19:47 am: That was very cool Freddy! Yes, agreed, she was very fair of face, indeed!!

I almost got dizzy with those grasses waving and bending to and fro (as they would say). ;)

Other than a slight delay in the speech from typed to spoken, I liked it quite a lot and the 3D (maybe) world seemed very nicely done as well.
I thought I recognized some AIML coding in there as well which is fine but could stand with some behind the scenes tweaking and rewriting as
we all have come to know. Certainly not a shortcoming of yours by any means. "then you must know the password, etc...."

Is this more of Unity's magic? Looking really good!!
Title: Re: 3D Avatar Project
Post by: Freddy on October 20, 2014, 01:28:38 am: Thanks Art, yes this is Unity still. It does some neat things.

Yes this character has a pretty face, it's the new one I mentioned. :)

You are right of course, this is Jess connected to my Pandorabots account. The AIML in there is pretty much as it came. I've not done any actual writing of new AIML for her yet - hence the standard replies. I was just seeing how the chatbot aspect would go.

There are a few delays with the Pandorabots server. I'm still coding my own AIML interpreter and will run that on my Linux box. It should be a lot faster.

Really it's a proof of concept, but I admit I couldn't wait to 'talk' to her, that was cool ;D
Title: Re: 3D Avatar Project
Post by: 8pla.net on October 20, 2014, 03:44:38 am: Wow! Hope to see some more. That was fun! Wonder if adding a male voice synthesis (text to speech) to read the input may make it like a TV show? How about writing your own original simple chatbot engine in Unity using C# or JavaScript to follow a story script?

None of these questions are critiques or even suggestions. They are simply brainstorming. I do like it the way it is now. AIML is a great choice for interactive A.I. Maybe instead of fake Captain Kirk, you can use AIML to create a fake Xena Warrior Princess? Hey, that was cool to watch in full screen.
Title: Re: 3D Avatar Project
Post by: Freddy on October 20, 2014, 02:08:15 pm: Quote from: Art on October 19, 2014, 03:57:15 pm
"I went to a restaurant that serves "Breakfast Anytime". So I ordered french toast during the Renaissance."

I remember that joke, he's a really funny man. :)

@8pla,

I haven't programmed in C++ for years, I'm currently using C# which I really like.

Thanks for the Wow! I don't get those very often :)

There are male voices, the only thing is the added time to get the sound. I still cannot find a way to get the sound to load in the background and by doing so stop the jitters.

As for AIML, yes I think it will work well for this project. I have partly coded an AIML interpreter - I have so far made a working Graphmaster (http://www.alicebot.org/documentation/matching.html) in PHP as I was going to run it as a web service on my Linux box. As it stands it could hold simple conversations but I would like to implement the whole of AIML 1.0 as best I can. I think I may start that this week actually.

Quote
Hey, that was cool to watch in full screen.

Cool. It lost a lot of resolution from the way it appears on my screen, it's actually a lot more detailed live.
Title: Re: 3D Avatar Project
Post by: Data on October 20, 2014, 02:46:04 pm: I like the way she sways and moves around a bit, better than just standing there all stiff.

My little brain is trying to work out why the 3D engine hangs when waiting for the response. I know it's waiting for the PB server but why is that stopping the engine ? I'm clutching at straws but is it something to do with CPU affinity maybe.
Title: Re: 3D Avatar Project
Post by: Freddy on October 20, 2014, 02:51:12 pm: Nope it's not that. It's because it waits for web connections to end - I'm trying to find a way to do it asynchronously.
Title: Re: 3D Avatar Project
Post by: Data on October 20, 2014, 03:40:34 pm: OK, so multithread it and assign a core to the chat bit, that should leave the 3D engine to function.

Well something like that ;)
Title: Re: 3D Avatar Project
Post by: Freddy on October 20, 2014, 04:18:29 pm: That would be the nice way yes. But I can only call the Unity web services on the main thread. Which to my way of thinking is not optimal.

I think I have fixed the delay with waiting for audio to be sent from the server though. I just sent in a large audio request to my server and while waiting the GUI was uninterrupted.

As for the pandorabots bit I am still looking into it...
Title: Re: 3D Avatar Project
Post by: Art on October 20, 2014, 05:51:37 pm: Freddy,

Isn't the AIML standard currently at 2.0? Just curious since you mentioned 1.0. Type-o or a particular reason?
(don't say because 1 comes before 2) ::)

Cant weight two sea mor uv yor wurk! (Here bots...check that for grammar & usage). ;)
Title: Re: 3D Avatar Project
Post by: Freddy on October 20, 2014, 06:29:51 pm: Fixed the lag :supercontent:

There's a way to use coroutines that run asynchronously. Three things were causing the lag - the audio, the phoneme timings (used to animate the mouth) and the reply from Pandorabots. Now these run to the side and there's no more lag on screen.

Art, yes AIML 2.0 is the newest. I may go for that eventually -I haven't really looked at the new stuff - but for now I am working towards AIML 1.0
Title: Re: 3D Avatar Project
Post by: Data on October 20, 2014, 08:15:00 pm: Well done Freddy, knew you could do it :)
Title: Re: 3D Avatar Project
Post by: 8pla.net on October 23, 2014, 02:11:57 am: Blender lip sync dub stip (http://www.youtube.com/watch?v=3sJfBJIbpk8#ws)

Just having a little fun playing around with Blender, different colors, looping, etc.
Hey look: Resized this video to 640x360 and those black side bars are gone.

Music: Dub Zap
Artist: Gunnar Olsen
(Music provided by YouTube as a video enhancement)
Title: Re: 3D Avatar Project
Post by: Freddy on October 23, 2014, 06:31:11 pm: Quote from: 8pla.net on October 23, 2014, 02:11:57 am
Just having a little fun playing around with Blender, different colors, looping, etc.
Hey look: Resized this video to 640x360 and those black side bars are gone.

Kinda retro that 8)

I have been playing with the environment a little more. Added shadows and stuff. Getting the hang of the terrain generator slowly.

(https://aidreams.co.uk/forum/proxy.php?request=http%3A%2F%2Fi799.photobucket.com%2Falbums%2Fyy275%2FRogad%2Fmisc%2Fjes-in-the-woods_zps4428b19a.jpg&hash=6234cd1bfb97b1e3e31c86fc66ac010eea3ed096)
Title: Re: 3D Avatar Project
Post by: 8pla.net on November 06, 2014, 09:20:00 pm: Stopy Playing Me

Stop Playing Me (http://www.youtube.com/watch?v=KlZ0A_Mjx0c#ws)

All original low poly face model rendered in Linux using Blender. OpenGL is used for the lipsync animation. The intention is to try to display the vertices of this low poly face model. Espeak is used for the voice synthesis. Although it may be missing a few tweaks, it's a short clip which overall turned out to be a lot of fun to make.
Title: Re: 3D Avatar Project
Post by: Freddy on November 07, 2014, 02:04:54 pm: Looking good - do you use morph targets ?
Title: Re: 3D Avatar Project
Post by: 8pla.net on November 07, 2014, 10:41:06 pm: Yes Freddy...

In Blender, instead of the term Morph Targets... they use the term Shape Keys. I found this completely free (for personal or commercial use) low poly female model. So I added some polygons to the eyeballs, and hairstyle, and I colored in the rest of the mesh. For text to speech, I plan to go with the female voice of eSpeak, since it may be a better match, to lipsync a low poly model.

Now, we all know there are high poly 3D models out there, which look amazing, but they also tend to be a lot of file formats in order to render 3D eye candy... Playing around with low poly 3D modeling, it turns out, is like watching cartoons... It can be easier to do as a hobby, with just a single simple file. And well, I think low poly models are more funny!
Title: Re: 3D Avatar Project
Post by: Freddy on November 07, 2014, 10:57:37 pm: Yep shape-keys, I know what you mean as I was using them for a previous incarnation.

Low poly is fine - just look at Minecraft 8)

I've stepped away from the avatar for a time, I don't know what I'll do with it next. Essentially all the difficult stuff is done and I have a nice work flow now. I'll experiment with other animations, like waving and smiling etc - maybe put in some moods like Haptek used to do.

I'm working on my AIML interpreter again, it's going well. I plan to make some medieval buildings some time soon in Blender to use in my virtual world. Fun, fun fun.
Title: Re: 3D Avatar Project
Post by: 8pla.net on November 08, 2014, 01:17:52 am: Another cartoon style, exaggerating the eyes more. Bigger eyes, and also added eyebrows. Bottom Jaw is open too.

The long hair was edited with Gimp, (not 3D modeled with Blender) to see how a long hairstyle might look first, before adding it to the model.

8) Playing the slide show in the forum software is an interesting way to compare short hair to long hair.
Title: Re: 3D Avatar Project
Post by: Calhoone on March 18, 2015, 10:55:41 am: I really like what you are doing here Freddy!! It's some awesome stuff!
Title: Re: 3D Avatar Project
Post by: Freddy on March 18, 2015, 11:58:24 am: Thanks Calhoone :)

I will be getting back into this soon as they released Unity 5 recently, which I was waiting for.
Title: Re: 3D Avatar Project
Post by: Freddy on April 21, 2015, 02:11:40 pm: Last week I started making things for Jess' little world. I'm actually selling these too on the Unity Store. So just to show you I am still working on this, here's what I have named the Dragon Tavern which I made using Blender, and imported into Unity.

All modelled and textured by myself. I made the signs and benches and there's a few plants that I made in there too. The world I generated with a terrain generator.

(https://aidreams.co.uk/forum/MGalleryItem.php?id=413)
Title: Re: 3D Avatar Project
Post by: Snowman on April 21, 2015, 09:05:48 pm: It looks nice; you're making lots of excellent progess. Now all you need is a rather large 3D printer and you'll be in the Realty business in no time.
Title: Re: 3D Avatar Project
Post by: 8pla.net on April 21, 2015, 09:55:02 pm: Nice product. I particularly like the large glass windows. I think this design will be convenient for videos.
Title: Re: 3D Avatar Project
Post by: Data on April 22, 2015, 11:10:07 am: Excellent work Freddy 8)
Title: Re: 3D Avatar Project
Post by: ivan.moony on April 22, 2015, 12:33:45 pm: Blender is a pretty complicated stuff. I'm glad you've managed to domesticate that beast.
Title: Re: 3D Avatar Project
Post by: Freddy on April 22, 2015, 03:33:30 pm: Thanks everyone, I really enjoyed making it - after years of tinkering Blender is starting to click.

@Snowman, good to see you :) Funny you should mention 3D printing as I was thinking of getting Jess printed. Daz will be offering this service soon I read.

@8pla,net I was fussy about the windows, glad you approve. Videos would be cool, but ultimately people will be able to visit her in world. I'm still working on my new scripting language which I am going to build her with.

@Data, thanks again mate :)

@Ivan, The beast is doing what I want at the moment, helped by Youtube videos. I feel more confident with it now, it's very well designed and I'm even remembering keyboard short cuts that previously had been a mine field as there are so many.
Title: Re: 3D Avatar Project
Post by: Freddy on April 16, 2016, 04:29:17 pm: Wow, nearly a year since I last updated this.

I've been working with Unity off and on the past month or two. I upgraded to the new Daz generation figure and had to tweak a few things to get back to where I was as a few things changed in Unity.

My scripting language is coming along now and I might have a video this weekend. I'm trying to decide what environment I want my character Jess to live in. Currently she is residing in a new world I made. I may put her inside a building though, not sure.
Title: Re: 3D Avatar Project
Post by: Freddy on January 14, 2017, 05:25:24 pm: (https://aidreams.co.uk/forum/proxy.php?request=http%3A%2F%2Fi799.photobucket.com%2Falbums%2Fyy275%2FRogad%2Fmisc%2Fjess_zpsav4ce62h.png&hash=a26d666052b668389a0240fa205fc9f92d03f4c1)

Time for an update, been a while. For some time now I have been developing my ElfScript language and this clip shows some examples of what it can do.

The idea is to teach Jess through chatting to her as well as making a lot of scripts. Here she learns variables and new patterns as we go along. So she remembers things you tell her like what a car or bicycle is. There's a couple of tricks like remembering what she was talking about when you say "It" - making teaching easier and more natural.

She can follow up on things that were mentioned earlier in the conversation and can tell you when she already knows something and then asks to learn something else about it.

So this chat is more about showing the features than being a sensible chat - I have a lot more scripting to do. Using the dictionary that I built (with thanks to Infurl for the links) she can make judgements on how to use a word or phrase. And she builds simple relations between words and phrases in her memory - like how she learns a couple of ways to get around in the following video.

In this example I also ask her who David Bowie was and she finds that out online. There were a couple of times she used the same acknowledgement phrase so I will give her a bigger pool of phrases to choose from for that part.

I suppose the other thing is that I have improved the lighting, she doesn't look so washed out as in some of my older videos. I've also learnt which Daz models work well - a lot of it regarding the textures for her skin etc.

In this video I've also introduced random facial movement, raising eyebrows, puckering etc. I had also written some code to do blinking and eye movement, but found one for sale that was better than mine for a good price. She can also track and watch the user as I tried to show at the start of the video.

My ElfScript also allows me to pass commands to the Unity player - so that I can trigger say, raised eyebrows, when she says something in particular. You can see it in the video at stages.

As this is in Unity, I have also been able to meet up with her in virtual reality, which is strange but cool ! This video is just about typed input, but I had some limited success with voice recognition, so you can actually talk to her and she will respond. I've used Google Voice Recognition API for that - but either it's got some work to be done, or my accent is confusing it, as it's all a bit hit and miss. I'm sure they will improve that, it is only in Beta at the moment.

I recorded the video at 1440p, but my movie maker only goes as far as 720p or something. It's not too bad though. Well that's me done with my rambling...over to Jess. Oh and you will see she knows her name ;)

https://www.youtube.com/watch?v=aQycNqCuwGw (https://www.youtube.com/watch?v=aQycNqCuwGw)

Transcript :

Quote
Freddy: hello jenny

Jess: My name is Jess actually.

Freddy: hello Jess

Jess: Hi there !

Freddy: got a favourite song

Jess: Secrets I think.

Freddy: who sings that

Jess: The Pierces.

Freddy: i like it too

Jess: Cat is so amusing in the video.

Freddy: a car is a good way to get around

Jess: I always like to learn new things.

Freddy: a bicycle is a good way to get around

Jess: Okay thanks.

Freddy: it is good for keeping you fit

Jess: Okay thanks.

Freddy: it is pretty cheap to buy

Jess: Okay thanks.

Freddy: it is pretty cheap to buy

Jess: Okay it is, I already knew it's that. What colour is it ?

Freddy: red

Jess: I didn't know that was it's colour.

Freddy: what is a bicycle

Jess: A bicycle is good way to get around and pretty cheap to buy.

Freddy: name a good way to get around

Jess: Okay, I know car and bicycle.

Freddy: who was David Bowie

Jess: David Bowie was a British musician, actor, record producer and arranger.

Freddy: who do you read

Jess: Dan Brown is my favourite authour.

Freddy: bye

Jess: I have nothing to say to you.

Freddy: cya

Jess: Cheerio.
Title: Re: 3D Avatar Project
Post by: 8pla.net on January 14, 2017, 06:19:05 pm: I have a few suggestions.

On a new experimental interface, I suggest:

1. Moving the textbox up across a (daylight) sky,
2. Increase the length of the textbox to handle a bigger font
3. Add a male text to speech to narrate the textbox input.

Backup the current interface. It is excellent, especially
for a stand alone application. My suggestions are
video related modifications. I think they may make it
look like a television commercial which would be fun.
Title: Re: 3D Avatar Project
Post by: Freddy on January 14, 2017, 06:35:05 pm: Thanks :)

When I started out early this morning I had the intention of doing a recording of me in VR talking to her through voice recognition, so I was hoping to get my voice in there to show you guys. But after trying various more complicated sentences it was clear it's just not going to work yet.

It could cope with short phrases like "hello there" and "what are you doing" etc. but if I said something like "a fish is an animal" I would get it interpreted as "a phish is an annum" or some kind of nonsense like that. And even with my fuzzy pattern matching that was creating problems ! So I reverted back to typing for now. Maybe I will have to get elocution lessons. ;D

1) The text might be better high up, it's just more difficult to read on my screen like that - where it is now is a comfortable eye-line.

2) Are you on a laptop ? The text box is really quite large on my screen. Try viewing the video full screen to get some idea.

3) Could be a plan for future videos yes.

The interface I kept quite basic, the environment is the default one in Unity. As you've seen maybe, I have had her in a forest in the past. I was playing with some indoor interiors, but for this demo I decided to keep it simple and focus on the new features/improvements of the character. I think I might try making some scenes for her to appear in some time.

The television commercial is a fun idea :)
Title: Re: 3D Avatar Project
Post by: Data on January 14, 2017, 07:47:24 pm: The AI learning is neat Freddy, she grasped it first time and it feels quite natural as well. You did a great job on the characters appearance and animations too.

What else could I say, it's good :)
Title: Re: 3D Avatar Project
Post by: Art on January 14, 2017, 08:42:16 pm: Very good, Freddy.

Actually in Full Screen the Text box is split by the red 'progress bar' at the bottom whereas just viewing as a smaller window on the site, was not shown in than manner.
I prefer to read text along the bottom or under the character rather than above one. Perhaps after so many years of most bots having their output windows beneath them it could just be one of those adopted preferences. Making the Output box have a whitish background and a black text and Jess's window just above it (if you were able to size it), might be a worthwhile combination.

Of note, I did like her facial animations, the eyebrows raising, and movements, plus her other gestures were also nicely done.

I too was waiting to hear your voice, but one problem might be that Google's Speech was fitted for that other English across the pond...maybe Amerish or similar. Try it without your accent...or is that even possible? Hmm... ??? ;)
Title: Re: 3D Avatar Project
Post by: Freddy on January 14, 2017, 10:16:04 pm: Quote from: Data on January 14, 2017, 07:47:24 pm
The AI learning is neat Freddy, she grasped it first time and it feels quite natural as well. You did a great job on the characters appearance and animations too.

What else could I say, it's good :)

Thanks my friend. I did a lot of work on the learning side and there's different approaches that you can use depending on the situation you might find yourself in. I'm glad it seemed fairly natural. Jess' responses look better than mine as I made some sentence tidying routines for her. Being human I don't benefit from that ;)

I'm really pleased you like her appearance. A few things just clicked into place over the past couple of days, one of them was my tinkering with the environment light and the other was getting the eyes to look better - that was just a fluke as I flicked through various render settings.
Title: Re: 3D Avatar Project
Post by: Freddy on January 14, 2017, 10:23:23 pm: Quote from: Art on January 14, 2017, 08:42:16 pm
Actually in Full Screen the Text box is split by the red 'progress bar' at the bottom whereas just viewing as a smaller window on the site, was not shown in than manner.
I prefer to read text along the bottom or under the character rather than above one. Perhaps after so many years of most bots having their output windows beneath them it could just be one of those adopted preferences. Making the Output box have a whitish background and a black text and Jess's window just above it (if you were able to size it), might be a worthwhile combination.

Thanks Art. I see what you mean about the split - I'll nudge it up in future. I'll try black on white too. You can resize the player when you run it, but it's not laid out like a regular Windows form type affair.

Quote
Of note, I did like her facial animations, the eyebrows raising, and movements, plus her other gestures were also nicely done.

Makes her seem more alive doesn't it. There is a host of expressions to choose from as well, I just touched on a few.

Quote
I too was waiting to hear your voice, but one problem might be that Google's Speech was fitted for that other English across the pond...maybe Amerish or similar. Try it without your accent...or is that even possible? Hmm...

Well I set the locale to GB English, so I would have thought it would take care of it. It was really early in the morning and I wasn't at my peak. I'll play with it some more though. There is another option using Windows speech recognition - which I think is what powers Cortana - but unfortunately Unity broke that in some update, so people are waiting for them to fix it.
Title: Re: 3D Avatar Project
Post by: 8pla.net on January 15, 2017, 05:55:05 am: Yes, you are right, the full screen does show the font size larger.

In Unity, can you create a fraction of a second delay on the audio track?
I may have noticed slight lip movement after the audio is finished. I think
slightly sliding the audio track to the right of the video track on the timeline
may sync it up perfectly.

Sometimes, I add a fraction of a second to the beginning of the audio track.
So, I'll just offer this tiny tweak as feedback. My suggestions on this thread
are elective, not constructive criticism in this case. This is high quality work.
Title: Re: 3D Avatar Project
Post by: Freddy on January 15, 2017, 05:12:37 pm: Yes I noticed that too, at the end of some sentences. I think this is due to upgrading my graphics card as the decay on the lip movement morphs is tied to the fps.

It's not an audio track as such so I can't slide it. This version is using SAPI and it fires the phoneme events in realtime.

What I will do is try to limit video to 60 fps and that might fix it. Currently I think it is running fps quicker than needed and is messing with the phoneme decay. Or in my adjustments when moving from MaryTTS I made an error on one of the phonemes.

Or just move it to a fixed rated timer.

Well picked up :)