Ai Dreams Forum
Member's Experiments & Projects => General Project Discussion => Topic started by: Freddy on February 06, 2013, 05:48:05 pm
-
For the past few days I have been working on an avatar system for my projects. So far I have done all this in vb.net.
For a long time I have wanted to get the visual quality of Daz/Poser renders into a real time animation. This project is based on animation in real time, but the frames are pre-rendered. In a way it's a little like the Verbots characters and the system Bragi uses for his project. It was inspired by both.
So this little animation is really my prototype. I wrote a few scripts to get Daz to render the required frames and model positions to save a lot of tiresome work. I haven't yet used high quality renders as I want to quickly get the basic idea working. So this animation is the basic Victoria 4 model without any enhancements.
The frame count is a modest 81 frames. The idea here for me at first is to work on the 'idling' animation. So that in real time she just moves about a bit, rather than a fixed frame looking out at you - instead her head should move about a little.
Enough rambling, remember, it's early days and it could be a lot smoother with more frames, but what do folks think so far ?
Animated Head Idle (http://www.youtube.com/watch?v=7un4aRFf2SA#)
-
BTW, the project name 'Flixz' comes from the word 'Flicks' as in the word for movies, but more in thought of when we were kids and drew little animations in the corners of books and flicked through them. ;D
-
Very cool. Perhaps when it's working the way you like, you can integrate your library with my app?
-
Thanks. :) I'm still working on the idle algorithm to try and improve it, but it's interesting work.
And yes, I will eventually turn it into a dll/COM so it can be used in other projects.
I need to work on direct control next - moving from one place to another. So for example a command to turn right, moves her head to the right following the frame path.
And I also need to do visemes. Rather than render every single viseme for every position (81 frames x 20 visemes = 1620 extra frames) - I think I will just have her talk in the forward facing position. I think that is more natural too as people tend to look at you when they talk. We will see.
-
Only just spotted this.
Looking good, nicely done Freddy O0
-
Rather than render every single viseme for every position (81 frames x 20 visemes = 1620 extra frames) - I think I will just have her talk in the forward facing position.
ye that's one of the problems you get with this technique. Of course, movies are also stored and played as partial images, so it should be doable. Keeping all the frames in memory is probably not a good idea. You could group the viseme images per head position, and only load the ones that are currently being used. You'll probably best store all the images in a single file instead of separate images, cause searching for a file is a big cost, once it is open, reading a block can be pretty fast.
Also: crop all the images so that they are as small as possible. Saves lots of CPU time and disk space.
-
Yeah it's tricky. So far I believe I have managed to store the 81 frames within memory - they are very small in size. The impact on available memory is minimal. Starting my program only takes up about 2-3% more memory. Obviously it would be nice to have bigger images too.
At the moment I am trying to find ways to reduce CPU use, but as it is, it's not too bad. With everything running so far I max out at about 30% CPU usage, and that includes the IDE of course. All 81 images add up to only 3MB.
I am building in ways to minimize CPU use though - I already have a frames per second adjustment and a way to adjust how frequently the frames change. So it should be possible to find a sweet spot. Mostly I am working on 30fps at the moment and it seems okay.
This it never going to be a smooth and flexible as real-time 3d but I will get it as close as I can.
-
And thanks Data :)
-
This it never going to be a smooth and flexible as real-time 3d but I will get it as close as I can.
yep. That's true.
-
After a lot of fiddling I got a bit further, I am now using WPF which gives me DirectX, which results in less CPU use. Thanks to Bragi for suggesting that :)
It's a small video, watch it on the higher resolution to see her better.
Now she is on the desktop. I also invested in the new Genesis figure Victoria 5 from Daz, so the model is a lot nicer and with a much better skin.
The button is there so I can simulate a command(s). For this demo she simply looks down and to the right , pauses and then moves to the top left, pause and then goes back to idling.
I still need to work on the eye movement. I think I need to do some tweaks with the new models. Not sure what to do with that aspect at the moment to be honest.
On to visemes for her next I think.
Desktop Avatar #2 (http://www.youtube.com/watch?v=BdvTO6xs4LY#)
-
There was a sign of disapproval when you removed the "new folder", a shake of the head, either she likes new folders or thinks you a little crazzy, or am I reading to much into this :)
Cool Freddy 8)
-
Haha, well that's just coincidental unless its some kind of bizarre emergent thing ;)
Other than the two times I pressed the button, she pretty much looks anywhere she likes, within a defined area. The button pushes her outside of that area so she can do more extreme movements. I found after a while working on this, that you don't need a lot of idle head movement, but just enough to not be static. My early trials had her moving her head a lot and she looked a bit mad, so I reduced that a bit.
Thanks for taking a look :)
-
I really liked that Freddy! Really cute character and a LOT more lifelike and believable than some others out there!
Good work so far!! O0
-
Great work there Freddy, She looks fabulous. Keep up the great work :)
-
yep, starting to come along nicely.
-
Thsi looks great Freddy. I've always wanted something like this for Mitsuku rather than a plain 2d image.
-
Thanks everyone :)
Well today I got a lot more done, now she can talk :)
So what we have here now is a slightly larger avatar (350x350 pixels) hooked up to an AIML bot. CPU use is really low still.
Again, swap to high resolution or click through to see her larger on You Tube.
Desktop Avatar #3 (http://www.youtube.com/watch?v=vIXKT7aJFA8#)
A couple of glitches like pronouncing a full-stop (period) and 'Ai Dreams' not quite right - I probably should have separated the 'A' and 'I' but you get the idea.
She's not called Tyler, that's just the AIML I had handy.
-
Very nice, Freddy. She's getting better with each posting.
Over here we don't say or use "Full-Stop". Never heard of it until a friend from Pakistan used it a few times last year.
For Internet / Web usage it has evolved to the word "Dot" whereas in English classes here it is simply referred to as a Period.
And that concludes today's presentation of the Full-Dot-Stop-Period-End of sentence punctuation. or should that have been a "!" ?
(curious, is there such a thing as a "Half-Stop" or is that the same as a comma , or something not in the same context at all.)
curiouser...and curiouser.... ;)
-
Agreed Art, Definitely getting better with each posting :)
Marvelous work Freddy 2x [thumbs up]
-
Thanks guys. What did you think of the way she looks towards the user when she talks ?
The way I wrote it is that she first moves so she is looking forward and then talks, then goes back to being idle.
It would be possible to define a region where mouth shapes can be shown, so that she can talk in a slightly 'off centre' position. But I don't know if that's really necessary. It requires a lot more frames.
What do people think of that, is it necessary ?
Today I need to make the avatar scale-able some how and take a look at how I might turn this into a library so it can be used in other projects. I already have some things done for an assistant, but I think it would be nice to make this modular.
-
the avatar scale-able some how
Take a look at WPF's RenderTransfor and LayoutTransform. One of those should do the trick pretty easy.
-
Thanks for the tip, I managed to get scaling working, using a slider. Probably the same way you did. O0
-
Yep.
-
Freddy,
After re-watching the video again, I picked up on the fact that "Tyler" never blinked once during the entire video. Blinking is an unconscious human occurrence and IMHO would certainly help add to the overall effect. Some occasional blinking from time to time (perhaps set randomly) would be nice. If her eyebrows could also slightly raise during her speech (again randomly) it would likewise help the overall illusion.
Just some thoughts, not bashing. I really like the direction she seems to be taking (or maybe that YOU are taking). ;) O0
-
Thanks guys. What did you think of the way she looks towards the user when she talks ?
I like it and would leave it as is, My reason is: when some is talking to you, you would expect them to look you in the eye.
Random Blinking, and Random Brow Lift feature would be an improvement. Is it possible instead of random brow lift to be controlled by certain key words instead? you could then use different heights and variations for assigned words.
-
Art, yes blinking is something I plan to introduce and is pretty straight forward, I just haven't got around to it yet.
The eyebrow raising could be done in a similar way, at least when she is idle.
I just had an idea about how to introduce some random face movements too.
I did think about raising her eyebrows when she said something with an AY sound in it like 'play' or 'hey' - as I think we tend to do that when we make that sound, not sure, might just be me.
Syber : I like it and would leave it as is, My reason is: when some is talking to you, you would expect them to look you in the eye.
Yes that was my original thought too. 8)
Thanks chaps I will play some more with it. The good thing is the CPU use is so low now that it uses DirectX that I can add more things :)
-
Here you go Art, here is some blinking. I forgot to add the eyebrow thing to some of the animations so that will have to come later.
New hair do as well, reminds me of Leeloo in The Fifth Element a little. A different voice this time too.
Desktop Avatar #3 (http://www.youtube.com/watch?v=MHBxeYnUMb0#)
-
Very good Freddy!! Nice touch with the added random Blinking.
Here's another suggestion for you...well...for her. ;)
Yes, that's it...a smile! How about doing a graphic that would allow her to show
lips together and turned up at the corners as in a closed mouth smile?!
This would of course accompany her telling something amusing and joke or witty
remark! Kind of the sparkly eye, cute smile, looking at you as if to say, "I got you!"
I liked the short hair and the long hair. She still displays a cute face! ^-^
Overall, You're doing some really nice work, my friend!! O0
(don't forget the eyebrows...).
-
Thanks Art, yes the smile is on the list too, I just need to work out the best way for these animations to all overlap. Thanks for taking a look :)
-
You're doing some nice things with her Freddy. I also like the Ivona "Emma" voice you've chosen for her! O0
-
Thanks, actually that last voice was an old one I have from NextUp (TextAloud) - called Karen. She is Australian.
The other video, with the red hair was with the Emma voice though.
I will have to see if Ivona have any new voices.
-
Freddy,
Seems to be coming along Great. keep it up. still with Haptek naturally and your way ahead again.
Carl2
-
Thanks Carl, sorry for the late response, have been in hospital, all sorted now (touch wood). Looking forward to working on this some more.
I think it will have to be her smile next. :)
-
Animation is coming along great, I like the new hair do as well.
Seeing Trinity uses Emma voice, I liked it better than the former voice of Emma.
Note: Ivona has a couple new voices since I bought my Emma voice almost a year ago, They also upgraded a few of there older voices as well.
-
Thanks Syber.
I've had Emma for a while now too, I'll have to see if there is an update. I like this Scottish voice here :
http://www.cereproc.com/en/node/229 (http://www.cereproc.com/en/node/229)
I emailed Loquendo/Nuance about buying their voices but they never wrote back.
http://www.loquendo.com/en/demo-center/interactive-tts-demo/ (http://www.loquendo.com/en/demo-center/interactive-tts-demo/)
If anyone knows where to buy these please let me know.
-
I have and use the Heather voice for my Hal's character.
I agree, those Loquendo voices would add so much more realism to a "talking head" that one could soon forget they were artificial.
Wonder why they don't respond?
-
Hi Freddy
I have probably the same target as you, and I'm gathering information on how to render DAZ 3D Genesis avatars in my own Windows window.
I already have an avatar, not DAZ nor sophisticated, a bit realistic, running and talking in a WPF window since mid 2012.
But now I need to replace it with something like Victoria 5 with hair and cloths. I already bought an extensive bundle on Victoria5, didn't have much time to try to get it rendered in WPF yet.
Apparently you managed to get Victoria5 rendered live into WPF, congratulations.
Can you tell me some of the steps you took, starting from a Victoria 5 bundle, like file conversions with which tool x?
Until now, I used my own interpolation of viseme vertices to have my primitive avatar speaking.
It would be much better to manipulate the avatar at a much higher level of course.
Everything needs to be realtime in my case. I also want to dynamically modify rendering resolution, depending for instance on the actual workload of an animation server or on an internet video transmission bottleneck.
If you don't like to publish your information, we can privately mail, and see if we can help each other.
Filip
-
Hi welcome to the site :)
I don't know if it's exactly what you think.. I'm not displaying a 3D head as such...I'm displaying a pre-rendered head. This has a lot of limitations compared to what you can do with true 3D of course, but I have tried real time 3D with these hires models and not got very far. Unity, Leaderworks and many more have all failed (for me) to render the complex models in their entirety - though it's been a while so maybe things have moved on.
I'll send you a PM with some things I discussed with Art a member here and how I hopefully explained what I was doing. I'm not sure where I want to go with this at the moment, I have other things occupying me. I was thinking of doing a series of tutorials, it's all a matter of time.
Anyway, check your messages here in a bit :)