Jump to content
Sign in to follow this  
Chris Millar

Real-time motion-capture cameras - is there a standard?

Recommended Posts

Hello all,


Motion capture - i.e. actor moves and both their pose and facial expression are captured for use in pre-vis and animation - but I'm talking about the real-time variety, as in it's all rigged in software to happen instantaneously/on-line (or in reality at least within a processing delay that renders the output 'acceptable').


Have a look here for a nice example:





> What cameras are used? (I wonder if it was the same one used to capture the imagery in the above example?)

> What frame rates, delay, resolutions, colour space? (these probably trade-off against each other huh?)

> What communication/connections to the PC?

> Any particular model that has become a de-facto standard?

> Off-line - what are the standards? With data and processing speeds increasing, is there a major distinction nowadays anyway?


Thanks for any input :)

Share this post

Link to post
Share on other sites

She moves slowly and the frame rate can be slow. If the green lines are what are used to track her face they can be projected onto her face every other frame.

The closeups during 1:08 - 1:15 shows how well the computed net follows her face. The technology is fast, but kinda crude.



Share this post

Link to post
Share on other sites

I suspect the tracking is via the dots, which look retroreflective, in which case they may well be using infra-red cameras as are used in common motion tracking scenarios.


These are at least sometimes just normal cameras. Types intended for security applications tend to lack infra-red cut filtering, and may be retrofitted with infra-red pass filtering to yeild an image containing almost nothing but the tracking markers. All of the face tracking for Avatar was simply recorded as video, with only the circle takes being processed into 3D representations ("solving" the face is computationally intensive).



Share this post

Link to post
Share on other sites

Phil Rhodes: "...the tracking is via the dots, which look retroreflective ..."

Fascinating! I didn't know of this technology. Is it appropriate for faces?

Two fixed video cameras, each with its nearby lightsource, can be shooting a retroreflective dot while the 3D location of that dot is computed by triangulation. The dot is made retroreflective so a camera with nearby lightsource sees it as distinctly bright. The retroreflective dot can be a tiny sphere of very high refractive index glass, so it retroreflects independently of angle.

With retroreflective dots and separate tracking cameras I don't see what's gained by their working in the infrared. They can use a weak white light that doesn't affect the principal photography.

The sample video doesn't show so much apparatus, so it is probably doing something simpler, and cruder, than full triangulation.

My main question is about placing retroreflective dots on a face if the goal is to include photographs of the normal face. There are about 20 retroreflective dots on the face. Sometimes they appear bright, when there is a light source near the camera, and other times they appear dark like freckles. But they always appear. Any way you build a retroreflector it has its diffuse reflectance too, and it won't perfectly match skin as the face moves around. Can the appearances of the dots on the face be eliminated by image processing?

I wonder if an infrared dot method couldn't do better than a retroreflective dot method. The infrared dot is a spot of color that the infrared cameras can see but the principal camera can't. For example, suppose we had a dye that works as a short-pass filter transmitting fully at wavelengths 400nm-650nm while absorbing fully wavelengths past 750nm. A spot of that dye in transparent carrier on the skin will be black to the infrared camera while the principal camera will see the skin as it is.

Share this post

Link to post
Share on other sites

You can see the retroreflective dots (scotchlite paint?) in action throughout the video - you'll note that they are bright in the frontal view but dull in the side shot. Not sure why they are using them yet, as markerless tracking has been a viable option for a while now, markers make the system much more robust, but then again, you've got to both apply them, and then remove them - or leave them be as in this case as they don't detract from the aesthetic. I think the green lines are added, if not even a red-herring for the geeks out there? (keen to learn however if I'm incorrect in that assumption).


From a bit of web hunting and looking at the first few shots from the video I've found this:






As it says, plug'and'play ...


6DoF and IR or not as you choose.


A bit out of my price range for the intents of my project, however it is likely I'll opt for two of the sensors that are used in the product:






It's my bag that I have to develop the backend and GUI for the application :D


Lot's of jargon - 'Harr-features/Eigenfaces/AAM/POSIT/etc.' - but all doable, especially with libraries like openCV available. Reading the original papers from the 90's is interesting with them talking about 1fps rates, looks like it's more around 100fps now, although I imagine careful consideration of the algorithmic complexity would be required to avoid a brute force approach.


I have control over the setting and it can arguably be less general and be optimised for specific cases - i.e. certain faces - which might afford a kind of offline pre-processing of sorts, might speed things up...


Developing on my webcam at the moment is crippling once I extend the features to look at beyond face and eyes, e.g. add 'pose' (pitch roll and yaw) - let alone eye and mouth movement... Work in progress however!


Although the video is great, it was just one example of many and I meant the question in a more general sense - so ok, Avatar was just standard video, but was it real time at any stage, maybe for visualization in video village?

Share this post

Link to post
Share on other sites

Not sure why they are using them yet


Oops! duh, it's kind of obvious - you cant track a face without markers if the images you are projecting on it make it look like anything but a face.


I guess as you know what you're projecting, you could do something with differences - but the bit-depth of the resulting 'image' would be hard to work with.

Edited by Chris Millar

Share this post

Link to post
Share on other sites

Although it seems I'm talking to myself ...


I have the camera here now - it can run 120fps out of the box, but the algorithmic load of markerless tracking that I've managed to optimise so far, means the system is realistically running at around 76fps. I just drop frames and use the latest available in the instance that I can't use a frame before the latest arrives.


I get the feeling there is maybe one thing I'm not noticing that might sneak the process up to 100fps - but that's just for personal satisfaction, it's running quite well otherwise.


All good :)

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

  • Gamma Ray Digital Inc

    Paralinx LLC

    Visual Products

    Rig Wheels Passport

    Metropolis Post

    Ritter Battery

    Media Blackout - Custom Cables and AKS


    Abel Cine

    Broadcast Solutions Inc


    Serious Gear

    New Pro Video - New and Used Equipment

    Wooden Camera

    Tai Audio

  • Create New...