Contact address logo Eisko, 24 rue de l'Est 75020 Paris
Contact phone logo +33 (0)1 85 08 52 51
Contact mail logo contact@eisko.com
> contact us!
Eisko logo

BLOG

tech friday #2 : Eisko capture protocol

November 28, 2014

Hi folks!

Last week we started to explain our capture system and approach. Today, let's focus on the protocol : what does the scanned person have to do while being scanned?

When it comes to capturing expressions, a lot of things have to be planned in advance: you cannot afford to recall the person later because you missed a pose. Nor you can capture thousands of expressions just to be on the safe side: in production, things usually need to be done very quickly.

Having worked on human capture for years, we have explored all of the existing capture protocols, and have taken the best aspects of each one to create our own. After years of iteration and improvment, we think we have quite a well-optimized capture protocol. It basically has three main parts:

- Muscle deformations of face (eye ball deformations, opening the mouth, jaw movements, etc.)

- Emotionnal expressions. These are grouped in two categories: primary emotions (happiness, anger, fear, etc.) and secondary emotions (interest, irritation, disregard, sarcastic, etc.)

- Visemes deformations. Visemes are a subset of phonemes. Basically, they represent the shapes that a face makes while speaking. Recording them is a bit more complicated than reciting the letters of the alphabet, as the same as the same phoneme induce different deformations in different contexts (depending on surrounding words and emotional state).
Visemes fall into three main categories:  widened consonants, rounded consonants and monophtongs.

By asking the subject to recite a standard phrase that contains all the visemes in the English or French language, we can quickly capture all the speech-related facial deformations. These phrase are derived from our study of various scientific papers on the subject.


This protocol allows us to capture and characterize all the possible deformations linked to the 104 physiological degree-of-freedom of your face, or my face, or anyone else's.

Once this is done, we are left with Gigabits of data. What we do with that data is a question we will answer in our future Tech fridays. Stay tuned!

Eisko


Tech Friday #1: Yet another capture system? Or: why photogrammetry just isn't good enough!

November 21, 2014



Hi Folks,

In our first Tech Friday post, we wanted to give you a brief insight into the motivations and approach underlying our capture system.

Since their introduction for computer assisted design and quality inspection, a number of different approaches to capturing 3D objects have been investigated:





    scanner scan                                      contact measurment                           structured light




These approaches can be classified according to:
  • The physical properties they can acquire:  '3D scanning' generally refers to geometry capture, potentially supplemented by photos mapping.
  • Wheter capture is passive or active: does the capture process modify/interfer the phenomena it records ?
  • The type of object captured: does the process record static or mobile/articulated/deformable objects ?
While contact measurments and manual scanner are not well adapted to capture non-rigid and deformable objects like the face, structured-light and photogrammetry are potential good candidates for geometry capture.




Photogrammetry is based on the detection and registration of similar features in multiple DSLR images. External and internal calibration of the camera sensors enables to triangulate the associated position in space.

Such capture rigs are easy to set-up and progressively more affordable, thanks to the increasing availability of good-quality cameras. This approach enables to deal with a subject in different poses, since shooting is almost instantaneous.

But looking closer to it, the resulting scan is noisy and unstructured...


It took us some time to realize that such artefacts result from two fundamental causes, independent of the number and resolution of the cameras used:
  • Light scatering caused by specular and subsurface properties of the skin, surface fuzz, eyelashes which reflects and absorb lights depending on the direction so that epipolar features can not be identical in different camera views.
  • The difficulty to achieve a perfectly accurate geometric calibration of the cameras - which still remains an open problem for photogrammetry.
After years of technology survey and experimentations on this thematic, we finally decided to develop our own capture system. Our objective was to ensure the simultaneous acquisition of both detailed geometry and physically-based textures which characterize the precise appearance of any subject's pose.


To do so, we do not consider a direct acquisition, but analyze and dissociate the way in which the skin, the mouth, the eyes diffuse, reflect and absorb light under precisely calibrated illumination conditions.

Our capture system is made of 1.600 LED lights which permit us to control the intensity and direction of the lighting very accurately. Our high-frequency 'material cameras' enable us to separate the diffuse and specular components of the surfaces being recorded, and extract normal information for each RGB color channel, among other things.

The result of this analysis is a clean 3D geometric scan, resolution-independent down to 20 micrometers, complete with tiny details like wrinkles and pores, totally free from reflection and refracton side effects traditionnal in photogrammetry :)







I hope this post has helped you understand why we decided to build another capture system. In future Tech Fridays, we'll look at what makes Eisko unique both on the capture and reconstruction sides.

Eisko

It's alive !

November 18, 2014

We have had the pleasure of collaborating with the dancer, choreographer and actress Tatiana Seguin. With her kind permission, we have been able to reconstruct her digital double.

This week Tatiana saw her digital double, animated in Unity. She was so taken with it that I started filming her reactions with one of our capture camera. It is always extremely impressive to discover your digital self on screen.

"It's alive!" she said :)

(Yes, there is a drawing of rosetta probe on the white board behind Tatiana)


Eisko