The project aimed to bring two 3D characters to life. Eventually, we edited two characters so that communication through them looks natural. Therefore, we adjusted the movement of their mouth to the language by manipulating the animation of their facial expressions.
Visemes
We consider the following words
- glass
- grass
These are two different words with different meanings. However, the two words only differ at one point (marked in color [red]). Thus, the two sounds [l, r] are different phonemes. [1] Phonemes are an abstract class of all sounds that have the same meaning-distinguishing function in a spoken language.
The positioning the lips when pronouncing a phoneme is called a viseme. They are the visual counterparts of phonemes.
Using visemes in computer graphics is advantageous because it means that a 3D character only needs a few visemes to simulate the pronunciation of a word. With the corona infection protection measures, it has become clear how crucial the movement of the mouth area is for the intelligibility of speech.
The design of facial features, including the speech visem, also influences the human-machine communication: Using visemes in computer graphics is advantageous because it means that a 3D character only needs a few visemes to simulate the pronunciation of a word.
The design of the visemes on the characters was based on the results of the study by Dawid Jachimski, Andrzej Czyzewski and Tomaz Ciszewski.
A text (in ASCII) is broken down into its phonemes via a text-to-speech system. In addition, a synthetic speech is generated. The process of converting text into human speech is known as speech synthesis. The resulting phonemes of the TTS system reach our characters via a UDP listener. This will address the characters’ predefined visemes.
Creation of a 3D character
We used the Fuse and Mixamo applications from Adobe to create the characters. Fuse offers the possibility to assemble a character piece by piece and to personalize.
Fuse enables a direct connection to the Mixamo application. Mixamo offers the possibility to the rigging of the 3D object and to download the character for 3D software. Including the associated textures and material maps as well as a basic loop animation of our character.
Design of the animation processes
The further design of the visem can now be created using the blend shapes on the character by including files using the drag-and-drop method.
The development course and the result of the project are available via the GitLab-Repository https://gitlab.reutlingen-university.de/ghebremu/unity-characters
[1] http://www.fb10.uni-bremen.de/homepages/hackmack/phonmorph/pdf/Phonem.pdf