Motion-sync
Updated: 10/17/2023
By utilizing the motion-sync function, realistic mouth movements can be achieved that match with the voice.
The specified audio data is converted into a time-series of visemes and mouth motions are automatically generated by blending the corresponding shapes for each viseme pre-defined by the user.
Note: We are distributing a sample model “Kay” that features motion-sync, so please make use of this resource.
Displaying the Settings dialog box
Click the palette menu in the [Parameter] palette -> [Motion-sync Settings].
Alternatively, the dialog box can be accessed via the [Modeling] menu -> [Parameter] -> [Motion-sync Settings].
Motion-sync settings
| Number | Item | Details |
|---|---|---|
| (1) | Menu | Various menus are displayed. Details are explained in the “Menu details” section of this page. |
| (2) | Load audio | Loads an audio file (.wav). |
| (3) | First frame | Returns the audio to the beginning. |
| (4) | Playback/Stop | Plays/stops audio while moving the mouth of the model placed in the view according to the motion-sync settings. |
| (5) | Repeat | Switch repeat playback on/off. |
| (6) | Motion-sync Settings | You can add and edit motion-sync settings that are linked to the model. |
| (7) | Sample rate | Set the number of audio elements sampled per second. The larger the value, the finer the analysis results, but the greater the vibration and analysis error. For example, if you have fast speech, raise the value to about 60 to make the motion more suitable for the speech. |
| (8) | Blending ratio | You can set the degree to which the audio elements are blended. The smaller the value, the more the shape is determined mainly by the audio element with the highest weight. The higher the value, the more evenly all audio element weights are blended to determine the shape. |
| (9) | Smoothing | This setting smoothes out the results of audio analysis and mitigates vibrations. The larger the value, the smoother the movement. |
| (10) | Audio analysis type | Select a library for audio analysis. Note: Currently, only [Lip-sync (CRI LipSync)] can be selected. Lip-sync (CRI LipSync): All platforms are supported. |
| (11) | Preset | This is a preset in which the [Mapping settings], [Audio element], and [Parameter] are preconfigured. You can also add the presets from the [add] menu. |
| (12) | Mapping settings | Set the value of the parameter for each audio element. |
| (13) | Audio element | You can adjust settings for elements analyzed from audio data. |
| (14) | Parameter | Parameters that control motions can be set. |
About audio files
Tips
Only [WAV format] audio files can be read.
Some formats, even [WAV format], are not supported and may not be loaded with a warning message.
In such cases, encoding to the corresponding [WAV format (16 bit, 44100 Hz)] may allow reading.
Precautions
Stereo audio may be less accurate because of the process of converting audio to mono during analysis.
In addition, audio with low volume may not be recognized correctly, in which case, please adjust the volume of the audio file beforehand.
Sample rate
Set the number of audio elements sampled per second.
The larger the value, the finer the analysis results, but the greater the vibration and analysis error.
| Value of [Sample rate] | 15 | 100 |
| Change of movement | Gentle | Intense |
| Image |  |  |
Blending ratio
You can set the degree to which the audio elements are blended.
The smaller the value, the more the shape is determined mainly by the audio element with the highest weight.
The higher the value, the more evenly all audio element weights are blended to determine the shape.
| Value of [Blending ratio] | 0.0 | 1.0 |
| Change of movement | Briskly | Smooth |
| Image |  |  |
The figure below compares the graphs of “Mouth Open/Close.”
Green: Blending ratio = 0.0
Red: Blending ratio = 1.0
Smoothing
This setting smoothes out the results of audio analysis and mitigates vibrations.
The larger the value, the smoother the movement.
| Value of [Smoothing] | 1 | 100 |
| Change of movement | Intense | Smooth |
| Image |  |  |
The figure below compares the graphs of “Mouth Open/Close.”
Green: Smoothing = 1
Red: Smoothing = 100
Precautions
The sample rate, blending ratio, and smoothing are not included in the presets.
Menu details
| Item | Details |
|---|---|
| Edit | Undo: Restores the state immediately before the last operation. Shortcut key [Ctrl] + [Z] Redo: Executes the operation performed before going back one step. Shortcut key [Ctrl] + [Y] |
| Motion-sync | Motion-sync Export: Write out the .motionsync3.json file. See the “Data for Embedded Use” page for a detailed explanation of exported data. Loading motion-sync: Load the .motionsync3.json file. Initialize presets: Initializes the preset. Please note that the custom presets you have created will also be deleted. |
| Help | Motion-sync Settings: Open the motion-sync page. |
Motion-sync Settings
The settings in which “Audio analysis type,” “Mapping settings,” “Audio element,” and “Parameter” are stored.
These settings are linked to the model data (.cmo3).
It is also possible to export and import these settings in .motionsync3.json format.
Selecting, adding, and editing settings
| Number | Item | Details |
|---|---|---|
| (1) | Setting | Select the motion-sync setting. |
| (2) | Add | Create and add a new motion-sync setting. |
| (3) | Edit | The [Motion-sync Settings List] dialog box opens to edit each setting. |
Motion-sync Settings List
The [Motion-sync Settings List] dialog box allows you to duplicate or rename motion-sync settings.
| Number | Item | Details |
|---|---|---|
| (1) | Duplicate | Duplicate the selected motion-sync setting. |
| (2) | Name change | Rename the selected motion-sync setting. |
| (3) | Delete | Delete the selected motion-sync setting. |
Preset
This is a preset in which the [Mapping settings], [Audio element], and [Parameter] are preconfigured.
Presets allow for easy configuration of motion-sync settings.
You can also add presets.
Selecting, adding, and editing presets
| Number | Item | Details |
|---|---|---|
| (1) | Select preset. | Select the default preset or a preset you have added. |
| (2) | Add | Create and add a new preset. |
| (3) | Edit | The [Preset List] dialog box opens to edit each setting. |
Preset list
In the [Preset List] dialog box, you can duplicate or rename presets.
| Number | Item | Details |
|---|---|---|
| (1) | Duplicate | Duplicate the selected preset. |
| (2) | Name change | Rename the selected preset. |
| (3) | Delete | Delete the selected preset. Note: Default presets cannot be deleted. |
Default preset list
Two default presets are provided for the motion-sync settings.
| Number | Preset name | Details |
|---|---|---|
| (1) | Basic | This can be used in basic models built with two parameters: “Mouth Deformation” and “Mouth Open/Close.” This preset is designed to create the parameter on two axes, “Mouth Deformation” and “Mouth Open/Close,” as shown in the figure below.  The following is a guideline for the settings. (1): A (2): I (3): O (4): U |
| (3) | Vowels only (blend shape) | If you want to create a parameter for each viseme, this preset is recommended. |
Mapping settings
The mapping setting sets up a form for each audio element.
| Number | Item | Details |
|---|---|---|
| (1) | Audio element | Parameters for which [Valid] in [Audio element] is checked. You can use the combo box to switch and set a value for each audio element. Please refer to the images displayed below for the shape of each audio element. |
| (2) | Apply | The values of the Mapping settings are updated to the values on the current [Parameter] palette. |
| (3) | Confirm | You can update the values on the [Parameter] palette to the values of the Mapping settings and see the form on the canvas. |
| (4) | Parameter name | Parameters that control motions. Note: If a parameter matching the parameter ID, minimum value, and maximum value in the [Parameter] list does not exist in the model, it is displayed in red. Note: Also, the setting values of parameters indicated in red are ignored. |
| (5) | Value | Set the value of the parameter for the form corresponding to each audio element. |
Mapping setting example
As an example, the mapping settings for [A] are described below.
Select [A] from [Audio element] in the mapping settings and set the value for each parameter on the right side.
After moving the sliders in the [Parameter] palette to determine the shape, you can easily set the shape by pressing the [Apply] button in the [Motion-sync Settings] dialog box.
After setting, set other audio elements as well.
Set all necessary audio elements to complete the mapping setting.
Point
Mapping settings are preconfigured in the presets.
If you want to use the presets as they are, you do not need to change any settings.
If you wish to change the mapping settings, please refer to the example settings above.
Audio element
Create settings for elements analyzed from audio data.
Motions can be adjusted by adjusting each setting.
| Number | Item | Details |
|---|---|---|
| (1) | Valid | Enables/disables audio analysis. Disabled items are excluded from audio analysis, and are also excluded from [Audio element] in the Mapping settings. |
| (2) | Viseme | The type of audio element to be mapped. Silence: Silence A, I, U, E, O: Vowels |
| (3) | Scale | The value by which to multiply the calculated value of each viseme. The higher this value, the more the target viseme is emphasized. |
Scale
The value by which to multiply the calculated value of each viseme.
The higher this value, the more the viseme is emphasized.
| Value of [Scale] for A | 1.0 | 10.0 |
| Change of movement | — | Emphasis on the adjusted viseme |
| Image |  |  |
The figure below compares the graphs of “A.”
Green: Scale = 1.0
Red: Scale = 10.0
Parameter
Set parameters that control motions.
| Number | Item | Details |
|---|---|---|
| (1) | Add | Add parameters that control motions. |
| (2) | Delete | Delete parameters that control motions. |
| (3) | Name | Parameters that control motions. Note: If a parameter matching the parameter ID, minimum value, and maximum value in this list does not exist in the model, it is displayed in red. Note: Also, the setting values of parameters indicated in red are ignored. |
| (4) | ID | The ID of the parameter. |
| (5) | Threshold | This setting mitigates fine vibration. The larger the value, the more effective the vibration mitigation. |
| (6) (7) | Minimum value Maximum value | The minimum and maximum values of the parameter |
Threshold
This setting mitigates fine vibration.
The larger the value, the more effective the vibration mitigation.
| Value of [Threshold] | 0.0 | 0.1 |
| Change of movement | — | Vibration is mitigated |
| Image |  |  |
The figure below compares the graphs of “Mouth Open/Close.”
Green: Threshold = 0.0
Red: Threshold = 0.1
Procedure for Motion-sync Settings
This section describes the procedure for setting up motion-sync.
- Click the palette menu in the [Parameter] palette -> [Motion-sync Settings] to open the dialog box.
- Click [add] in the motion-sync settings to add motion sync settings. Enter a setting name and select an audio analysis type and a preset for the setting.
- If the preset does not have the parameters you want, delete the parameters you do not need from the [Parameter] tab and add the parameters you want to move in motion-sync.
- Set the parameter values for each audio element in [Mapping Settings].
The [Apply] and [Confirm] buttons can be used for efficiency.
- Click [Load audio] to play the motion.
If you want to mitigate vibrations or emphasize certain audio elements, please use [Audio element] or [Parameter] for more detailed settings.