MouthMovement

[Last update: 01/30/2020]

Summary

MouthMovement is a function that applies an open/closed state value to the current value of a parameter for lip-sync.
Lip-sync curves set in motion or real-time sampled values from a playing audio file can be applied to the model.
See here for information on how to set lip-sync parameters for a model.

The only thing set in MouthMovement is the state of mouth opening and closing.
It is not possible to manipulate the shape of the mouth to match the vowel.

To specify parameters for lip-sync in Unity, the model can be set in the Cubism editor, or the user can specify them arbitrarily in Unity.

MouthMovement in Cubism SDK for Unity consists of three types of elements.

  1. Components for specifying parameters
  2. Components that apply values to each parameter
  3. Manipulation of values to be applied in 2

 

1. Components for specifying parameters

Use CubismMouthParameter to specify the parameters to be used for MouthMovement.

CubismMouthParameter is a component that inherits from MonoBehaviour and
attaches to GameObjects placed under [Prefab root]/Parameters/.
The parameter with the same ID as the GameObject to which it is attached is treated as a parameter for lip-sync.

If the model itself has parameters for lip-sync, the CubismMouthParameter will be attached to the GameObject for that parameter during import.

CubismMouthParameter is used as a marker to get a reference, so it does not process anything internally and has no data.

 

2. Components that apply values to each parameter

Use CubismMouthController to apply opening and closing values to each parameter.
This is a component that inherits from MonoBehaviour and attaches to the root of Cubism's Prefab when used.

Gets a reference to all CubismMouthParameters attached to Prefab at initialization.
If you add/remove parameters for blinking during execution, call CubismMouthController.Refresh() to get the reference again.

 

CubismMouthController applies the value of CubismMouthController.MouthOpening to the parameters marked by CubismMouthParameter at the timing of LateUpdate() in every frame.

 

The value set for MouthOpening ranges from 0.0f to 1.0f.
CubismMouthController applies this value to the target parameter using the calculation method set in CubismMouthController.BlendMode.

By manipulating this MouthOpening value from the outside, the model's mouth can be opened and closed.

 

 

3. Manipulation of values to be applied in 2

As described in “2. Components to apply values to each parameter,” values can be applied to parameters for lip-sync by manipulating the value of CubismMouthController.MouthOpening.

The Cubism SDK for Unity includes three different ways to manipulate this value.

  • Manipulate values by motion
  • Periodic value manipulation
  • Sampling and manipulating values from AudioClip

Users can also customize their own lip-sync speed and timing by implementing a process to manipulate this value.

Tips
MouthOpening may not work as intended if the order of execution of the components operating CubismMouthController.MouthOpening is later than CubismMouthController.
If a problem arises, it is possible to work around it by explicitly controlling the order in which components are executed on the user side.
Cubism SDK for Unity controls the execution order of each component with CubismUpdateController, which can also be used.

In addition, since each of the above three setting methods manipulates the same value, it is difficult for them to coexist in a single model without any innovations.

 

Manipulate values by motion

When creating motion in Cubism's Animator using a model with parameters set for blinking, it is possible to set a curve for blinking.

If a .motion3.json with a curve set for blinking is imported into a Unity project, the AnimationClip will have that curve generated for the CubismMouthController.MouthOpening value.
Therefore, the value of CubismMouthController.MouthOpening is manipulated by playing that AnimationClip in the Animator component, etc.

 

Periodic value manipulation

To have the value for lip-sync manipulated periodically, use CubismAutoMouthInput.
CubismAutoMouthInput is a component that calculates and sets the lip-sync value with a sine wave.

To use CubismAutoMouthInput, attach it to the root of Cubism's Prefab.

CubismAutoMouthInput has one setting item.

  • Timescale

The period of the sine wave changes.

 

 

Sampling and manipulating values from AudioClip

Use CubismAudioMouthInput to set the lip-sync value from the audio played on Unity.

CubismAudioMouthInput generates and sets lip-sync values in real-time by sampling from the audio information during playback obtained from AudioSource.

To use CubismAudioMouthInput, attach it to the root of Cubism's Prefab.

 

CubismAudioMouthInput has four settings.

  • AudioInput

Reference to the AudioSource to be sampled.

  • SamplingQuality

The accuracy of the audio to be sampled.

  • Gain

The multiplier of the sampled value.
Equal to 1, the larger the value, the larger the lip-sync value.

  • Smoothing

The amount of smoothing of the sampled value.
The larger the value, the smoother the lip-sync value changes.

© 2010 - 2022 Live2D Inc.