MouthMovement (Cocos Creator)

Updated: 03/14/2023

Summary

MouthMovement is a function that applies an open/closed state value to the current value of a parameter for lip-sync.
Lip-sync curves set in motion or real-time sampled values from a playing audio file can be applied to the model.
Click here for information on how to set lip-sync parameters for a model.

The only thing set in MouthMovement is the state of mouth opening and closing.
It is not possible to manipulate the shape of the mouth to match the vowel.

To specify parameters for lip-sync in Cocos Creator, set them as models in the Cubism editor, or specify them as desired in Cocos Creator.

MouthMovement in the Cubism SDK for Cocos Creator consists of three types of elements.

1. Components for specifying parameters
2. Components that apply values to each parameter
3. Manipulation of the values applied in number 2

1. Components for Specifying Parameters

Use CubismMouthParameter to specify the parameters to be used for MouthMovement.

CubismMouthParameter is a component that inherits from Component and attaches to the Node placed under [Prefab root]/Parameters/.

The parameter with the same ID as the Node to which it is attached is treated as a parameter for lip-sync.

If the model itself has parameters for lip-sync, the CubismMouthParameter will be attached to the Node for that parameter during import.

CubismMouthParameter is used as a marker to get a reference, so it does not process anything internally and has no data.

2. Components that Apply Values to Each Parameter

Use CubismMouthController to apply opening and closing values to each parameter.
This is a component that inherits from Component and attaches to the root of Cubism’s Prefab when used.

Gets a reference to all CubismMouthParameters attached to Prefab at initialization.
If you add/remove parameters for eye blinking during execution, call CubismMouthController.refresh() to get the reference again.

  public refresh() {
    const model = CoreComponentExtensionMethods.findCubismModel(this);

    // Fail silently...
    if (model == null || model.parameters == null) {
      return;
    }

    // Cache destinations.
    const tags = ComponentExtensionMethods.getComponentsMany(
      model.parameters,
      CubismMouthParameter
    );

    this.destinations = new Array(tags.length);

    for (let i = 0; i < tags.length; ++i) {
      this.destinations[i] = tags[i].getComponent(CubismParameter);
    }

    // Get cubism update controller.
    this.hasUpdateController = this.getComponent(CubismUpdateController) ! = null;
  }

...

  protected start() {
    // Initialize cache.
    this.refresh();
  }

CubismMouthController applies the value of CubismMouthController.MouthOpening to the parameters marked by CubismMouthParameter at the timing of lateUpdate() in every frame.

  protected onLateUpdate(deltaTime: number) {
    // Fail silently.
    if (!this.enabled || this.destinations == null) {
      return;
    }

    // Apply value.
    CubismParameterExtensionMethods.blendToValueArray(
      this.destinations,
      this.blendMode,
      this.mouthOpening
    );
  }

The value set for MouthOpening ranges from 0.0f to 1.0f.
CubismMouthController applies this value to the target parameter using the calculation method set in CubismMouthController.blendMode.

By manipulating this MouthOpening value from the outside, the model’s mouth can be opened and closed.

  @property({ type: CCFloat, slide: true, range: [0.0, 1.0, 0.01] })
  public mouthOpening: number = 1.0;

3. Manipulation of the Values Applied in Number 2

As described in “2. Components that Apply Values to Each Parameter,” values can be applied to parameters for lip-sync by manipulating the value of CubismMouthController.mouthOpening.

The Cubism SDK for Cocos Creator provides the following three ways to manipulate this value.

• Manipulate values by motion
• Manipulate values periodically
• Manipulate values by sampling from AudioClip

You can also customize your own lip-sync speed and timing by implementing a process to manipulate this value.

Tips

It may not work as intended if the order of execution of the components operating CubismMouthController.mouthOpening is later than CubismMouthController.
If a problem arises, it is possible to work around it by explicitly controlling the order in which components are executed on the user side.
Cubism SDK for Cocos Creator controls the execution order of each component with CubismUpdateController, which can also be used.

In addition, since each of the above three setting methods manipulates the same value, it is difficult for them to coexist in a single model without an innovative solution.

Manipulate values by motion

When creating motion in Cubism’s Animator using a model with parameters set for eye blinking, it is possible to set a curve for eye blinking.

If a .motion3.json file with a curve set for eye blinking is imported into a Cocos Creator project, the Animation will have that curve generated for the CubismMouthController.mouthOpening value.
Therefore, the value of CubismMouthController.mouthOpening is manipulated by playing that AnimationClip in the Animator component, etc.

Manipulate values periodically

To have the value for lip-sync manipulated periodically, use CubismAutoMouthInput.
CubismAutoMouthInput is a component that calculates and sets the lip-sync value with a sine wave.

To use CubismAutoMouthInput, attach it to the root of Cubism’s Prefab.

CubismAutoMouthInput has one setting item.

• Timescale
  
  The period of the sine wave changes.

  @property(CCFloat)
  public Timescale: number = 10.0;

  lateUpdate(deltaTime: number) {
    // Fail silently.
    if (this.Controller == null) {
      return;
    }

    // Progress time.
    this.T += deltaTime * this.Timescale;

    // Evaluate.
    this.Controller.mouthOpening = Math.abs(Math.sin(this.T));
  }

Manipulate values by sampling from audio

Use CubismAudioMouthInput to set the lip-sync value from the audio played in Cocos Creator.

CubismAudioMouthInput generates and sets lip-sync values in real-time by sampling from the audio information during playback obtained from AudioSource.

To use CubismAudioMouthInput, attach it to the root of Cubism’s Prefab.

  protected update(deltaTime: number) {
    const { audioInput, samples, target, sampleRate, gain, smoothing } = this;

    // 'Fail' silently.
    if (audioInput == null || target == null || samples == null || sampleRate == 0) {
      return;
    }
    const { trunc, sqrt } = Math;

    const { currentTime } = audioInput;
    const pos = trunc(currentTime * this.sampleRate);
    let length = 256;
    switch (this.samplingQuality) {
      case CubismAudioSamplingQuality.veryHigh:
        length = 256;
        break;
      case CubismAudioSamplingQuality.maximum:
        length = 512;
        break;
      default:
        length = 256;
        break;
    }

    // Sample audio.
    let total = 0.0;

    for (let i = 0; i < length; i++) {
      const sample = samples.getData((pos + i) % samples.length);
      total += sample * sample;
    }

    // Compute root mean square over samples.
    let rms = sqrt(total / length) * gain;

    // Clamp root mean square.
    rms = math.clamp01(rms);

    // Smooth rms.
    const output = MathExtensions.Float.smoothDamp(
      this.lastRms,
      rms,
      this.velocityBuffer,
      smoothing * 0.1,
      undefined,
      deltaTime
    );

    rms = output[0];
    this.velocityBuffer = output[1];

    // Set rms as mouth opening and store it for next evaluation.
    target.mouthOpening = rms;

    this.lastRms = rms;
  }

CubismAudioMouthInput has four settings.

  @property(AudioSource)
  public audioInput: AudioSource | null = null;

  @property({ type: Enum(CubismAudioSamplingQuality) })
  public samplingQuality: CubismAudioSamplingQuality = CubismAudioSamplingQuality.high;

  @property({ type: CCFloat, slide: true, range: [1.0, 10.0, 0.01] })
  public gain: number = 1.0;

  @property({ type: CCFloat, slide: true, range: [0.0, 1.0, 0.01] })
  public smoothing: number = 0.0;

• audioInput
  
  Reference to the AudioSource to be sampled.
• samplingQuality
  
  The accuracy of the audio to be sampled.
• gain
  
  The multiplier of the sampled value.
  
  When this is set to 1, the multiplier is 1. The larger the value, the larger the lip-sync value.
• smoothing
  
  The amount of smoothing of the sampled value.
  
  The larger the value, the smoother the lip-sync value changes.