MouthMovement

Updated: 01/30/2020

Summary

MouthMovement is a function that applies an open/closed state value to the current value of a parameter for lip-sync.
Lip-sync curves set in motion or real-time sampled values from a playing audio file can be applied to the model.
Click here for information on how to set lip-sync parameters for a model.

The only thing set in MouthMovement is the state of mouth opening and closing.
It is not possible to manipulate the shape of the mouth to match the vowel.

To specify parameters for lip-sync in Unity, set them as models in the Cubism editor, or specify them as desired in Unity.

MouthMovement in Cubism SDK for Unity consists of three types of elements.

1. Components for specifying parameters
2. Components that apply values to each parameter
3. Manipulation of the values applied in number 2

1. Components for Specifying Parameters

Use CubismMouthParameter to specify the parameters to be used for MouthMovement.

CubismMouthParameter is a component that inherits from MonoBehaviour and attaches to GameObjects placed under [Prefab root]/Parameters/.

The parameter with the same ID as the GameObject to which it is attached is treated as a parameter for lip-sync.

If the model itself has parameters for lip-sync, the CubismMouthParameter will be attached to the GameObject for that parameter during import.

CubismMouthParameter is used as a marker to get a reference, so it does not process anything internally and has no data.

2. Components that Apply Values to Each Parameter

Use CubismMouthController to apply opening and closing values to each parameter.
This is a component that inherits from MonoBehaviour and attaches to the root of Cubism’s Prefab when used.

Gets a reference to all CubismMouthParameters attached to Prefab at initialization.
If you add/remove parameters for eye blinking during execution, call CubismMouthController.Refresh() to get the reference again.

        /// <summary>
        /// Refreshes controller. Call this method after adding and/or removing <see cref="CubismMouthParameter"/>s.
        /// </summary>
        public void Refresh()
        {
            var model = this.FindCubismModel();


            // Fail silently...
            if (model == null)
            {
                return;
            }


            // Cache destinations.
            var tags = model
                .Parameters
                .GetComponentsMany<CubismMouthParameter>();


            Destinations = new CubismParameter[tags.Length];


            for (var i = 0; i < tags.Length; ++i)
            {
                Destinations[i] = tags[i].GetComponent<CubismParameter>();
            }

            // Get cubism update controller.
            HasUpdateController = (GetComponent<CubismUpdateController>() != null);
        }

        ...

        /// <summary>
        /// Called by Unity. Makes sure cache is initialized.
        /// </summary>
        private void Start()
        {
            // Initialize cache.
            Refresh();
        }

CubismMouthController applies the value of CubismMouthController.MouthOpening to the parameters marked by CubismMouthParameter at the timing of LateUpdate() in every frame.

        /// <summary>
        /// Called by cubism update controller. Updates controller.
        /// </summary>
        /// <remarks>
        /// Make sure this method is called after any animations are evaluated.
        /// </remarks>
        public void OnLateUpdate()
        {
            // Fail silently.
            if (!enabled || Destinations == null)
            {
                return;
            }


            // Apply value.
            Destinations.BlendToValue(BlendMode, MouthOpening);
        }

The value set for MouthOpening ranges from 0.0f to 1.0f.
CubismMouthController applies this value to the target parameter using the calculation method set in CubismMouthController.BlendMode.

By manipulating this MouthOpening value from the outside, the model’s mouth can be opened and closed.

        /// <summary>
        /// The opening of the mouth.
        /// </summary>
        [SerializeField, Range(0f, 1f)]
        public float MouthOpening = 1f;

3. Manipulation of the Values Applied in Number 2

As described in “2. Components that Apply Values to Each Parameter,” values can be applied to parameters for lip-sync by manipulating the value of CubismMouthController.MouthOpening.

The Cubism SDK for Unity includes three different ways to manipulate this value.

• Manipulate values by motion
• Manipulate values periodically
• Manipulate values by sampling from AudioClip

You can also customize your own lip-sync speed and timing by implementing a process to manipulate this value.

Manipulate values by motion

When creating motion in Cubism’s Animator using a model with parameters set for eye blinking, it is possible to set a curve for eye blinking.

If a .motion3.json file with a curve set for eye blinking is imported into a Unity project, the AnimationClip will have that curve generated for the CubismMouthController.MouthOpening value.
Therefore, the value of CubismMouthController.MouthOpening is manipulated by playing that AnimationClip in the Animator component, etc.

Manipulate values periodically

To have the value for lip-sync manipulated periodically, use CubismAutoMouthInput.
CubismAutoMouthInput is a component that calculates and sets the lip-sync value with a sine wave.

To use CubismAutoMouthInput, attach it to the root of Cubism’s Prefab.

CubismAutoMouthInput has one setting item.

• Timescale

The period of the sine wave changes.

        /// <summary>
        /// Timescale.
        /// </summary>
        [SerializeField]
        public float Timescale = 10f;

        /// <summary>
        /// Called by Unity. Updates controller.
        /// </summary>
        /// <remarks>
        /// Make sure this method is called after any animations are evaluated.
        /// </remarks>
        private void LateUpdate()
        {
            // Fail silently.
            if (Controller == null)
            {
                return;
            }


            // Progress time.
            T += (Time.deltaTime * Timescale);


            // Evaluate.
            Controller.MouthOpening = Mathf.Abs(Mathf.Sin(T));
        }

Manipulate values by sampling from AudioClip

Use CubismAudioMouthInput to set the lip-sync value from the audio played in Unity.

CubismAudioMouthInput generates and sets lip-sync values in real-time by sampling from the audio information during playback obtained from AudioSource.

To use CubismAudioMouthInput, attach it to the root of Cubism’s Prefab.

        /// <summary>
        /// Samples audio input and applies it to mouth controller.
        /// </summary>
        private void Update()
        {
            // 'Fail' silently.
            if (AudioInput == null)
            {
                return;
            }


            // Sample audio.
            var total = 0f;


            AudioInput.GetOutputData(Samples, 0);


            for (var i = 0; i < Samples.Length; ++i)
            {
                var sample = Samples[i];


                total += (sample * sample);
            }


            // Compute root mean square over samples.
            var rms = Mathf.Sqrt(total / Samples.Length) * Gain;


            // Clamp root mean square.
            rms = Mathf.Clamp(rms, 0.0f, 1.0f);


            // Smooth rms.
            rms = Mathf.SmoothDamp(LastRms, rms, ref VelocityBuffer, Smoothing * 0.1f);


            // Set rms as mouth opening and store it for next evaluation.
            Target.MouthOpening = rms;


            LastRms = rms;
        }

CubismAudioMouthInput has four settings.

        /// <summary>
        /// Audio source to sample.
        /// </summary>
        [SerializeField]
        public AudioSource AudioInput;


        /// <summary>
        /// Sampling quality.
        /// </summary>
        [SerializeField]
        public CubismAudioSamplingQuality SamplingQuality;


        /// <summary>
        /// Audio gain.
        /// </summary>
        [Range(1.0f, 10.0f)]
        public float Gain = 1.0f;

        /// <summary>
        /// Smoothing.
        /// </summary>
        [Range(0.0f, 1.0f)]
        public float Smoothing;

• AudioInput

Reference to the AudioSource to be sampled.

• SamplingQuality

The accuracy of the audio to be sampled.

• Gain

The multiplier of the sampled value.
When this is set to 1, the multiplier is 1. The larger the value, the larger the lip-sync value.

• Smoothing

The amount of smoothing of the sampled value.
The larger the value, the smoother the lip-sync value changes.