LipSync (UE)

Updated: 09/05/2024

This page contains statements regarding the alpha version.

Summary

LipSync is a function that applies values to the mouth open/close parameter values on the basis of the volume of the set sound source file. In addition to manual operation, values can be manipulated automatically based on parameters set for the component.

How to handle the component

LipSync in the Cubism SDK for Unreal Engine can be used by adding a CubismLipSyncComponent to child components of a CubismModel actor.

If the parameters for lip-sync are set in .model3.json, CubismLipSyncComponent is automatically added to the CubismModel actor as a child component during importing, so no configuration on the user side is required.

How to add target parameters

If the parameters for lip-sync are set in .model3.json, CubismLipSyncComponent will be automatically added as a child component to the CubismModel actor upon importing. If you want to manually add parameters for LipSync, select CubismLipSync from the “Details” tab, open “Live2D Cubism” -> “Ids” from the list displayed at the bottom, and manually rewrite the names of the specified parameters or insert new indexes and manually write parameter names.

Applying parameters

CubismLipSyncComponent applies the value in Value uniformly to all parameters corresponding to the IDs set in Ids on the basis of the blend method specified in BlendMode.

for (const FString& Id : Ids)
{
	UCubismParameterComponent* Destination = Model->GetParameter(Id);

	if (!Destination)
	{
		continue;
	}

	switch (BlendMode)
	{
		case ECubismParameterBlendMode::Overwrite:
		{
			Destination->SetParameterValue(Value);
			break;
		}
		case ECubismParameterBlendMode::Additive:
		{
			Destination->AddParameterValue(Value);
			break;
		}
		case ECubismParameterBlendMode::Multiplicative:
		{
			Destination->MultiplyParameterValue(Value);
			break;
		}
	}
}

Automatically manipulating values

If AutoEnabled is set to true, a value that acts like mouth open/close is automatically applied. This behavior can be controlled by the parameters provided in CubismLipSyncComponent.

• TimeScale
  Indicates the time scale of the behavior. Increasing the value makes the behavior quicker, and decreasing the value makes the behavior slower.

Setting the sound source

If AutoEnabled is set to false, a reference to the sound source data can be added to Source to apply a value on the basis of the volume played from the sound source to the specified parameter. This behavior can be controlled by the parameters provided in CubismLipSyncComponent.

• Gain
  Specify the gain, or amplification factor, of the volume acquired from the sound source. The larger the value, the larger the value applied.
• Smoothing
  Specify the smoothness of mouth movements on the basis of the volume obtained from the sound source.
  The larger the value, the more smoothly mouth movements respond to volume changes.