Expression

Updated: 01/30/2020

Summary

Expression is a function that handles Cubism’s Expression Motion, which can be set by adding or multiplying the value of a parameter for an expression to the value of the current parameter.
Animator, Unity’s standard motion playback function, does not have Multiply as a Blend Mode that can be set for layers, so it is not possible to express this.

The settings file for the expression function is exported in .exp3.json format.
Click here for more information on the mechanism of facial expressions.

To use Expression, it is necessary to set up UpdateController and ParameterStore in Prefab in advance.

Click here for the corresponding tutorial article.

To play back the Expression Motion, perform the following steps.

1. Create [.exp3.asset]
2. Create [.expressionList.asset] and add [.exp3.asset]
3. Play back Expression Motion
4. Calculate and apply Expression Motion

Create [.exp3.asset]

[.exp3.asset] is a ScriptableObject asset converted from [exp3.json].
Normal operation cannot be guaranteed if [.exp3.asset] is changed.

To convert [exp.json] to [.exp3.asset], do the following.

1. Parse [exp.json]
2. Create CubismExpressionData
3. Create [.exp3.asset]

Parse [exp.json]

To parse [exp3.json], use CubismExp3Json.LoadFrom(string exp3Json) or CubismExp3Json.LoadFrom(TextAsset exp3JsonAsset).

Create CubismExpressionData

CubismExpressionData is a class that records information for the parsed .exp3.json and holds the following data.

Field	Type	Description
Type	string	Type of json file. For .exp3.json, it is set to “Live2D Expression.”
FadeInTime	float	Time for the expression to fade in. The unit of value is seconds.
FadeOutTime	float	Time for the expression to fade out. The unit of value is seconds.
Parameters	SerializableExpressionParameter[]	ID of the parameter to which the expression is applied, the value to be applied, and the calculation method.

Use CubismExpressionData.CreateInstance(CubismExp3Json json) to create a new [.exp3.asset].
Use CubismExpressionData.CreateInstance(CubismExpressionData expressionData, CubismExp3Json json) to overwrite [.exp3.asset].

• CubismExp3Json json: CubismExp3Json is parsed data of [exp3.json].
• CubismExpressionData expressionData: CubismExpressionData to be overwritten.

Create [.exp3.asset]

The [.exp3.asset] is created as follows using AssetDatabase.CreateAsset.

// When creating a new asset
expressionData = CubismExpressionData.CreateInstance(ExpressionJson);
AssetDatabase.CreateAsset(expressionData, AssetPath.Replace(".exp3.json", ".exp3.asset"));
 
// When updating a previously created asset
expressionData = CubismExpressionData.CreateInstance(oldExpressionData, ExpressionJson);
EditorUtility.CopySerialized(expressionData, oldExpressionData);
expressionData = oldExpressionData;
 
// Update Unity database.
EditorUtility.SetDirty(expressionData);

This is done by CubismExpression3JsonImporter.Import().

Note: For runtime, it is not necessary to .asset CubismExpressionData.

Create [.expressionList.asset] and add [.exp3.asset]

[.expressionList.asset] is an asset that lists references to [.exp3.asset] for each model.
This is used in CubismExpressionController to retrieve the expression motion to be played from CurrentExpressionIndex.
The order of the [.exp3.asset] list is added in the import order of [exp3.json].

Note: This item is processed by CubismExpression3JsonImporter.Import().

Creating [.expressionList.asset]

The [.expressionList.asset] is created in the same hierarchy as the model prefab.

// Get existing expressionList
var expressionList = AssetDatabase.LoadAssetAtPath<CubismExpressionList>(expressionListPath);
 
// Create asset
AssetDatabase.CreateAsset(expressionList, expressionListPath);

Note: For runtime, it is not necessary to .asset the CubismExpressionList.

Adding [.exp3.asset]

To add [.exp3.asset] to [.expressionList.asset], do the following.

// When adding a new one
expressionIndex = expressionList.CubismExpressionObjects.Length;
Array.Resize(ref expressionList.CubismExpressionObjects, expressionIndex + 1);
expressionList.CubismExpressionObjects[expressionIndex] = expressionData;
 
// Update previously set elements.
expressionList.CubismExpressionObjects[expressionIndex] = expressionData;
 
// Update Unity database.
EditorUtility.SetDirty(expressionList);

Play back Expression Motion

Expression motions can be played or changed by setting the index of the expression motion to be played to CubismExpressionController.CurrentExpressionIndex.

The following process is used to play back the expression motion.

1. Set the end time of the expression motion currently playing
2. Initialize new expression motions and add them to the playback list of expression motions

Set the end time of the expression motion currently playing

If there is an expression motion in playback, set the end time so that the expression motion in playback ends.

// Set last expression end time
if(_playingExpressions.Count > 0)
{
    var playingExpression = _playingExpressions[_playingExpressions.Count - 1];
    playingExpression.ExpressionEndTime = playingExpression.ExpressionUserTime + playingExpression.FadeOutTime;
    _playingExpressions[_playingExpressions.Count - 1] = playingExpression;
}

This is done by CubismExpressionController.StartExpression().

Initializing new expression motions and adding them to the playback list of expression motions

Create a CubismPlayingExpression with playback information for the facial expression motion.

CubismPlayingExpression holds the following information.

Field	Type	Description
Type	string	Type of json file. For .exp3.json, it is set to “Live2D Expression.”
FadeInTime	float	Time for the expression to fade in. The unit of value is seconds.
FadeOutTime	float	Time for the expression to fade out. The unit of value is seconds.
Weight	float	Expression weights. Values range from 0 to 1.
ExpressionUserTime	float	Elapsed time since the start of facial expression playback. The unit of value is seconds.
ExpressionEndTime	float	The time at which the playback of the facial expression ends. The unit of value is seconds.
Destinations	CubismParameter[]	An array of parameters to which the expression applies.
Value	float[]	The value that the expression applies to the parameter. The number of elements and the order are the same as for Destinations.
Blend	CubismParameterBlendMode[]	A calculation method in which expressions are applied to parameters. The number of elements and the order are the same as for Destinations. The calculation methods set are Override, Additive, and Multiply.

CubismPlayingExpression can be created using CubismPlayingExpression.Create(CubismModel model, CubismExpressionData expressionData).

• CubismModel model: A model that reproduces expression motion.
• CubismExpressionData expressionData: The [exp3.asset] data of the expression motion to be played.

The created expression motion is added to the playback list as follows.

// Create playback facial expressions
var palyingExpression = CubismPlayingExpression.Create(_model, ExpressionsList.CubismExpressionObjects[CurrentExpressionIndex]);
 
// Add to expression playback list
_playingExpressions.Add(palyingExpression);

This is done by CubismExpressionController.StartExpression().

Calculate and apply Expression Motion

Expression calculation and application is called from CubismExpressionController.OnLateUpdate().

Note: The expression must be applied after ParameterStore.SaveParameters().
Expression is a function that applies a new value for expression each frame to the current value.
If applied before ParameterStore.SaveParameters(), the result applied by the expression will be saved, so the values of the parameters for the expression will be added or multiplied indefinitely.