Expression (Cocos Creator)

Updated: 03/14/2023

Summary

Expression is a function that handles Cubism’s Expression Motion, which can be set by adding or multiplying the value of a parameter for an expression to the value of the current parameter.
Animator, Cocos Creator’s standard motion playback function, does not have Multiply as a Blend Mode that can be set for layers, so it is not possible to express this.

The settings file for the expression function is exported in .exp3.json format.
Click here for more information on the mechanism of facial expressions.

To use Expression, it is necessary to “How to Set Up UpdateController” and “Save/Restore the Values to Be Manipulated” in Prefab in advance.

Click here for the corresponding tutorial article.

To play back the Expression Motion, perform the following steps.

1. Create [.exp3.asset]
2. Create [.expressionList.asset] and add [.exp3.asset]
3. Play back Expression Motion
4. Calculate and apply Expression Motion

Create [.exp3.asset]

[.exp3.asset] is a ScriptableObject asset converted from [exp3.json].
Normal operation cannot be guaranteed if [.exp3.asset] is changed.

To convert [exp.json] to [.exp3.asset], do the following.

1. Parse [exp.json]
2. Create CubismExpressionData
3. Create [.exp3.asset]

Parse [exp.json]

To parse [exp3.json], use CubismExp3Json.LoadFrom(string exp3Json) or CubismExp3Json.LoadFrom(TextAsset exp3JsonAsset).

Create CubismExpressionData

CubismExpressionData is a class that records information for the parsed .exp3.json and holds the following data.

Field	Type	Description
Type	string	Type of json file. For .exp3.json, it is set to “Live2D Expression.”
FadeInTime	number	Time for the expression to fade in. The unit of value is seconds.
FadeOutTime	number	Time for the expression to fade out. The unit of value is seconds.
Parameters	SerializableExpressionParameter[]	ID of the parameter to which the expression is applied, the value to be applied, and the calculation method.

Use CubismExpressionData.CreateInstance(CubismExp3Json json) to create a new [.exp3.asset].

Use CubismExpressionData.CreateInstance(CubismExpressionData expressionData, CubismExp3Json json) to overwrite [.exp3.asset].

• CubismExp3Json json: CubismExp3Json is parsed data of [exp3.json].
• CubismExpressionData expressionData: CubismExpressionData to be overwritten.

Create [.exp3.asset]

Create [.exp3.asset] as follows.

    const jsonSrc = readFileSync(asset.source, UTF8);
    const json = CubismExp3Json.loadFrom(jsonSrc);
    if (json == null) {
      console.error('CubismExp3Json.loadFrom() is failed.');
      return false;
    }
    const data = CubismExpressionData.createInstance(json);
    const serialized = EditorExtends.serialize(data);
    writeFileSync(outputFilePath, jsonSrc);
    asset.saveToLibrary('.json', serialized);
    refresh(outputFilePath);

This is done by CubismExpression3JsonImporter.import().

Note: For runtime, it is not necessary to .asset CubismExpressionData.

Create [.expressionList.asset] and add [.exp3.asset]

[.expressionList.asset] is an asset that lists references to [.exp3.asset] for each model.
This is used in CubismExpressionController to retrieve the expression motion to be played from CurrentExpressionIndex.
The order of the [.exp3.asset] list is added in the import order of [exp3.json].

Creating [.expressionList.asset]

The [.expressionList.asset] is created in the same hierarchy as the model prefab.

    const source = JSON.parse(readFileSync(asset.source, UTF8));
    const expDataUuidsSource = Array.isArray(source.cubismExpressionObjects)
      ? (source.cubismExpressionObjects as any[])
      : undefined;
    if (expDataUuidsSource == null) {
      console.error('CubismExpressionListImporter.import(): parse error.');
      return false;
    }
    const expDataUuids = purseCubismExpressionObjects(expDataUuidsSource);
    if (expDataUuids == null) {
      console.error('CubismExpressionListImporter.import(): CubismExpressionObjects parse error.');
      return false;
    }
    const cubismExpressionObjects = new Array<CubismExpressionData>(expDataUuids.length);
    for (let i = 0; i < cubismExpressionObjects.length; i++) {
      const uuid = expDataUuids[i];
      // @ts-ignore
      cubismExpressionObjects[i] = EditorExtends.serialize.asAsset(
        uuid,
        CubismExpressionData
      ) as CubismExpressionData;
    }
    const list = new CubismExpressionList();
    list.cubismExpressionObjects = cubismExpressionObjects;
    await asset.saveToLibrary('.json', EditorExtends.serialize(list));

Note: This item is processed by CubismExpressionListImporter.import().

Adding [.exp3.asset]

To add [.exp3.asset] to [.expressionList.asset], do the following.

    const jsonSrc = readFileSync(asset.source, 'utf8');
    const json = CubismExp3Json.loadFrom(jsonSrc);
    if (json == null) {
      console.error('CubismExp3Json.loadFrom() is failed.');
      return false;
    }
    const data = CubismExpressionData.createInstance(json);
    const serialized = EditorExtends.serialize(data);
    asset.saveToLibrary('.json', serialized);
    refresh(outputFilePath);

    const dirPath = Path.join(Path.dirname(asset.source), '../');
    const dirName = Path.basename(dirPath);
    if (dirPath.length == 0) {
      console.warn('CubismExpressionDataImporter.import(): Not subdirectory.');
      return true;
    }
    const expressionListFilePath = Path.join(
      dirPath,
      dirName + `.${CubismExpressionListImporter.extension}`
    );
    updateExpressionListFile(expressionListFilePath, asset.uuid);
    refresh(expressionListFilePath);

Note: This item is processed by CubismExpressionDataImporter.import().

Play back Expression Motion

Expression motions can be played or changed by setting the index of the expression motion to be played to CubismExpressionController.CurrentExpressionIndex.

The following process is used to play back the expression motion.

1. Set the end time of the expression motion currently playing
2. Initialize new expression motions and add them to the playback list of expression motions

Set the end time of the expression motion currently playing

If there is an expression motion in playback, set the end time so that the expression motion in playback ends.

    const playingExpressions = this._playingExpressions;
    if (playingExpressions.length > 0) {
      const playingExpression = playingExpressions[playingExpressions.length - 1];
      playingExpression.expressionEndTime =
        playingExpression.expressionUserTime + playingExpression.fadeOutTime;
      playingExpressions[playingExpressions.length - 1] = playingExpression;
    }

This is done by CubismExpressionController.startExpression().

Initializing new expression motions and adding them to the playback list of expression motions

Create a CubismPlayingExpression with playback information for the facial expression motion.

CubismPlayingExpression holds the following information.

Field	Type	Description
Type	string	Type of json file. For .exp3.json, it is set to “Live2D Expression.”
FadeInTime	number	Time for the expression to fade in. The unit of value is seconds.
FadeOutTime	number	Time for the expression to fade out. The unit of value is seconds.
Weight	number	Expression weights. Values range from 0 to 1.
ExpressionUserTime	number	Elapsed time since the start of facial expression playback. The unit of value is seconds.
ExpressionEndTime	number	The time at which the playback of the facial expression ends. The unit of value is seconds.
Destinations	CubismParameter[]	An array of parameters to which the expression applies.
Value	number[]	The value that the expression applies to the parameter. The number of elements and the order are the same as for Destinations.
Blend	CubismParameterBlendMode[]	A calculation method in which expressions are applied to parameters. The number of elements and the order are the same as for Destinations. The calculation methods set are Override, Additive, and Multiply.

CubismPlayingExpression can be created using CubismPlayingExpression.create(CubismModel model, CubismExpressionData expressionData).

• CubismModel model: A model that plays back expression motion.
• CubismExpressionData expressionData: The [exp3.asset] data of the expression motion to be played.

The created expression motion is added to the playback list as follows.

    const playingExpression = CubismPlayingExpression.create(
      this._model,
      expressionsList.cubismExpressionObjects[this.currentExpressionIndex]
    );

    if (playingExpression == null) {
      return;
    }

    // Add to PlayingExList.
    playingExpressions.push(palyingExpression);

This is done by CubismExpressionController.startExpression().

Calculate and apply Expression Motion

Expression calculation and application is called from CubismExpressionController.onLateUpdate().

Note: The expression must be applied after ParameterStore.saveParameters().
Expression is a function that applies a new value for expression each frame to the current value.
If applied before ParameterStore.saveParameters(), the result applied by the expression will be saved, so the values of the parameters for the expression will be added or multiplied indefinitely.