How to Use CubismJavaFramework Directly

Updated: 10/06/2022

The sample project included in the Cubism SDK for Java uses the CubismJavaFramework (under /Framework/ in the SDK package) to handle Cubism models.
When creating a project that handles Cubism models, it is best to use the sample project as a basis, but there are cases where you may want to use the CubismJavaFramework API directly for implementation in an existing application or in your own engine.
However, the included sample project is sophisticated, so it is not easy to understand and isolate the structure just by referring to it.

The following is a small snippet that directly calls CubismFramework to handle models for users like those mentioned above.

CubismFramework Cycle

The procedure for working with CubismFramework is as follows:

1. Initialize CubismFramework
2. Get the path to the model file
3. Load the model
4. Update process
5. Discard the model
6. Exit process of CubismFramework

1. Initialize CubismFramework

The initialization process of CubismFramework is as follows:

CubismFramework is initialized using CubismFramework.startUp().
Cubism SDK for Native takes an allocator (ICubismAllocator) as its first argument, but Cubism SDK for Java does not use this.

public class LAppDelegate{
    public void onSurfaceCreated() {
        ...
        // Initialize Cubism SDK framework
        CubismFramework.initialize();

        // Initialize shaders.
        _view.initializeShader();
    }

    private LAppDelegate() {
        ...
        // Set the level of log output. If LogLevel.VERBOSE, create a detailed log output.
        _cubismOption = new CubismFramework.Option();
        _cubismOption.logFunction = new LAppPal.PrintLogFunction();
        _cubismOption.loggingLevel = LAppDefine.cubismLoggingLevel;

        CubismFramework.cleanUp();
        // Set parameters necessary to initialize CubismJavaFramework.
        CubismFramework.startUp(_cubismOption);
    }
  
    private CubismFramework.Option _cubismOption;
}

2. Get the path to the model file

The set of data for Cubism embedding is described in .model3.json with the relative paths.
You can directly specify MOC3 files, textures, etc. to be loaded, but typically we recommend that you get the path from .model3.json and load them accordingly.

Parsing of .model3.json uses CubismFramework’s CubismModelSettingJson class.

public class LAppModel{
    ...
    private static byte[] createBuffer(final String path) {
        if (LAppDefine.DEBUG_LOG_ENABLE) {
            LAppPal.printLog("create buffer: " + path);
        }
        return LAppPal.loadFileAsBytes(path);
    }
}

// Load model3.json.
final String modelSettingJsonPath = _modelHomeDir + modelSettingJsonName;
byte[] buffer = createBuffer(modelSettingJsonPath);
ICubismModelSetting setting = new CubismModelSettingJson(buffer);

// Get the path of the model described in model3.json.
String moc3Path = _modelSetting.getModelFileName();
moc3Path = _modelHomeDir + moc3Path;

3. Load the model

The model is loaded through the CubismModel interface.

Instances of CubismModel are held by CubismUserModel._model in CubismJavaFramework.
When using this, it is recommended to handle it from a class inheriting from CubismUserModel.
Resource management for textures, motion, facial expression motion, etc. can also be done externally.

CubismUserModelExtend, which inherits from CubismUserModel, is used as an example of how this works.

// Create an instance of the model.
CubismUserModelExtend userModel = new CubismUserModelExtend();

// Load moc3 file.
buffer = CreateBuffer(moc3Path);
userModel.loadModel(mocBuffer);

The path to the MOC3 file to be loaded can be obtained in .model3.json as described above. Similarly, the path to facial expressions, physics, poses, eye blinking, lip-sync, user data, and motion can be obtained from .model3.json.
The SDK sample loads these at the same time as the model is loaded.
Please refer to the following for each loading method.

• About Expression Motion
• Physics
• About Pose
• Automatic eye-blinking
• Lip-sync
• About User Data
• About Motion

4. Update process

The update process of a Cubism model is performed by calling CubismModel.update(), which is the interface of CubismModel, as well as model initialization.
Calling CubismModel.update() updates the Cubism Core and updates vertex information from the values of parameters and parts that have been set so far.

public class CubismUserModelExtend{
    public void update() {
        // Parameter operation.
        _model.setParameterValue(CubismFramework.getIdManager().getId("ParamAngleX"), value);

        // Part opacity operation.
        _model.setPartOpacity(CubismFramework.getIdManager().getId("PartArmL"), opacity);

        // Update model vertex information.
        _model.update();
    }
}

The update process always performs eye tracking, physics, motion playback, etc. before update().

Manipulation of parameter values after CubismModel.update() is not reflected.
CubismModel.update() can be called again later to reflect the changes, but this process is highly burdensome, so we recommend that you combine them into one.

// Reflected in vertices.
_model.setParameterValue(CubismFramework.getIdManager().getId("ParamAngleX"), value);

// Update model vertex information.
_model.update();

// Not reflected in vertices.
_model.setParameterValue(CubismFramework.getIdManager().getId("ParamAngleX"), value);

CubismMotionManager.updateMotion(), which plays the motion, overwrites any ID parameter values used in the motion to be played.
Therefore, any manipulation of parameter values prior to this process will be overwritten by CubismMotionManager.updateMotion().
It is recommended that value operations such as eye tracking and physics calculations be performed first, followed by motion playback processing.

// Reflect the motion being played to the model.
_motionManager.updateMotion(_model, deltaTimeSeconds);

// Value operations such as eye tracking and physics processing

// Update model vertex information.
_model.update();

Also, if not all parameters are used in the motion to be played back, or if the parameter values are not manipulated because playback of the motion is stopped, the result of the value manipulation done in the previous frame will remain.
Therefore, the results of subsequent relative parameter value manipulations may be unintended.
By calling CubismModel.loadParameter() and CubismModel.saveParameters() before and after the process of applying motion to the model, this can reset any subsequent relative value operations.

// Restore the values of all parameters to the state at the time of saving.
_model.loadParameters();

// Reflect the motion being played to the model.
_motionManager.updateMotion(_model, deltaTimeSeconds);

// Save the values of all parameters.
_model.saveParameters();

// Relative value manipulation processing

// Update model vertex information.
_model.update();

Please refer to the respective pages for information on eye tracking, physics calculations, motion playback, and other processes.

• About Expression Motion
• Physics
• About Pose
• Automatic eye-blinking
• Lip-sync
• About User Data
• About Motion

By passing physics operations and vertex information updated by the _model.update function to the renderer, the Cubism model can be drawn on the screen.

5. Discard the model

Cubism SDK for Java typically leaves resource release to garbage collection, but model generation is achieved by accessing Native functions using the Java Native Interface (JNI).
Therefore, a delete operation must be performed for this location.

// Discard model data.
userModel.delete();

6. Exit Process of Cubism Framework

To release common part resources allocated by CubismFramework, call CubismFramework.dispose().
Do not call CubismFramework.dispose() before calling CubismFramework.initialize().

// Destroy CubismFramework.
CubismFramework.dispose();