How to Use CubismNativeFramework Directly

Updated: 01/30/2020

The sample project included in the Cubism SDK for Native uses the CubismNativeFramework (under /Framework/ in the SDK package) to handle Cubism models.
When creating a project that handles Cubism models, it is best to use the sample project as a basis, but there are cases where you may want to use the CubismNativeFramework API directly for implementation in an existing application or in your own engine.
However, the included sample project is sophisticated, so it is not easy to understand and isolate the structure just by referring to it.

The following is a small snippet that directly calls CubismFramework to handle models for users like those mentioned above.

CubismFramework Cycle

The procedure for working with CubismFramework is as follows:

1. Initialize CubismFramework
2. Get the path to the model file
3. Load the model
4. Update process
5. Discard the model
6. Exit process of CubismFramework

Initialize CubismFramework

The initialization process of CubismFramework is as follows:

CubismFramework allocates memory using the allocator passed as the first argument to CubismFramework::StartUp().
Cubism SDK for Native includes allocator samples in CubismNativeSamples.
The allocator used in CubismFramework also requires an implementation that specifies and allocates an alignment.
Therefore, we recommend that you use the sample because it can be incorporated quickly. However, if you want to customize the memory allocation for your own requirements, you can replace it with your own implementation or modify the sample to incorporate your changes.

For specific implementation of allocators, please click here, and for notes on creating custom memory allocators, please click here.

    // Set logging and other options.
    CubismFramework::Option _cubismOption;
    
    // Allocator.
    LAppAllocator _cubismAllocator;
    
    // Message output function.
    static void PrintMessage(const Csm::csmChar* message);

    // Set the level of log output. If LogLevel_Verbose, create a detailed log output.
    _cubismOption.LoggingLevel = CubismFramework::Option::LogLevel_Verbose;
    _cubismOption.LogFunction = PrintMessage;
    
    // Sets the parameters required to initialize CubismNativeFramework.
    CubismFramework::StartUp(&_cubismAllocator, &_cubismOption);
    
    // Initialize CubismFramework.
    CubismFramework::Initialize();

Get the path to the model file

The set of data for Cubism embedding is described in model3.json with the relative paths.
You can directly specify .moc3 files, textures, etc. to be loaded, but typically we recommend that you get the path from model3.json and load them accordingly.

Parsing of model3.json uses CubismFramework’s CubismModelSettingJson class.

    csmByte* CreateBuffer(const csmChar* path, csmSizeInt* size)
    {
        if (DebugLogEnable)
        {
            LAppPal::PrintLog("[APP]create buffer: %s ", path);
        }
        return LAppPal::LoadFileAsBytes(path, size);
    }

    void DeleteBuffer(csmByte* buffer, const csmChar* path = "")
    {
        if (DebugLogEnable)
        {
            LAppPal::PrintLog("[APP]delete buffer: %s", path);
        }
        LAppPal::ReleaseBytes(buffer);
    }

    // Load model3.json.
    csmSizeInt size;
    const csmString modelSettingJsonPath = _modelHomeDir + modelSettingJsonName;
    csmByte* buffer = CreateBuffer(modelSettingJsonPath, &size);
    ICubismModelSetting* setting = new CubismModelSettingJson(buffer, size);
    DeleteBuffer(buffer, modelSettingJsonPath.GetRawString());
    
    // Get the path of the model described in model3.json.
    csmString moc3Path = _modelSetting->GetModelFileName();
    moc3Path = _modelHomeDir + moc3Path;

Load the model

The model is loaded through the CubismModel interface.

Instances of CubismModel are held by CubismUserModel::_model in CubismNativeFramework.
If you use this, we recommend that you handle it from a class inheriting from CubismUserModel.

Resource management for textures, motion, facial expression motion, etc. can also be done externally.

CubismUserModelExtend, which inherits from CubismUserModel, is used as an example of how this works.

    // Create an instance of the model.
    CubismUserModelExtend* userModel = new CubismUserModelExtend();
	
    // Load moc3 file.
    buffer = CreateBuffer(moc3Path.GetRawString(), &size);
    userModel->LoadModel(mocBuffer, mocSize);
    DeleteBuffer(buffer, moc3Path.GetRawString());

The path to the moc3 file to be loaded can be obtained in model3.json as described above. Similarly, the path to facial expressions, physics, poses, eye blinking, lip-sync, user data, and motion can be obtained from model3.json.
The SDK sample loads these at the same time as the model is loaded.
Please refer to the following for each loading method.

• Facial expression
• Physics
• Pose
• Eye blinking
• Lip-sync
• User Data
• Motion

Update process

The update process of a Cubism model is performed by calling CubismModel::Update(), which is the interface of CubismModel, as well as model initialization.
Calling CubismModel::Update() updates the Cubism Core and updates vertex information from the values of parameters and parts that have been set so far.

    void CubismUserModelExtend::Update()
    {
        // Parameter operation.
        _model->SetParameterValue(CubismFramework::GetIdManager()->GetId("ParamAngleX"), value);
        
        // Part opacity operation.
        _model->SetPartOpacity(CubismFramework::GetIdManager()->GetId("PartArmL"), opacity);
        
        // Update model vertex information.
        _model->Update();
    }

The update process always performs eye tracking, physics, motion playback, etc. before Update().

Manipulation of parameter values after CubismModel::Update() is not reflected.
CubismModel::Update() can be called again later to reflect the change, but this process is highly burdensome, so we recommend that you combine them into one.

    // Reflected in vertices.
    _model->SetParameterValue(CubismFramework::GetIdManager()->GetId("ParamAngleX"), value);
    
    // Update model vertex information.
    _model->Update();
    
    // Not reflected in vertices.
    _model->SetParameterValue(CubismFramework::GetIdManager()->GetId("ParamAngleX"), value);

Cubism MotionManager::UpdateMotion(), which plays the motion, overwrites any ID parameter values used in the motion to be played.
Therefore, any manipulation of parameter values prior to this process will be overwritten by Cubism MotionManager::UpdateMotion().
It is recommended that value operations such as eye tracking and physics calculations be performed first, followed by motion playback processing.

    // Reflect the motion being played to the model.
    _motionManager->UpdateMotion(_model, deltaTimeSeconds);
    
    // Value operations such as eye tracking and physics processing
    
    // Update model vertex information.
    _model->Update();

Also, if not all parameters are used in the motion to be played back, or if the parameter values are not manipulated because playback of the motion is stopped, the result of the value manipulation done in the previous frame will remain.
Therefore, the results of subsequent relative parameter value manipulations may be unintended.
By calling CubismModel::LoadParameter() and CubismModel::SaveParameter() before and after the process of applying motion to the model, this can reset any subsequent relative value operations.

    // Restore the values of all parameters to the state at the time of saving.
    _model->LoadParameters();
    
    // Reflect the motion being played to the model.
    _motionManager->UpdateMotion(_model, deltaTimeSeconds);
    
    // Save the values of all parameters.
    _model->SaveParameters();
    
    
    // Relative value manipulation processing
    
    
    // Update model vertex information.
    _model->Update();

Please refer to the respective pages for information on eye tracking, physics calculations, motion playback, and other processes.

• Facial expression
• Physics
• Pose
• Eye blinking
• Lip-sync
• User data
• Motion

By passing this updated vertex information, etc. to the renderer, the Cubism model can be drawn on the screen.

Discard the model

To destroy the model, destroy the instance of the generated CubismUserModel derived class.
This destroys the motion, facial expressions, physics, and other information held by this model from the destructor of the CubismUserModel.

    // Discard model data.
    delete userModel;

Exit process of CubismFramework

To release common part resources allocated by CubismFramework, call CubismFramework::Dispose().
Do not call CubismFramework::Dispose() before calling CubismFramework::Initialize().

Also, when calling CubismFramework::Dispose(), all models must be destroyed beforehand.
This is because the process of destroying the model uses the allocator Deallocate().

    // Destroy CubismFramework.
    CubismFramework::Dispose();