Lip-sync based on volume of wav files (Web)

[Last update: 05/13/2021]

 

Summary

The Cubism SDK for Web sample uses the lip-sync function to provide real-time lip-sync movement based on the volume of the audio data in the wav file.
This sample is provided with Cubism 4 SDK for Web R3 beta1 or later.

 

Usage rules

Preparation

Describe the wav file associated with the motion in a .json file. Specify the wav file path corresponding to the “Sound” key for each motion.

Place the wav file in the location specified above; if the path to the wav file is incorrect, lip-sync will not be performed.

 

Start of volume acquisition

In the sample, information is acquired from a wav file via the LAppWavFileHandler class.
Executing LAppWavFileHandler::Start() reads audio data from a wav file and initializes the internal state necessary to perform lip-sync.

 

 

Status update and volume acquisition

LAppWavFileHandler::Update() is executed to measure the volume of the part corresponding to the elapsed time. The resulting measured volume can be obtained with LAppWavFileHandler::GetRms(). Set the obtained volume as a lip-sync value to the model with the CubismModel::AddParameterValue function.

In the sample, the acquired volume is multiplied by 0.8 to obtain the lip-sync value. Here, it is also possible to set the lip-sync value to a volume obtained from a sound library or other source.

 

 

Supplement

  • The sample does not have the ability to play audio on the device.
  • LAppWavFileHandler::GetRms() returns the current volume value in the range 0 to 1.
    • The unit for sound volume is RMS (root mean square).
    • Calculates the average of all channel audio.
      • For example, in the case of stereo audio, the average value of the audio including the left and right channels is calculated.

 

Restrictions

The sample supports loading the following wav files. Lip-sync will not be performed if a file that does not support the following formats is loaded.

  • Microsoft WAV (Little Endian format)
  • Linear PCM
    • WAV encoded in μ-raw, ADPCM, etc. are not supported.
  • Number of channels: Mono/Stereo
  • Supported Bit Depths: 8, 16, 24-bit signed integer

© 2010 - 2022 Live2D Inc.