Mask preprocessing method (Java)

[Last update: 10/06/2022]

Cubism SDK for Java is currently in alpha version, and the specifications may change in the beta or official version.

In Live2D Cubism SDK for Java (alpha version), in order to maintain rendering speed on smartphones and other devices, the
the “pre-processing method” in which all mask shapes are drawn on a single mask buffer at the beginning of the model drawing process.

In the principle drawing method, the mask shape is drawn each time when a Drawable that requires a mask is drawn (see figure).
This method would result in a relatively expensive process of switching render targets, clearing buffers, etc. each time Drawable needs a mask.
This may cause slow rendering speeds on smartphones and other devices.

 

However, simply preparing masks in advance would require multiple mask buffers, which would squeeze memory.
To solve this problem, the following processing can be performed on a single mask buffer to minimize memory pressure while treating it as if multiple mask buffers were used.

 

Mask Integration

Since all masks are generated in advance, Drawables with the same mask specification can use the same mask image to reduce the number of masks to be generated.

This is done by the CubismClippingManagerAndroid.initialize function in a call to CubismRendererAndroid.initialize function.

 

 

Separation by color information

The mask buffer is actually an RGBA video array, just like a normal texture buffer.
The normal mask process uses only this A channel to apply the mask, but not the RGB channels.
Therefore, by having separate mask data for RGBA, one mask buffer can be treated as four mask buffers.

 

 

Partitioning

When 4 mask images are not enough, the number of masks can be increased by handling the mask buffer in 2, 4, or 9 divisions.
There is also a division by color information so that up to 36 different masks (4 x 9) can be held.

Also, to prevent the mask image from being crushed, the mask is drawn on all Drawable rectangles to which the mask is applied.
This requires range generation, mask generation, and matrix generation with the use of masks.

Rectangle Confirmation

In the first step of mask generation, for each mask, check the rectangle that fits all the mask application destinations.

 

 

Layout determination subject to color separation and divisional separation

Defines the color channel and division position of the mask buffer that belongs to each mask.

 

 

Mask drawing, matrix generation using masks

Prepare transformation matrices for mask generation and mask use based on the rectangle area and the location of the rectangle examined before drawing.

 

 

Dynamic resizing of mask buffer

The OpenGL ES 2.0 renderer provides an API to resize the mask buffer at runtime.
Currently, the mask buffer size is initially set to 256*256 (pixels), but if the mask generation area is to be cut into 9 pieces, the mask shape drawn in a rectangular area of 85*85 (pixels) will be further enlarged and used as the clipping area.
As a result, the edges of the clipping result are blurred or blotchy.
As a solution to this problem, an API is provided to change the size of the mask buffer at program execution time.

For example, if the mask buffer size is 256*256 ⇒ 1024*1024 and the mask generation area is cut into 9 pieces, the mask shape can be drawn in a rectangular area of 341*341, so even if it is enlarged and used as a clipping area, blurred edges and blurring of the clipping result can be resolved. This allows you to eliminate blurred edges and blurring of the clipping result even when using it as a clipping area.

* Increase the size of the mask buffer => The more pixels to be processed, the slower the speed, but the cleaner the drawing result.
* Reduce the size of the mask buffer => Faster speed because fewer pixels are processed, but the drawing result will be dirtier.

 

 

Why pre-processing methods can improve performance

As a situation specific to mobile devices, the processing cost of the Clear instruction and rendering target switching instruction to the GPU may be higher than other instructions.
When drawing in the principle method, these instructions with high processing costs are executed as many times as the number of drawables that require masks.
However, with the pre-processing method, the number of times these instructions are executed can be reduced, resulting in improved performance in smartphones and other devices.

We are actually conducting experiments to ascertain its effectiveness.
For the measurement method and results of this experiment, please refer to "Reasons why the preprocessing method can improve performance" in "(Native) Mask preprocessing method ".

 

Switch to a high-definition method for processing masks

The method of generating a mask each time a drawing is made will affect performance on low-spec devices.

However, this method is suitable when screen quality is more important than performance at runtime, as in the case of final output as video.

Cubism SDK for Java (alpha version) can switch mask processing to a high-definition method.
To switch to the high-definition method, pass true to the following API.

© 2010 - 2022 Live2D Inc.