Mask Preprocessing Methods (Web)

Updated: 09/03/2019

In order to maintain drawing speed on smartphones and other devices, the Live2D Cubism SDK for Web uses the “preprocessing method” in which all mask shapes are drawn in a single mask buffer at the beginning of the model drawing process.

In the general drawing method, the mask shape is drawn each time a Drawable that requires a mask is drawn (see figure).
Although this method can express high-definition masks, it would result in a relatively expensive process of switching render targets, clearing buffers, etc. each time the Drawable needs a mask.
This may cause slow rendering speeds on smartphones and other devices.

However, simply preparing masks in advance requires multiple mask buffers, which can overwhelm memory.
To solve this problem, the following processing can be performed on a single mask buffer to minimize memory usage while treating it as if multiple mask buffers were used.

Mask Integration

Since all masks are generated in advance, Drawables with the same mask specification can use the same mask image to reduce the number of masks to be generated.

This is done, in the CubismRenderer_WebGL.initialize function call, by the CubismClippingManager_WebGL.initialize function.

public initialize(model: CubismModel, drawableCount: number, drawableMasks: Int32Array[], drawableMaskCounts: Int32Array): void
{
    // Register all drawable objects that use clipping masks
    // Clipping masks should normally be used only for a few pieces
    for(let i: number = 0; i < drawableCount; i++)
    {
        if(drawableMaskCounts[i] <= 0)
        {
            // ArtMeshes without clipping masks (often not used)
            this._clippingContextListForDraw.pushBack(null);
            continue;
        }

        // Check if it is the same as an already existing ClipContext
        let clippingContext: CubismClippingContext =
            this.findSameClip(drawableMasks[i], drawableMaskCounts[i]);
        if(clippingContext == null)
        {
            // Generate if the same mask does not exist.
            clippingContext = new CubismClippingContext(this, drawableMasks[i],
                drawableMaskCounts[i]);
            this._clippingContextListForMask.pushBack(clippingContext);
        }

        clippingContext.addClippedDrawable(i);

        this._clippingContextListForDraw.pushBack(clippingContext);
    }
}

Separation by Color Information

The mask buffer is an RGBA video array, just like a normal texture buffer, etc.
The normal mask process uses only this A channel to apply the mask, but not the RGB channels.
Therefore, by having separate mask data for R, G, B, and A, one mask buffer can be treated as four mask buffers.

Separation by Dividing

When 4 mask images are not enough, the number of masks can be increased by handling the mask buffer in 2, 4, or 9 divisions.
Combined with the separation by color information, up to 36 (4 x 9) different masks can be held.

Also, to prevent the mask image from being crushed, the mask is drawn on all Drawable rectangles to which the mask is applied.
This requires range generation as well as matrix generation for the mask generation and the use of masks.

Checking rectangles

In the first step of mask generation, for each mask, check the rectangle that fits all the mask application destinations.

public calcClippedDrawTotalBounds(model: CubismModel, clippingContext: CubismClippingContext): void
{
    // The entire rectangle of the clipped mask (drawable object to be masked)
    let clippedDrawTotalMinX: number = Number.MAX_VALUE;
    let clippedDrawTotalMinY: number = Number.MAX_VALUE;
    let clippedDrawTotalMaxX: number = Number.MIN_VALUE;
    let clippedDrawTotalMaxY: number = Number.MIN_VALUE;

    // Determine if this mask is actually needed
    // If even one “Drawable Object” that uses this clipping is available, a mask must be generated
    const clippedDrawCount: number = clippingContext._clippedDrawableIndexList.length;

    for(let clippedDrawableIndex: number = 0; clippedDrawableIndex < clippedDrawCount;
        clippedDrawableIndex++)
    {
        // Find the rectangle to be drawn for the drawable object that uses the mask
        const drawableIndex: number =
            clippingContext._clippedDrawableIndexList[clippedDrawableIndex];

        const drawableVertexCount: number = model.getDrawableVertexCount(drawableIndex);
        let drawableVertexes: Float32Array = model.getDrawableVertices(drawableIndex);

        let minX: number = Number.MAX_VALUE;
        let minY: number = Number.MAX_VALUE;
        let maxX: number = Number.MIN_VALUE;
        let maxY: number = Number.MIN_VALUE;

        let loop: number = drawableVertexCount * Constant.vertexStep;
        for(let pi: number = Constant.vertexOffset; pi < loop; pi += Constant.vertexStep)
        {
            let x: number = drawableVertexes[pi];
            let y: number = drawableVertexes[pi + 1];

            if(x < minX)
            {
                minX = x;
            }
            if(x > maxX)
            {
                maxX = x;
            }
            if(y < minY)
            {
                minY = y;
            }
            if(y > maxY)
            {
                maxY = y;
            }
        }

        // If a single valid point was not obtained, skip it
        if(minX == Number.MAX_VALUE)
        {
            continue;
        }

        // Reflected in the entire rectangle
        if(minX < clippedDrawTotalMinX)
        {
            clippedDrawTotalMinX = minX;
        }
        if(minY < clippedDrawTotalMinY)
        {
            clippedDrawTotalMinY = minY;
        }
        if(maxX > clippedDrawTotalMaxX)
        {
            clippedDrawTotalMaxX = maxX;
        }
        if(maxY > clippedDrawTotalMaxY)
        {
            clippedDrawTotalMaxY = maxY;
        }

        if(clippedDrawTotalMinX == Number.MAX_VALUE)
        {
            clippingContext._allClippedDrawRect.x = 0.0;
            clippingContext._allClippedDrawRect.y = 0.0;
            clippingContext._allClippedDrawRect.width = 0.0;
            clippingContext._allClippedDrawRect.height = 0.0;
            clippingContext._isUsing = false;
        }
        else
        {
            clippingContext._isUsing = true;
            let w: number = clippedDrawTotalMaxX - clippedDrawTotalMinX;
            let h: number = clippedDrawTotalMaxY - clippedDrawTotalMinY;
            clippingContext._allClippedDrawRect.x = clippedDrawTotalMinX;
            clippingContext._allClippedDrawRect.y = clippedDrawTotalMinY;
            clippingContext._allClippedDrawRect.width = w;
            clippingContext._allClippedDrawRect.height = h;
        }
    }
}

Layout settings with color separation and divisional separation

Defines the color channel and divisional position of the mask buffer that each mask belongs to.

public setupLayoutBounds(usingClipCount: number): void
{
    // Layout masks using a single RenderTexture that is as full as possible
    // If the number of mask groups is 4 or less, 1 mask is placed on each RGBA channel; if the number is between 5 and 6, 2 masks are in each RG channel and 1 mask is in each BA channel

    // Use RGBA in order
    let div: number = usingClipCount / ColorChannelCount; // Basic number of masks to be placed on one channel.
    let mod: number = usingClipCount % ColorChannelCount; // Allocate the remainder, one by one, to this numbered channel.

    // Decimal points are rounded down
    div = ~~div;
    mod = ~~mod;

    // Provide a channel for each RGBA (0: R, 1: G, 2: B, 3: A)
    let curClipIndex: number = 0; // Set in order.

    for(let channelNo: number = 0; channelNo < ColorChannelCount; channelNo++)
    {
        // Number of layouts for this channel
        let layoutCount: number = div + (channelNo < mod ? 1 : 0);

        // Determine the separation method
        if(layoutCount == 0)
        {
            // Do nothing
        }
        else if(layoutCount == 1)
        {
            // Use everything as is
            let clipContext: CubismClippingContext =
                this._clippingContextListForMask.at(curClipIndex++);
            clipContext._layoutChannelNo = channelNo;
            clipContext._layoutBounds.x = 0.0;
            clipContext._layoutBounds.y = 0.0;
            clipContext._layoutBounds.width = 1.0;
            clipContext._layoutBounds.height = 1.0;
        }
        else if(layoutCount == 2)
        {
            for(let i: number = 0; i < layoutCount; i++)
            {
                let xpos: number = i % 2;

                // Decimal points are rounded down
                xpos = ~~xpos;

                let cc: CubismClippingContext =
                    this._clippingContextListForMask.at(curClipIndex++);
                cc._layoutChannelNo = channelNo;

                cc._layoutBounds.x = xpos * 0.5;
                cc._layoutBounds.y = 0.0;
                cc._layoutBounds.width = 0.5;
                cc._layoutBounds.height = 1.0;
                // Divide UV into 2 and use
            }
        }
        else if(layoutCount <= 4)
        {
            // Divide into 4 and use
            for(let i: number = 0; i < layoutCount; i++)
            {
                let xpos: number = i % 2;
                let ypos: number = i / 2;

                // Decimal points are rounded down
                xpos = ~~xpos;
                ypos = ~~ypos;

                let cc = this._clippingContextListForMask.at(curClipIndex++);
                cc._layoutChannelNo = channelNo;

                cc._layoutBounds.x = xpos * 0.5;
                cc._layoutBounds.y = ypos * 0.5;
                cc._layoutBounds.width = 0.5;
                cc._layoutBounds.height = 0.5;
            }
        }
        else if(layoutCount <= 9)
        {
            // Divide into 9 and use
            for(let i: number = 0; i < layoutCount; i++)
            {
                let xpos = i % 3;
                let ypos = i / 3;

                // Decimal points are rounded down
                xpos = ~~xpos;
                ypos = ~~ypos;

                let cc: CubismClippingContext =
                    this._clippingContextListForMask.at(curClipIndex++);
                cc._layoutChannelNo = channelNo;

                cc._layoutBounds.x = xpos / 3.0;
                cc._layoutBounds.y = ypos / 3.0;
                cc._layoutBounds.width = 1.0 / 3.0;
                cc._layoutBounds.height = 1.0 / 3.0;
            }
        }
        else
        {
            CubismLogError("not supported mask count : {0}", layoutCount);
        }
    }
}

Matrix generation for drawing and using masks

Prepare transformation matrices for mask generation and mask use based on the area and the location of the rectangle examined before drawing.

    // --- Actually draw one mask --
    let clipContext: CubismClippingContext = this._clippingContextListForMask.at(clipIndex);
    let allClipedDrawRect: csmRect = clipContext._allClippedDrawRect;   // Enclose the logical coordinates of all drawable objects that use this mask in a rectangle.
    let layoutBoundsOnTex01: csmRect = clipContext._layoutBounds; // Put the mask in this range.

    // Use a rectangle on the model coordinates with appropriate margins.
    const MARGIN: number = 0.05;
    this._tmpBoundsOnModel.setRect(allClipedDrawRect);
    this._tmpBoundsOnModel.expand(allClipedDrawRect.width * MARGIN,
                                  allClipedDrawRect.height * MARGIN);
    //########## Essentially, do not use the entire allocated area, but keep the size to the minimum necessary

    // Calculate the formula for shaders. The following is the case when rotation is not considered
    // movePeriod' = movePeriod * scaleX + offX                  [[ movePeriod' = (movePeriod - tmpBoundsOnModel.movePeriod)*scale + layoutBoundsOnTex01.movePeriod ]]
    const scaleX: number = layoutBoundsOnTex01.width / this._tmpBoundsOnModel.width;
    const scaleY: number = layoutBoundsOnTex01.height / this._tmpBoundsOnModel.height;

    // Obtain the matrix to be used when generating the mask.
    {
        // Obtain the matrix to be passed to the shader. <<<<<<<<<<<<<<<<<<<<<<<< Optimization required (can be simplified by calculating in reverse order).
        this._tmpMatrix.loadIdentity();
        {
            // Translate layout 0..1 to -1..1
            this._tmpMatrix.translateRelative(-1.0, -1.0);
            this._tmpMatrix.scaleRelative(2.0, 2.0);
        }
        {
            // view to layout0..1
            this._tmpMatrix.translateRelative(layoutBoundsOnTex01.x, layoutBoundsOnTex01.y);
            this._tmpMatrix.scaleRelative(scaleX, scaleY);  // new = [translate][scale]
            this._tmpMatrix.translateRelative(-this._tmpBoundsOnModel.x,
                                              -this._tmpBoundsOnModel.y);
            // new = [translate][scale][translate]
        }
        // tmpMatrixForMask is the result of calculation
        this._tmpMatrixForMask.setMatrix(this._tmpMatrix.getArray());
    }

    //--------- Calculate the matrix for mask reference at draw time
    {
        // Obtain the matrix to be passed to the shader. <<<<<<<<<<<<<<<<<<<<<<<< Optimization required (can be simplified by calculating in reverse order).
        this._tmpMatrix.loadIdentity();
        {
            this._tmpMatrix.translateRelative(layoutBoundsOnTex01.x, layoutBoundsOnTex01.y);
            this._tmpMatrix.scaleRelative(scaleX, scaleY);  // new = [translate][scale]
            this._tmpMatrix.translateRelative(-this._tmpBoundsOnModel.x,
                                              -this._tmpBoundsOnModel.y);
            // new = [translate][scale][translate]
        }
        this._tmpMatrixForDraw.setMatrix(this._tmpMatrix.getArray());
    }
    clipContext._matrixForMask.setMatrix(this._tmpMatrixForMask.getArray());
    clipContext._matrixForDraw.setMatrix(this._tmpMatrixForDraw.getArray());

Dynamic Resizing of Mask Buffers

CubismRenderer_WebGL provides an API to resize the mask buffer at runtime.
Currently, the mask buffer size is initially set to 256*256 (pixels), but if the mask generation area is to be cut into 9 pieces, the mask shape drawn in an 85*85 (pixels) rectangle area will be further enlarged and used as the clipping area.

As a result, the edges of the clipping result are blurred or blotchy.
As a solution to this problem, an API is provided to change the size of the mask buffer at program execution time.

For example, if the mask buffer size is 256*256 => 1024*1024 and the mask generation area is cut into 9 pieces, the mask shape can be drawn in a rectangle area of 341*341, so when enlarged and used as a clipping area, it eliminates edge blurring or blotches.

Note: Increase the size of the mask buffer => The more pixels to be processed, the slower the speed, but the cleaner the drawing result.
Note: Reduce the size of the mask buffer => The fewer pixels to be processed, the faster the speed, but the drawing result will be less clean.

public setClippingMaskBufferSize(size: number)
{
    // Destroy and recreate instances to resize the FrameBuffer
    this._clippingManager.release();
    this._clippingManager = void 0;
    this._clippingManager = null;

    this._clippingManager = new CubismClippingManager_WebGL();

    this._clippingManager.setClippingMaskBufferSize(size);

    this._clippingManager.initialize(
        this.getModel(),
        this.getModel().getDrawableCount(),
        this.getModel().getDrawableMasks(),
        this.getModel().getDrawableMaskCounts()
    );
}
Please let us know what you think about this article.