Eye Tracking Settings (SDK for Cocos Creator)

Updated: 03/14/2023

This section explains how to make the model’s line of sight, etc., follow the mouse cursor.

Summary

To set up eye tracking, the Cubism SDK uses a component called LookAt.

The SDK includes a sample scene called [LookAt] that uses this component, so please check that for reference as well.
Here is a sample that tracks the position of a Node that rotates above the model’s head with the line of sight.
/assets/resources/Samples/Lookat

To set up LookAt, do the following three things.

1. Attach components to manage LookAt
2. Specify parameters to be tracked
3. Set the target of eye tracking

Attach components to manage LookAt

Attach a component called CubismLookController, which manages eye tracking, to the Node that is the root of the model.

CubismLookController has four setting items.

• Blend Mode: This is the setting of how the parameters reflect the values that fluctuate when eye tracking. There are three options that can be set.
  • Multiply: Multiply the currently set value.
  • Additive: Add to the currently set value.
  • Override: Overwrite the currently set value.
• Center: Set the Node to be treated as the center of the coordinates to be tracked.
  The center is the center of the Bounds of the set Node.
  The Center will be set to a Node under [Model]/Drawables/.
  If nothing is set for this item, the center of the Node to which CubismLookController is attached is used.
• Damping: This is the time it takes to follow the target. The smaller the value, the faster the tracking speed.
• Target: Sets the target of eye tracking. Details are described below.

For this example, set Blend Mode to [Override].

Specify parameters to be tracked

Nodes that manage the parameters of the model are located under [Model]/Parameters/.
The name set for this Node is the ID of the parameter.
These are identical to those obtained with CubismModel.parameters.

From among the Nodes for this parameter, attach a component called CubismLookParameter to the one with the ID you want to track.

If a CubismLookParameter is attached to a Node for a parameter, the aforementioned CubismLookController will refer to it and follow it.

CubismLookParameter has two setting items.

• Axis: Specify which axis deformation to treat the set parameters as. For example, if X is specified, it is calculated and set based on the value of the Target’s X axis.
• Factor: Sets the number to multiply the calculated value by. Since the calculation result value ranges from -1 to +1, depending on the parameter, it may be more natural to increase or decrease its range, or invert the + and – values.
  

Set the target of eye tracking

Finally, prepare the target to be followed.

The [Target] of the CubismLookController component should be set to a component that implements the [ICubismLookTarget] interface.

Depending on the target settings, it is possible to set specific conditions, such as the mouse cursor or a Node position as the target for eye tracking, or to track only while dragging.

Create a TypeScript script called “CubismLookTarget” and write the code as follows.
Here, the coordinates of the mouse while dragging are the target of tracking.

import { _decorator, Component, Node, Input, EventTouch, input, math, Vec3, __private, Camera, Vec2, screen } from 'cc';
import ICubismLookTarget from '../../Framework/LookAt/ICubismLookTarget';
const { ccclass, property } = _decorator;

@ccclass('CubismLookTarget')
export class CubismLookTarget extends Component implements ICubismLookTarget {
  readonly [ICubismLookTarget.SYMBOL]: typeof ICubismLookTarget.SYMBOL = ICubismLookTarget.SYMBOL;

  private _position: math.Vec3 = Vec3.ZERO;

  public getPosition(): math.Vec3 {
    return this._position;
  }

  public isActive(): boolean {
    return true;
  }

  protected start() {
    input.on(Input.EventType.TOUCH_MOVE,this.onTouchMove,this);
    input.on(Input.EventType.TOUCH_END,this.onTouchEnd,this);
  }

  public onTouchMove(event: EventTouch) {
    let targetPosition = event.getLocationInView();
    targetPosition.x = targetPosition.x / screen.resolution.width;
    targetPosition.y = targetPosition.y / screen.resolution.height;

    targetPosition = targetPosition.multiplyScalar(2).subtract(Vec2.ONE);

    this._position = new Vec3(targetPosition.x, targetPosition.y, this._position.z);
  }

  public onTouchEnd(event: EventTouch) {
    this._position = Vec3.ZERO;
  }
}

Create an empty Node and attach the above CubismLookTarget to it.

Select the model and drag and drop the Node created above from the Inspector View into the [Target] of the CubismLookController.

This completes the setup.

Running the scene and dragging the Game View with the left mouse button causes the model’s line of sight to follow.