Microphone, Face, and Eye tracking support on VRM format

LIV will try to animate your VRM when you are using the microphone, lip tracking, or eye tracking.

This is also true for .avatar. If you are using .avatar, please head over to the equivalent .avatar article.

This article does not describe how to artistically model the blendshapes; it only describes which technical names the LIV software expects to function.

For information about the artistic aspect, you should refer to other online resources.

- In VRM, we use A/I/U/E/O blendshapes for the microphone, and we recommend Apple ARKit blendshapes for face tracking

- In .avatar, we use Oculus Lipsync blendshapes for the microphone, and we recommend Vive SRAnipal blendshapes for face tracking

VRM Format

Naming convention

When using the VRM format, we attempt to find out if it is compatible with face tracking (lip and eyes).

We are using naming conventions by looking for specific BlendShape Clip names and animating them.

Microphone

For the microphone, we use the five standard A/I/U/E/O blendshapes. These blendshapes are commonly defined as part of the standard set of BlendShape Clips in the VRM format.

Eye gaze

Eye gaze works using the standard VRM LookAt components.

We have additional option named "Use Experimental Eye Gaze" located in the VR LIV menu. This option is aimed at tweaking eye tracking experience for VR use, as eye gaze in VR may feel different from using traditional webcam-based VTubing software:

- When disabled, it will strictly follow the configuration as defined in the VRM.

- When enabled, it will modify the behaviour of the VRM LookAt Bone Applyer component to more strictly match where you're looking at. This results in stronger eye rotations, which could look artistically inaccurate depending on your avatar.

Lip/Eye tracking blendshapes

We recommend using ARKit-styled naming for VRM compatibility with other software.

ARKit-styled naming is a popular convention in existing VRM-based software, so you will benefit from VRM compatibility with other software.

If you are specifically targeting VR software, there is no clear preference between ARKit-styled naming and SRAnipal-styled naming.

Some facial features available in SRAnipal are not available in ARKit, and inversely.

Some facial features in ARKit will not be animated because the VR hardware does not capture these features.

ARKit-styled naming (recommended for VRM)

If you avatar is designed with ARKit in mind, we will look for blendshapes named after ARKit-styled naming as defined in ARKitBlendShapeLocation, where the first letter is uppercase:

BrowDownLeft BrowDownRight BrowInnerUp BrowOuterUpLeft BrowOuterUpRight
CheekPuff CheekSquintLeft CheekSquintRight EyeBlinkLeft EyeBlinkRight
EyeLookDownLeft EyeLookDownRight EyeLookInLeft EyeLookInRight EyeLookOutLeft
EyeLookOutRight EyeLookUpLeft EyeLookUpRight EyeSquintLeft EyeSquintRight
EyeWideLeft EyeWideRight JawForward JawLeft JawOpen
JawRight MouthClose MouthDimpleLeft MouthDimpleRight MouthFrownLeft
MouthFrownRight MouthFunnel MouthLeft MouthLowerDownLeft MouthLowerDownRight
MouthPressLeft MouthPressRight MouthPucker MouthRight MouthRollLower
MouthRollUpper MouthShrugLower MouthShrugUpper MouthSmileLeft MouthSmileRight
MouthStretchLeft MouthStretchRight MouthUpperUpLeft MouthUpperUpRight NoseSneerLeft
NoseSneerRight TongueOut

You can also look at ARFaceAnchor.BlendShapeLocation documentation for reference but keep in mind the first letter must be uppercase.

All of the BlendShape Clips should be defined. Otherwise, MouthSmileLeft must be defined for your avatar to be recognized as compatible with ARKit-styled naming.

SRAnipal-styled naming

If your avatar does not have ARKit-styled blendshapes, we will also look for SRAnipal LipShape_v2 and EyeShape_v2 blendshapes.

We will look for blendshapes with the following names, case sensitive:

Eye_Left_Blink Eye_Left_Wide Eye_Left_Right Eye_Left_Left Eye_Left_Up
Eye_Left_Down Eye_Right_Blink Eye_Right_Wide Eye_Right_Right Eye_Right_Left
Eye_Right_Up Eye_Right_Down Eye_Frown Eye_Left_Squeeze Eye_Right_Squeeze
Jaw_Right Jaw_Left Jaw_Forward Jaw_Open Mouth_Ape_Shape
Mouth_Upper_Right Mouth_Upper_Left Mouth_Lower_Right Mouth_Lower_Left Mouth_Upper_Overturn
Mouth_Lower_Overturn Mouth_Pout Mouth_Smile_Right Mouth_Smile_Left Mouth_Sad_Right
Mouth_Sad_Left Cheek_Puff_Right Cheek_Puff_Left Cheek_Suck Mouth_Upper_UpRight
Mouth_Upper_UpLeft Mouth_Lower_DownRight Mouth_Lower_DownLeft Mouth_Upper_Inside Mouth_Lower_Inside
Mouth_Lower_Overlay Tongue_LongStep1 Tongue_LongStep2 Tongue_Down Tongue_Up
Tongue_Right Tongue_Left Tongue_Roll Tongue_UpLeft_Morph Tongue_UpRight_Morph
Tongue_DownLeft_Morph Tongue_DownRight_Morph

All of the BlendShape Clips should be defined. Otherwise, Mouth_Smile_Left must be defined for your avatar to be recognized as compatible with SRAnipal-styled naming.

Default behaviour

If your avatar has neither ARKit nor SRAnipal blendshapes, we will use the standard Joy and Sorrow BlendShape Clips as a fallback.

.avatar Format

If you are using .avatar, please head over to the equivalent .avatar article.

Was this article helpful?
0 out of 0 found this helpful