The Android XR SDK is now available in Developer Preview. We want your feedback! Visit our support page to reach out.

Incorporate face tracking in your app with ARCore for Jetpack XR

After the user grants permission for face tracking, your app can retrieve face shape information through ARCore for Jetpack XR. Face shape information can help your app display the user in the virtual world, for example, for a virtual glasses try-on.

Create an ARCore for Jetpack XR session

Obtain head pose information through an ARCore for Jetpack XR session. See Understand a Session's lifecycle to obtain a Session.

Configure the session

Face tracking is not enabled by default on XR sessions. To enable face tracking, configure the session and set the FaceTrackingMode.USER mode:

val newConfig = session.config.copy(
    faceTracking = Config.FaceTrackingMode.USER,
)
when (val result = session.configure(newConfig)) {
    is SessionConfigureSuccess -> TODO(/* Success! */)
    is SessionConfigureConfigurationNotSupported ->
        TODO(/* Some combinations of configurations are not valid. Handle this failure case. */)
    else ->
        TODO(/* The session could not be configured. See SessionConfigureResult for possible causes. */)
}FaceTracking.kt

Retrieve face data

Use Face.getUserFace(session) to retrieve user face data, which contains the following:

Face blendshape values: A face's possible expressions are a combination of 68 blend shape values. Each blend shape value represents a facial movement or deformation of the face, and the value indicates its intensity, ranging from 0.0 to 1.0. For an explanation of each blend shape type, see Understand face blendshape types.
Confidence values for regions: The face has three regions. Confidence values indicate the degree of certainty of accuracy for the given poses, ranging from 0.0 to 1.0, where 1.0 indicates the highest confidence.

val face = Face.getUserFace(session) ?: return
face.state.collect { state ->
    if (state.trackingState != TrackingState.TRACKING) return@collect

    val confidence = state.getConfidence(FaceConfidenceRegion.FACE_CONFIDENCE_REGION_LOWER)
    val blendShapeValue = state.blendShapes[FaceBlendShapeType.FACE_BLEND_SHAPE_TYPE_LIPS_TOWARD]
}FaceTracking.kt

Understand face blendshape types

The following tables list each type of face blend shape:

Upper region blendshapes

Name	Reference Images
`BROW_LOWERER_L`	`BROW_LOWERER_L = 0.0`	`BROW_LOWERER_L = 1.0`
`BROW_LOWERER_R`	`BROW_LOWERER_R = 0.0`	`BROW_LOWERER_R = 1.0`
`EYES_CLOSED_L`	`EYES_CLOSED_L = 0.0`	`EYES_CLOSED_L = 1.0`
`EYES_CLOSED_R`	`EYES_CLOSED_R = 0.0`	`EYES_CLOSED_R = 1.0`
`EYES_LOOK_DOWN_L`	`EYES_LOOK_DOWN_L = 0.0`	`EYES_LOOK_DOWN_L = 1.0`
`EYES_LOOK_DOWN_R`	`EYES_LOOK_DOWN_R = 0.0`	`EYES_LOOK_DOWN_R = 1.0`
`EYES_LOOK_LEFT_L`	`EYES_LOOK_LEFT_L = 0.0`	`EYES_LOOK_LEFT_L = 1.0`
`EYES_LOOK_LEFT_R`	`EYES_LOOK_LEFT_R = 0.0`	`EYES_LOOK_LEFT_R = 1.0`
`EYES_LOOK_RIGHT_L`	`EYES_LOOK_RIGHT_L = 0.0`	`EYES_LOOK_RIGHT_L = 1.0`
`EYES_LOOK_RIGHT_R`	`EYES_LOOK_RIGHT_R = 0.0`	`EYES_LOOK_RIGHT_R = 1.0`
`EYES_LOOK_UP_L`	`EYES_LOOK_UP_L = 0.0`	`EYES_LOOK_UP_L = 1.0`
`EYES_LOOK_UP_R`	`EYES_LOOK_UP_R = 0.0`	`EYES_LOOK_UP_R = 1.0`
`INNER_BROW_RAISER_L`	`INNER_BROW_RAISER_L = 0.0`	`INNER_BROW_RAISER_L = 1.0`
`INNER_BROW_RAISER_R`	`INNER_BROW_RAISER_R = 0.0`	`INNER_BROW_RAISER_R = 1.0`
`LID_TIGHTENER_L`	`LID_TIGHTENER_L = 0.0`	`LID_TIGHTENER_L = 1.0`
`LID_TIGHTENER_R`	`LID_TIGHTENER_R = 0.0`	`LID_TIGHTENER_R = 1.0`
`OUTER_BROW_RAISER_L`	`OUTER_BROW_RAISER_L = 0.0`	`OUTER_BROW_RAISER_L = 1.0`
`OUTER_BROW_RAISER_R`	`OUTER_BROW_RAISER_R = 0.0`	`OUTER_BROW_RAISER_R = 1.0`
`UPPER_LID_RAISER_L`	`UPPER_LID_RAISER_L = 0.0`	`UPPER_LID_RAISER_L = 1.0`
`UPPER_LID_RAISER_R`	`UPPER_LID_RAISER_R = 0.0`	`UPPER_LID_RAISER_R = 1.0`

Lower region blendshapes

Name	Reference Images
`CHEEK_PUFF_L`	`CHEEK_PUFF_L = 0.0`	`CHEEK_PUFF_L = 1.0`
`CHEEK_PUFF_R`	`CHEEK_PUFF_R = 0.0`	`CHEEK_PUFF_R = 1.0`
`CHEEK_RAISER_L`	`CHEEK_RAISER_L = 0.0`	`CHEEK_RAISER_L = 1.0`
`CHEEK_RAISER_R`	`CHEEK_RAISER_R = 0.0`	`CHEEK_RAISER_R = 1.0`
`CHEEK_SUCK_L`	`CHEEK_SUCK_L = 0.0`	`CHEEK_SUCK_L = 1.0`
`CHEEK_SUCK_R`	`CHEEK_SUCK_R = 0.0`	`CHEEK_SUCK_R = 1.0`
`CHIN_RAISER_B`	`CHIN_RAISER_B = 0.0`	`CHIN_RAISER_B = 1.0`
`CHIN_RAISER_T`	`CHIN_RAISER_T = 0.0`	`CHIN_RAISER_T = 1.0`
`DIMPLER_L`	`DIMPLER_L = 0.0`	`DIMPLER_L = 1.0`
`DIMPLER_R`	`DIMPLER_R = 0.0`	`DIMPLER_R = 1.0`
`JAW_DROP`	`JAW_DROP = 0.0`	`JAW_DROP = 1.0`
`JAW_SIDEWAYS_LEFT`	`JAW_SIDEWAYS_LEFT = 0.0`	`JAW_SIDEWAYS_LEFT = 1.0`
`JAW_SIDEWAYS_RIGHT`	`JAW_SIDEWAYS_RIGHT = 0.0`	`JAW_SIDEWAYS_RIGHT = 1.0`
`JAW_THRUST`	`JAW_THRUST = 0.0`	`JAW_THRUST = 1.0`
`LIP_CORNER_DEPRESSOR_L`	`LIP_CORNER_DEPRESSOR_L = 0.0`	`LIP_CORNER_DEPRESSOR_L = 1.0`
`LIP_CORNER_DEPRESSOR_R`	`LIP_CORNER_DEPRESSOR_R = 0.0`	`LIP_CORNER_DEPRESSOR_R = 1.0`
`LIP_CORNER_PULLER_L`	`LIP_CORNER_PULLER_L = 0.0`	`LIP_CORNER_PULLER_L = 1.0`
`LIP_CORNER_PULLER_R`	`LIP_CORNER_PULLER_R = 0.0`	`LIP_CORNER_PULLER_R = 1.0`
`LIP_FUNNELER_LB`	`LIP_FUNNELER_LB = 0.0`	`LIP_FUNNELER_LB = 1.0`
`LIP_FUNNELER_LT`	`LIP_FUNNELER_LT = 0.0`	`LIP_FUNNELER_LT = 1.0`
`LIP_FUNNELER_RB`	`LIP_FUNNELER_RB = 0.0`	`LIP_FUNNELER_RB = 1.0`
`LIP_FUNNELER_RT`	`LIP_FUNNELER_RT = 0.0`	`LIP_FUNNELER_RT = 1.0`
`LIP_PRESSOR_L`	`LIP_PRESSOR_L = 0.0`	`LIP_PRESSOR_L = 1.0`
`LIP_PRESSOR_R`	`LIP_PRESSOR_R = 0.0`	`LIP_PRESSOR_R = 1.0`
`LIP_PUCKER_L`	`LIP_PUCKER_L = 0.0`	`LIP_PUCKER_L = 1.0`
`LIP_PUCKER_R`	`LIP_PUCKER_R = 0.0`	`LIP_PUCKER_R = 1.0`
`LIP_STRETCHER_L`	`LIP_STRETCHER_L = 0.0`	`LIP_STRETCHER_L = 1.0`
`LIP_STRETCHER_R`	`LIP_STRETCHER_R = 0.0`	`LIP_STRETCHER_R = 1.0`
`LIP_SUCK_LB`	`LIP_SUCK_LB = 0.0`	`LIP_SUCK_LB = 1.0`
`LIP_SUCK_LT`	`LIP_SUCK_LT = 0.0`	`LIP_SUCK_LT = 1.0`
`LIP_SUCK_RB`	`LIP_SUCK_RB = 0.0`	`LIP_SUCK_RB = 1.0`
`LIP_SUCK_RT`	`LIP_SUCK_RT = 0.0`	`LIP_SUCK_RT = 1.0`
`LIP_TIGHTENER_L`	`LIP_TIGHTENER_L = 0.0`	`LIP_TIGHTENER_L = 1.0`
`LIP_TIGHTENER_R`	`LIP_TIGHTENER_R = 0.0`	`LIP_TIGHTENER_R = 1.0`
`LIPS_TOWARD`	`LIPS_TOWARD = 0.0`	`JAW_DROP = 1.0 and LIPS_TOWARD = 1.0`
`LOWER_LIP_DEPRESSOR_L`	`LOWER_LIP_DEPRESSOR_L = 0.0`	`LOWER_LIP_DEPRESSOR_L = 1.0`
`LOWER_LIP_DEPRESSOR_R`	`LOWER_LIP_DEPRESSOR_R = 0.0`	`LOWER_LIP_DEPRESSOR_R = 1.0`
`MOUTH_LEFT`	`MOUTH_LEFT = 0.0`	`MOUTH_LEFT = 1.0`
`MOUTH_RIGHT`	`MOUTH_RIGHT = 0.0`	`MOUTH_RIGHT = 1.0`
`NOSE_WRINKLER_L`	`NOSE_WRINKLER_L = 0.0`	`NOSE_WRINKLER_L = 1.0`
`NOSE_WRINKLER_R`	`NOSE_WRINKLER_R = 0.0`	`NOSE_WRINKLER_R = 1.0`
`UPPER_LIP_RAISER_L`	`UPPER_LIP_RAISER_L = 0.0`	`UPPER_LIP_RAISER_L = 1.0`
`UPPER_LIP_RAISER_R`	`UPPER_LIP_RAISER_R = 0.0`	`UPPER_LIP_RAISER_R = 1.0`
`TONGUE_OUT`	TONGUE_OUT = 0.0	TONGUE_OUT = 1.0
`TONGUE_LEFT`	TONGUE_LEFT = 0.0	TONGUE_LEFT = 1.0
`TONGUE_RIGHT`	TONGUE_RIGHT = 0.0	TONGUE_RIGHT = 1.0
`TONGUE_UP`	TONGUE_UP = 0.0	TONGUE_UP = 1.0
`TONGUE_DOWN`	TONGUE_DOWN = 0.0	TONGUE_DOWN = 1.0