XR_ANDROID_face_tracking OpenXR extension

Name String

XR_ANDROID_face_tracking

Extension Type

Instance extension

Registered Extension Number

459

Revision

1

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-06

IP Status

No known IP claims.

Contributors

Spencer Quin, Google

Jared Finder, Google

Levana Chen, Google

Overview

This extension enables applications to get weights of blend shapes and render facial expressions in XR experiences.

This extension is intended to provide the information needed to create realistic avatars and expressive representations of users in virtual space. The application can check facial calibration activation before getting weights of blend shapes.

Face Tracker

A face tracker is a sensing device that tracks facial expression through user-facing image streams and camera calibration. The main purpose of this extension is to map user facial expressions to their avatars in a virtual scene.

Face tracking data is sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications storing or transferring face tracking data always ask the user for active and specific acceptance to do so.

• Applications will receive XR_ERROR_PERMISSION_INSUFFICIENT when creating an active face tracker until the application has been allowed access to the face tracker.
• When getting face states by using xrGetFaceStateANDROID, XrFaceStateANDROID::isValid won't return XR_TRUE unless the application access has been allowed

Create a face tracker handle

XR_DEFINE_HANDLE(XrFaceTrackerANDROID)

The XrFaceTrackerANDROID handle represents a face tracker for face tracking.

This handle can be used to access face tracking data using other functions in this extension.

The xrCreateFaceTrackerANDROID function is defined as:

XrResult xrCreateFaceTrackerANDROID(
    XrSession                                   session,
    const XrFaceTrackerCreateInfoANDROID*       createInfo,
    XrFaceTrackerANDROID*                       faceTracker);

Parameter Descriptions

• session is an XrSession handle in which the face tracker will be active.
• createInfo is the XrFaceTrackerCreateInfoANDROID used to specify the face tracker.
• faceTracker is the returned XrFaceTrackerANDROID handle.

An application can create an XrFaceTrackerANDROID handle using xrCreateFaceTrackerANDROID function.

If the system does not support face tracking, xrCreateFaceTrackerANDROID will return XR_ERROR_FEATURE_UNSUPPORTED.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrCreateFaceTrackerANDROID
• session must be a valid XrSession handle
• createInfo must be a pointer to a valid XrFaceTrackerCreateInfoANDROID structure
• faceTracker must be a pointer to an XrFaceTrackerANDROID handle

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED

The XrFaceTrackerCreateInfoANDROID structure is described as follows:

typedef struct XrFaceTrackerCreateInfoANDROID {
    XrStructureType    type;
    void*              next;
} XrFaceTrackerCreateInfoANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

The XrFaceTrackerCreateInfoANDROID structure describes the information to create an XrFaceTrackerANDROID handle.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to using XrFaceTrackerCreateInfoANDROID
• type must be XR_TYPE_FACE_TRACKER_CREATE_INFO_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain

The xrDestroyFaceTrackerANDROID function is defined as:

XrResult xrDestroyFaceTrackerANDROID(
    XrFaceTrackerANDROID                        faceTracker);

Parameter Descriptions

• faceTracker is an XrFaceTrackerANDROID previously created by xrCreateFaceTrackerANDROID.

The xrDestroyFaceTrackerANDROID function releases the faceTracker and the underlying resources when the face tracking experience is over.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrDestroyFaceTrackerANDROID
• faceTracker must be a valid XrFaceTrackerANDROID handle

Thread Safety

• Access to faceTracker, and any child handles, must be externally synchronized

Return Codes

Success

• XR_SUCCESS

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_HANDLE_INVALID

Check face calibration

The xrGetFaceCalibrationStateANDROID function is defined as:

XrResult xrGetFaceCalibrationStateANDROID(
    XrFaceTrackerANDROID                        faceTracker,
    XrBool32*                                   faceIsCalibratedOutput);

Parameter Descriptions

• faceTracker is an XrFaceTrackerANDROID previously created by xrCreateFaceTrackerANDROID.
• faceIsCalibratedOutput indicates if the face tracker has been calibrated or not.

An application can check face calibration state using xrGetFaceCalibrationStateANDROID function.

If the system does not support face calibration, xrGetFaceCalibrationStateANDROID will return XR_ERROR_FEATURE_UNSUPPORTED. Otherwise, faceIsCalibratedOutput may be set to XR_TRUE to reflect the face calibration state.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrGetFaceCalibrationStateANDROID
• faceTracker must be a valid XrFaceTrackerANDROID handle
• faceIsCalibratedOutput must be a pointer to an XrBool32 value

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED

Get facial expressions

The xrGetFaceStateANDROID function returns blend shapes of facial expressions at a given time.

XrResult xrGetFaceStateANDROID(
    XrFaceTrackerANDROID                        faceTracker,
    const XrFaceStateGetInfoANDROID*            getInfo,
    XrFaceStateANDROID*                         faceStateOutput);

Parameter Descriptions

• faceTracker is an XrFaceTrackerANDROID previously created by xrCreateFaceTrackerANDROID.
• getInfo is a pointer to XrFaceStateGetInfoANDROID describing information to obtain face expressions.
• faceStateOutput is a pointer to XrFaceStateANDROID receiving the returned face tracking state and facial expressions.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to calling xrGetFaceStateANDROID
• faceTracker must be a valid XrFaceTrackerANDROID handle
• getInfo must be a pointer to a valid XrFaceStateGetInfoANDROID structure
• faceStateOutput must be a pointer to an XrFaceStateANDROID structure

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED
• XR_ERROR_TIME_INVALID

The XrFaceStateGetInfoANDROID structure describes the information to obtain facial expressions.

typedef struct XrFaceStateGetInfoANDROID {
    XrStructureType    type;
    void*              next;
    XrTime             time;
} XrFaceStateGetInfoANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• time is an XrTime at which the facial expressions are requested.

Applications should request a time equal to the predicted display time for the rendered frame.

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to using XrFaceStateGetInfoANDROID
• type must be XR_TYPE_FACE_STATE_GET_INFO_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain

XrFaceStateANDROID structure returns the face tracking state and facial expressions.

typedef struct XrFaceStateANDROID {
    XrStructureType               type;
    void*                         next;
    uint32_t                      parametersCapacityInput;
    uint32_t                      parametersCountOutput;
    float*                        parameters;
    XrFaceTrackingStateANDROID    faceTrackingState;
    XrTime                        sampleTime;
    XrBool32                      isValid;
} XrFaceStateANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• parametersCapacityInput is a uint32_t describing the capacity of the parameters array, or 0 to indicate a request to retrieve the required capacity.
• parametersCountOutput is a uint32_t describing the number of parameters, or the required capacity in the case that parametersCapacityInput is insufficient.
• parameters is a pointer to an application-allocated array of float that will be filled with weights of facial expression blend shapes.
• faceTrackingState is the XrFaceTrackingStateANDROID of validity status of face tracking.
• sampleTime is an XrTime time at which the returned expressions are tracked or extrapolated to. It equals the time at which the expression weights were requested if the extrapolation at the time was successful.
• isValid indicates if the data is valid even if it is not from the current frame.
• See the Buffer Size Parameters section for a detailed description of retrieving the required parameters size.

The application can set parametersCapacityInput to XR_FACE_PARAMETER_COUNT_ANDROID to get facial expressions which are indexed by XrFaceParameterIndicesANDROID.

The returned parameters represent the weights of blend shapes of current facial expressions.

The parameters array updates will be ordered so that the application can index elements using the corresponding facial expression enum (e.g. XrFaceParameterIndicesANDROID).

Valid Usage (Implicit)

• The XR_ANDROID_face_tracking extension must be enabled prior to using XrFaceStateANDROID
• type must be XR_TYPE_FACE_STATE_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain
• If parametersCapacityInput is not 0, parameters must be a pointer to an array of parametersCapacityInput float values
• faceTrackingState must be a valid XrFaceTrackingStateANDROID value

The XrFaceTrackingStateANDROID enumeration identifies the different states of the face tracker.

typedef enum XrFaceTrackingStateANDROID {
    XR_FACE_TRACKING_STATE_PAUSED_ANDROID = 0,
    XR_FACE_TRACKING_STATE_STOPPED_ANDROID = 1,
    XR_FACE_TRACKING_STATE_TRACKING_ANDROID = 2
} XrFaceTrackingStateANDROID;

The enums have the following meanings:

Conventions of blend shapes

This extension defines 63 blend shapes through XR_FACE_PARAMETER_COUNT_ANDROID, for the reduced G-Nome format. Each parameter in this enum is an index into a blend shape array whose values are type of float and the runtime normalizes to 1 - 0.

typedef enum XrFaceParameterIndicesANDROID {
    XR_FACE_PARAMETER_INDICES_BROW_LOWERER_L_ANDROID = 0,
    XR_FACE_PARAMETER_INDICES_BROW_LOWERER_R_ANDROID = 1,
    XR_FACE_PARAMETER_INDICES_CHEEK_PUFF_L_ANDROID = 2,
    XR_FACE_PARAMETER_INDICES_CHEEK_PUFF_R_ANDROID = 3,
    XR_FACE_PARAMETER_INDICES_CHEEK_RAISER_L_ANDROID = 4,
    XR_FACE_PARAMETER_INDICES_CHEEK_RAISER_R_ANDROID = 5,
    XR_FACE_PARAMETER_INDICES_CHEEK_SUCK_L_ANDROID = 6,
    XR_FACE_PARAMETER_INDICES_CHEEK_SUCK_R_ANDROID = 7,
    XR_FACE_PARAMETER_INDICES_CHIN_RAISER_B_ANDROID = 8,
    XR_FACE_PARAMETER_INDICES_CHIN_RAISER_T_ANDROID = 9,
    XR_FACE_PARAMETER_INDICES_DIMPLER_L_ANDROID = 10,
    XR_FACE_PARAMETER_INDICES_DIMPLER_R_ANDROID = 11,
    XR_FACE_PARAMETER_INDICES_EYES_CLOSED_L_ANDROID = 12,
    XR_FACE_PARAMETER_INDICES_EYES_CLOSED_R_ANDROID = 13,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_DOWN_L_ANDROID = 14,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_DOWN_R_ANDROID = 15,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_LEFT_L_ANDROID = 16,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_LEFT_R_ANDROID = 17,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_RIGHT_L_ANDROID = 18,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_RIGHT_R_ANDROID = 19,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_UP_L_ANDROID = 20,
    XR_FACE_PARAMETER_INDICES_EYES_LOOK_UP_R_ANDROID = 21,
    XR_FACE_PARAMETER_INDICES_INNER_BROW_RAISER_L_ANDROID = 22,
    XR_FACE_PARAMETER_INDICES_INNER_BROW_RAISER_R_ANDROID = 23,
    XR_FACE_PARAMETER_INDICES_JAW_DROP_ANDROID = 24,
    XR_FACE_PARAMETER_INDICES_JAW_SIDEWAYS_LEFT_ANDROID = 25,
    XR_FACE_PARAMETER_INDICES_JAW_SIDEWAYS_RIGHT_ANDROID = 26,
    XR_FACE_PARAMETER_INDICES_JAW_THRUST_ANDROID = 27,
    XR_FACE_PARAMETER_INDICES_LID_TIGHTENER_L_ANDROID = 28,
    XR_FACE_PARAMETER_INDICES_LID_TIGHTENER_R_ANDROID = 29,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_DEPRESSOR_L_ANDROID = 30,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_DEPRESSOR_R_ANDROID = 31,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_PULLER_L_ANDROID = 32,
    XR_FACE_PARAMETER_INDICES_LIP_CORNER_PULLER_R_ANDROID = 33,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_LB_ANDROID = 34,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_LT_ANDROID = 35,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_RB_ANDROID = 36,
    XR_FACE_PARAMETER_INDICES_LIP_FUNNELER_RT_ANDROID = 37,
    XR_FACE_PARAMETER_INDICES_LIP_PRESSOR_L_ANDROID = 38,
    XR_FACE_PARAMETER_INDICES_LIP_PRESSOR_R_ANDROID = 39,
    XR_FACE_PARAMETER_INDICES_LIP_PUCKER_L_ANDROID = 40,
    XR_FACE_PARAMETER_INDICES_LIP_PUCKER_R_ANDROID = 41,
    XR_FACE_PARAMETER_INDICES_LIP_STRETCHER_L_ANDROID = 42,
    XR_FACE_PARAMETER_INDICES_LIP_STRETCHER_R_ANDROID = 43,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_LB_ANDROID = 44,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_LT_ANDROID = 45,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_RB_ANDROID = 46,
    XR_FACE_PARAMETER_INDICES_LIP_SUCK_RT_ANDROID = 47,
    XR_FACE_PARAMETER_INDICES_LIP_TIGHTENER_L_ANDROID = 48,
    XR_FACE_PARAMETER_INDICES_LIP_TIGHTENER_R_ANDROID = 49,
    XR_FACE_PARAMETER_INDICES_LIPS_TOWARD_ANDROID = 50,
    XR_FACE_PARAMETER_INDICES_LOWER_LIP_DEPRESSOR_L_ANDROID = 51,
    XR_FACE_PARAMETER_INDICES_LOWER_LIP_DEPRESSOR_R_ANDROID = 52,
    XR_FACE_PARAMETER_INDICES_MOUTH_LEFT_ANDROID = 53,
    XR_FACE_PARAMETER_INDICES_MOUTH_RIGHT_ANDROID = 54,
    XR_FACE_PARAMETER_INDICES_NOSE_WRINKLER_L_ANDROID = 55,
    XR_FACE_PARAMETER_INDICES_NOSE_WRINKLER_R_ANDROID = 56,
    XR_FACE_PARAMETER_INDICES_OUTER_BROW_RAISER_L_ANDROID = 57,
    XR_FACE_PARAMETER_INDICES_OUTER_BROW_RAISER_R_ANDROID = 58,
    XR_FACE_PARAMETER_INDICES_UPPER_LID_RAISER_L_ANDROID = 59,
    XR_FACE_PARAMETER_INDICES_UPPER_LID_RAISER_R_ANDROID = 60,
    XR_FACE_PARAMETER_INDICES_UPPER_LIP_RAISER_L_ANDROID = 61,
    XR_FACE_PARAMETER_INDICES_UPPER_LIP_RAISER_R_ANDROID = 62
} XrFaceParameterIndicesANDROID;

Index	Name	Reference Image
0	BROW_LOWERER_L
1	BROW_LOWERER_R
2	CHEEK_PUFF_L
3	CHEEK_PUFF_R
4	CHEEK_RAISER_L
5	CHEEK_RAISER_R
6	CHEEK_SUCK_L
7	CHEEK_SUCK_R
8	CHIN_RAISER_B
9	CHIN_RAISER_T
10	DIMPLER_L
11	DIMPLER_R
12	EYES_CLOSED_L
13	EYES_CLOSED_R
14	EYES_LOOK_DOWN_L
15	EYES_LOOK_DOWN_R
16	EYES_LOOK_LEFT_L
17	EYES_LOOK_LEFT_R
18	EYES_LOOK_RIGHT_L
19	EYES_LOOK_RIGHT_R
20	EYES_LOOK_UP_L
21	EYES_LOOK_UP_R
22	INNER_BROW_RAISER_L
23	INNER_BROW_RAISER_R
24	JAW_DROP
25	JAW_SIDEWAYS_LEFT
26	JAW_SIDEWAYS_RIGHT
27	JAW_THRUST
28	LID_TIGHTENER_L
29	LID_TIGHTENER_R
30	LIP_CORNER_DEPRESSOR_L
31	LIP_CORNER_DEPRESSOR_R
32	LIP_CORNER_PULLER_L
33	LIP_CORNER_PULLER_R
34	LIP_FUNNELER_LB
35	LIP_FUNNELER_LT
36	LIP_FUNNELER_RB
37	LIP_FUNNELER_RT
38	LIP_PRESSOR_L
39	LIP_PRESSOR_R
40	LIP_PUCKER_L
41	LIP_PUCKER_R
42	LIP_STRETCHER_L
43	LIP_STRETCHER_R
44	LIP_SUCK_LB
45	LIP_SUCK_LT
46	LIP_SUCK_RB
47	LIP_SUCK_RT
48	LIP_TIGHTENER_L
49	LIP_TIGHTENER_R
50	LIPS_TOWARD
51	LOWER_LIP_DEPRESSOR_L
52	LOWER_LIP_DEPRESSOR_R
53	MOUTH_LEFT
54	MOUTH_RIGHT
55	NOSE_WRINKLER_L
56	NOSE_WRINKLER_R
57	OUTER_BROW_RAISER_L
58	OUTER_BROW_RAISER_R
59	UPPER_LID_RAISER_L
60	UPPER_LID_RAISER_R
61	UPPER_LIP_RAISER_L
62	UPPER_LIP_RAISER_R

Example code for face tracking

The following example code demonstrates how to get all weights for facial expression blend shapes.

XrSession session; // previously initialized, for example created at app startup.

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateFaceTrackerANDROID xrCreateFaceTrackerANDROID; // previously initialized
PFN_xrDestroyFaceTrackerANDROID xrDestroyFaceTrackerANDROID; // previously initialized
PFN_xrGetFaceStateANDROID xrGetFaceStateANDROID; // previously initialized
PFN_xrGetFaceCalibrationStateANDROID xrGetFaceCalibrationStateANDROID; // previously initialized

XrFaceTrackerANDROID faceTracker;
XrFaceTrackerCreateInfoANDROID
        createInfo{.type = XR_TYPE_FACE_TRACKER_CREATE_INFO_ANDROID,
                    .next = nullptr};
CHK_XR(xrCreateFaceTrackerANDROID(session, &createInfo, &faceTracker));

// If the system supports face calibration:
bool isCalibrated;
CHK_XR(xrGetFaceCalibrationStateANDROID(faceTracker, &isCalibrated));
if (!isCalibrated) {
    // Redirect the user to system calibration setting.
}

XrFaceStateANDROID faceState;
float faceExpressionParameters[XR_FACE_PARAMETER_COUNT_ANDROID];
faceState.type = XR_TYPE_FACE_STATE_ANDROID;
faceState.next = nullptr;
faceState.parametersCapacityInput = XR_FACE_PARAMETER_COUNT_ANDROID;
faceState.parameters = faceExpressionParameters;

while (1) {
    // ...
    // For every frame in the frame loop
    // ...
    XrFrameState frameState; // previously returned from xrWaitFrame

    XrFaceStateGetInfoANDROID faceGetInfo{
            .type = XR_TYPE_FACE_STATE_GET_INFO_ANDROID,
            .next = nullptr,
            .time = frameState.predictedDisplayTime,
    };

    CHECK_XR(xrGetFaceStateANDROID(faceTracker, &faceGetInfo, &faceState));
    if (faceState.isValid) {
        for (uint32_t i = 0; i < XR_FACE_PARAMETER_COUNT_ANDROID; ++i) {
            // parameters[i] contains a weight of specific blend shape
        }
    }
}

// after usage
CHK_XR(xrDestroyFaceTrackerANDROID(faceTracker));

New Object Types

• XrFaceTrackerANDROID

New Enum Constants

• XR_FACE_PARAMETER_COUNT_ANDROID

XrObjectType enumeration is extended with:

• XR_OBJECT_TYPE_FACE_TRACKER_ANDROID

XrStructureType enumeration is extended with:

• XR_TYPE_FACE_TRACKER_CREATE_INFO_ANDROID
• XR_TYPE_FACE_STATE_GET_INFO_ANDROID
• XR_TYPE_FACE_STATE_ANDROID

New Enums

• XrFaceParameterIndicesANDROID
• XrFaceTrackingStateANDROID

New Structures

• XrFaceTrackerCreateInfoANDROID
• XrFaceStateGetInfoANDROID
• XrFaceStateANDROID

New Functions

• xrCreateFaceTrackerANDROID
• xrDestroyFaceTrackerANDROID
• xrGetFaceStateANDROID
• xrGetFaceCalibrationStateANDROID

Issues

Version History

• Revision 1, 2024-09-05 (Levana Chen)
  • Initial extension description