XR_ANDROID_avatar_eyes OpenXR extension

Name String

XR_ANDROID_avatar_eyes

Extension Type

Instance extension

Registered Extension Number

457

Revision

1

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-30

IP Status

No known IP claims.

Contributors

Spencer Quin, Google

Jared Finder, Google

Levana Chen, Google

Overview

This extension enables applications to obtain position and orientation of the user's eyes, as well as eye tracking status.

This extension is intended to make eye pose and status representation for avatars more realistic. For that purpose:

• It allows for non-tracked states such as blinking.
• It allows for monocular or binocular tracking.

This extension should not be used for other eye tracking purposes. For interaction, XR_EXT_eye_gaze_interaction should be used.

Eye Tracker

An eye tracker is a sensing device that tracks eyes and accurately maps where the user is looking. The main purpose of this extension is to map user gazes to their avatars in a virtual scene.

Eye tracking data can be sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications that store or transfer eye tracking data always ask the user for active and specific acceptance to do so.

• An application will receive XR_ERROR_PERMISSION_INSUFFICIENT when attempting to create an active eye tracker until the application has been allowed access to the eye tracker.

Inspect system capability

An application can inspect whether the system supports avatar eyes by chaining an XrSystemAvatarEyesPropertiesANDROID structure to the XrSystemProperties when calling xrGetSystemProperties. If supportsAvatarEyes returns XR_FALSE, then an application will receive XR_ERROR_FEATURE_UNSUPPORTED from xrCreateEyeTrackerANDROID.

typedef struct XrSystemAvatarEyesPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsAvatarEyes;
} XrSystemAvatarEyesPropertiesANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• supportsAvatarEyes is an XrBool32, indicating if the current system supports avatar eyes.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to using XrSystemAvatarEyesPropertiesANDROID
• type must be XR_TYPE_SYSTEM_AVATAR_EYES_PROPERTIES_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain

Create an eye tracker handle

XR_DEFINE_HANDLE(XrEyeTrackerANDROID)

The XrEyeTrackerANDROID handle represents an eye tracker for tracking eyes and accurately maps what the user is looking at.

This handle can be used to access eye tracking data using other functions in this extension.

Eye tracking provides eye pose and status representation in the scene.

The xrCreateEyeTrackerANDROID function is defined as:

XrResult xrCreateEyeTrackerANDROID(
    XrSession                                   session,
    const XrEyeTrackerCreateInfoANDROID*        createInfo,
    XrEyeTrackerANDROID*                        eyeTracker);

Parameter Descriptions

• session is an XrSession handle in which the eye tracking will be active.
• createInfo is the XrEyeTrackerCreateInfoANDROID used to specify the eye tracking.
• eyeTracker is the returned XrEyeTrackerANDROID handle.

An application can create an XrEyeTrackerANDROID handle using xrCreateEyeTrackerANDROID function.

If the system does not support eye tracking, then XR_ERROR_FEATURE_UNSUPPORTED will be returned from xrCreateEyeTrackerANDROID.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to calling xrCreateEyeTrackerANDROID
• session must be a valid XrSession handle
• createInfo must be a pointer to a valid XrEyeTrackerCreateInfoANDROID structure
• eyeTracker must be a pointer to an XrEyeTrackerANDROID handle

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED

The XrEyeTrackerCreateInfoANDROID structure is defined as:

typedef struct XrEyeTrackerCreateInfoANDROID {
    XrStructureType    type;
    void*              next;
} XrEyeTrackerCreateInfoANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

The XrEyeTrackerCreateInfoANDROID structure describes the information to create an XrEyeTrackerANDROID handle.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to using XrEyeTrackerCreateInfoANDROID
• type must be XR_TYPE_EYE_TRACKER_CREATE_INFO_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain

The xrDestroyEyeTrackerANDROID function is defined as:

XrResult xrDestroyEyeTrackerANDROID(
    XrEyeTrackerANDROID                         eyeTracker);

Parameter Descriptions

• eyeTracker is an XrEyeTrackerANDROID previously created by xrCreateEyeTrackerANDROID.

xrDestroyEyeTrackerANDROID function releases the eyeTracker and the underlying resources when finished with eye tracking experiences.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to calling xrDestroyEyeTrackerANDROID
• eyeTracker must be a valid XrEyeTrackerANDROID handle

Thread Safety

• Access to eyeTracker, and any child handles, must be externally synchronized

Return Codes

Success

• XR_SUCCESS

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_HANDLE_INVALID

Getting eyes information

The xrGetEyesInfoANDROID function is defined as:

XrResult xrGetEyesInfoANDROID(
    XrEyeTrackerANDROID                         eyeTracker,
    const XrEyesGetInfoANDROID*                 getInfo,
    XrEyesANDROID*                              infoOutput);

Parameter Descriptions

• eyeTracker is an XrEyeTrackerANDROID previously created by xrCreateEyeTrackerANDROID.
• getInfo is a pointer to XrEyesGetInfoANDROID used to specify what output is required.
• infoOutput is a pointer to XrEyesANDROID that contains the returned eyes information including poses and states.

xrGetEyesInfoANDROID function gets the information for eye states and poses.

The eyes information is resolved and relative to the base space at the time of the call to xrGetEyesInfoANDROID using XrEyesGetInfoANDROID::time, XrEyesGetInfoANDROID::baseSpace.

At any point of time both the position and direction of the eye pose is tracked or untracked. This means that applications can expect that both XR_SPACE_LOCATION_POSITION_TRACKED_BIT and XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT will either be set or cleared on the supplied XrEyesANDROID::eyes, and that XrEyesANDROID::mode will indicate the tracking states.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to calling xrGetEyesInfoANDROID
• eyeTracker must be a valid XrEyeTrackerANDROID handle
• getInfo must be a pointer to a valid XrEyesGetInfoANDROID structure
• infoOutput must be a pointer to an XrEyesANDROID structure

Return Codes

Success

• XR_SUCCESS
• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED
• XR_ERROR_VALIDATION_FAILURE
• XR_ERROR_RUNTIME_FAILURE
• XR_ERROR_HANDLE_INVALID
• XR_ERROR_INSTANCE_LOST
• XR_ERROR_SESSION_LOST
• XR_ERROR_OUT_OF_MEMORY
• XR_ERROR_LIMIT_REACHED
• XR_ERROR_TIME_INVALID

XrEyesGetInfoANDROID structure contains the information required to retrieve eye poses and states.

typedef struct XrEyesGetInfoANDROID {
    XrStructureType    type;
    void*              next;
    XrTime             time;
    XrSpace            baseSpace;
} XrEyesGetInfoANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• time is the XrTime at which to evaluate the coordinates relative to the baseSpace.
• baseSpace the eye pose will be relative to this XrSpace at time.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to using XrEyesGetInfoANDROID
• type must be XR_TYPE_EYES_GET_INFO_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain
• baseSpace must be a valid XrSpace handle

XrEyesANDROID structure contains information on the tracked eyes.

typedef struct XrEyesANDROID {
    XrStructureType             type;
    void*                       next;
    XrEyeANDROID                eyes[XR_EYE_MAX_ANDROID];
    XrEyeTrackingModeANDROID    mode;
} XrEyesANDROID;

Member Descriptions

• type is the XrStructureType of this structure.
• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
• eyes is an array of XrEyeANDROID for the left and right eyes as indexed by XrEyeIndexANDROID.
• mode is the XrEyeTrackingModeANDROID to indicate if the eyes are tracking and which ones.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to using XrEyesANDROID
• type must be XR_TYPE_EYES_ANDROID
• next must be NULL or a valid pointer to the next structure in a structure chain
• Any given element of eyes must be a valid XrEyeANDROID structure
• mode must be a valid XrEyeTrackingModeANDROID value

XrEyeANDROID structure describes the state, position and orientation of an eye.

typedef struct XrEyeANDROID {
    XrEyeStateANDROID    eyeState;
    XrPosef              eyePose;
} XrEyeANDROID;

Member Descriptions

• eyeState is the XrEyeStateANDROID of an eye.
• pose is an XrPosef defining the position and orientation of the origin of an eye within the reference frame of the corresponding XrEyesGetInfoANDROID::baseSpace. An identity orientation here represents a coordinate axes with +Z into the user's eyes, +X to the right and +Y up.

Valid Usage (Implicit)

• The XR_ANDROID_avatar_eyes extension must be enabled prior to using XrEyeANDROID
• eyeState must be a valid XrEyeStateANDROID value

The XrEyeStateANDROID enumeration identifies the different states of tracked eyes.

typedef enum XrEyeStateANDROID {
    XR_EYE_STATE_INVALID_ANDROID = 0,
    XR_EYE_STATE_GAZING_ANDROID = 1,
    XR_EYE_STATE_SHUT_ANDROID = 2
} XrEyeStateANDROID;

The enums have the following meanings:

The XrEyeIndexANDROID enumeration identifies the index of the left or right eye.

typedef enum XrEyeIndexANDROID {
    XR_EYE_INDEX_LEFT_ANDROID = 0,
    XR_EYE_INDEX_RIGHT_ANDROID = 1
} XrEyeIndexANDROID;

The enums have the following meanings:

The XrEyeTrackingModeANDROID enumeration identifies the different modes of tracked eyes.

typedef enum XrEyeTrackingModeANDROID {
    XR_EYE_TRACKING_MODE_NOT_TRACKING_ANDROID = 0,
    XR_EYE_TRACKING_MODE_RIGHT_ANDROID = 1,
    XR_EYE_TRACKING_MODE_LEFT_ANDROID = 2,
    XR_EYE_TRACKING_MODE_BOTH_ANDROID = 3
} XrEyeTrackingModeANDROID;

The enums have the following meanings:

Example code for eye tracking

The following example code demonstrates how to get eye information relative to a view space.

XrSession session; // previously initialized, for example, created at app startup.
XrSpace viewSpace; // space created for XR_REFERENCE_SPACE_TYPE_VIEW.

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateEyeTrackerANDROID xrCreateEyeTrackerANDROID; // previously initialized
PFN_xrDestroyEyeTrackerANDROID xrDestroyEyeTrackerANDROID; // previously initialized
PFN_xrGetEyesInfoANDROID xrGetEyesInfoANDROID; // previously initialized

// This will use the XrSession that is bound to the eye tracker done at time of creation.
XrEyeTrackerANDROID eyeTracker;
XrEyeTrackerCreateInfoANDROID createInfo{
    .type = XR_TYPE_EYE_TRACKER_CREATE_INFO_ANDROID,
    .next = nullptr};
CHK_XR(xrCreateEyeTrackerANDROID(session, &createInfo, &eyeTracker));

while (1) {
    // ...
    // For every frame in frame loop
    // ...

    XrFrameState frameState;  // previously returned from xrWaitFrame
    const XrTime time = frameState.predictedDisplayTime;
    XrEyesANDROID eyesInfo{.type = XR_TYPE_EYES_ANDROID,
                           .next = nullptr,
                           .mode = XR_EYE_TRACKING_MODE_BOTH_ANDROID};
    XrEyesGetInfoANDROID eyesGetInfo{.type = XR_TYPE_EYES_GET_INFO_ANDROID,
                                     .next = nullptr,
                                     .time = time,
                                     .baseSpace = viewSpace};
    CHK_XR(xrGetEyesInfoANDROID(eyeTracker, &eyesGetInfo, &eyesInfo));

    // eyes tracking information is now available:
    // drawLeftEye(eyesInfo.eyes[XR_EYE_INDEX_LEFT_ANDROID].eyePose);
    // drawRightEye(eyesInfo.eyes[XR_EYE_INDEX_RIGHT_ANDROID].eyePose);

    // ...
    // Finish frame loop
    // ...
}

// after usage
CHK_XR(xrDestroyEyeTrackerANDROID(eyeTracker));

New Object Types

• XrEyeTrackerANDROID

New Enum Constants

• XR_EYE_MAX_ANDROID

XrObjectType enumeration is extended with:

• XR_OBJECT_TYPE_EYE_TRACKER_ANDROID

XrStructureType enumeration is extended with:

• XR_TYPE_EYES_ANDROID
• XR_TYPE_EYE_TRACKER_CREATE_INFO_ANDROID
• XR_TYPE_EYES_GET_INFO_ANDROID
• XR_TYPE_SYSTEM_AVATAR_EYES_PROPERTIES_ANDROID

New Enums

• XrEyeIndexANDROID
• XrEyeStateANDROID
• XrEyeTrackingModeANDROID

New Structures

• XrEyeANDROID
• XrEyesANDROID
• XrEyesGetInfoANDROID
• XrEyeTrackerCreateInfoANDROID
• XrSystemAvatarEyesPropertiesANDROID

New Functions

• xrCreateEyeTrackerANDROID
• xrDestroyEyeTrackerANDROID
• xrGetEyesInfoANDROID

Issues

Version History

• Revision 1, 2024-09-04 (Levana Chen)
  • Initial extension description