XR_ANDROID_trackables OpenXR extension

Name String

XR_ANDROID_trackables

Extension Type

Instance extension

Registered Extension Number

456

Revision

1

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-30

IP Status

No known IP claims.

Contributors

Spencer Quin, Google

Nihav Jain, Google

John Pursey, Google

Jared Finder, Google

Levana Chen, Google

Kenny Vercaemer, Google

Overview

This extension allows the application to access trackables from the physical environment, and create anchors attached to a trackable.

This extension defines plane trackables. Other extensions may add additional trackable types. For example XR_ANDROID_trackables_object adds object trackables, and XR_ANDROID_depth_texture adds depth buffers that allow raycasting to arbitrary points in the environment.

A trackable is something that is tracked in the physical environment (see XrTrackableTypeANDROID):

  • a plane (e.g. wall, floor, ceiling, table)
  • an object (e.g. keyboard, mouse, laptop)

Create a trackable tracker

An XrTrackableTrackerANDROID is a handle that represents the resources required to discover and update trackables of a given XrTrackableTypeANDROID in the environment.

XR_DEFINE_HANDLE(XrTrackableTrackerANDROID)

The xrCreateTrackableTrackerANDROID function is defined as: 

XrResult xrCreateTrackableTrackerANDROID(
    XrSession                                   session,
    const XrTrackableTrackerCreateInfoANDROID*  createInfo,
    XrTrackableTrackerANDROID*                  trackableTracker);

Parameter Descriptions

The application can use the xrCreateTrackableTrackerANDROID function to create a trackable tracker.

  • XR_ERROR_FEATURE_UNSUPPORTED will be returned if the system does not support trackables of the specified type.
  • XR_ERROR_PERMISSION_INSUFFICIENT will be returned if the required permissions have not been granted to the calling app.

The application can use the returned tracker handle in subsequent API calls. The XrTrackableTrackerANDROID handle must be eventually freed using the xrDestroyTrackableTrackerANDROID function.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_VALIDATION_FAILURE
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_FEATURE_UNSUPPORTED

The XrTrackableTrackerCreateInfoANDROID structure is defined as: 

typedef struct XrTrackableTrackerCreateInfoANDROID {
    XrStructureType           type;
    void*                     next;
    XrTrackableTypeANDROID    trackableType;
} XrTrackableTrackerCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • trackableType is the XrTrackableTypeANDROID that the tracker will track.

The XrTrackableTrackerCreateInfoANDROID structure provides creation options for the XrTrackableTrackerANDROID when passed to xrCreateTrackableTrackerANDROID.

Extensions may define structures that can be attached to next to allow additional configuration for the trackable trackers.

Valid Usage (Implicit)

The XrTrackableTypeANDROID enum is defined as: 

typedef enum XrTrackableTypeANDROID {
    XR_TRACKABLE_TYPE_NOT_VALID_ANDROID = 0,
    XR_TRACKABLE_TYPE_PLANE_ANDROID = 1,
    XR_TRACKABLE_TYPE_DEPTH_ANDROID = 1000463000,
    XR_TRACKABLE_TYPE_OBJECT_ANDROID = 1000466000
} XrTrackableTypeANDROID;

The xrDestroyTrackableTrackerANDROID function is defined as: 

XrResult xrDestroyTrackableTrackerANDROID(
    XrTrackableTrackerANDROID                   trackableTracker);

Parameter Descriptions

The xrDestroyTrackableTrackerANDROID function destroys the trackable tracker.

If there is no other valid XrTrackableTrackerANDROID that was created with the same XrTrackableTypeANDROID, the system may disable the tracking services required for that trackable type to save system resources.

Valid Usage (Implicit)

Thread Safety

  • Access to trackableTracker, and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID

Get all trackables

The XrTrackableANDROID atom is defined as:

XR_DEFINE_ATOM(XrTrackableANDROID)

XrTrackableANDROID is used to represent a single trackable and is valid only within the lifecycle of its associated XrTrackableTrackerANDROID.

The xrGetAllTrackablesANDROID function is defined as: 

XrResult xrGetAllTrackablesANDROID(
    XrTrackableTrackerANDROID                   trackableTracker,
    uint32_t                                    trackableCapacityInput,
    uint32_t*                                   trackableCountOutput,
    XrTrackableANDROID*                         trackables);

Parameter Descriptions

  • trackableTracker is the XrTrackableTrackerANDROID to query.

  • trackableCapacityInput is the capacity of the trackables array, or 0 to indicate a request to retrieve the required capacity.

  • trackableCountOutput is a pointer to the count of trackables written, or a pointer to the required capacity in the case that trackables is insufficient.

  • trackables is a pointer to an array of XrTrackableANDROID. It can be NULL if trackableCapacityInput is 0.

  • See the Buffer Size Parameters section for a detailed description of retrieving the required trackables size.

xrGetAllTrackablesANDROID fills an array of XrTrackableANDROID representing the trackables found in the environment. The XrTrackableTypeANDROID of the returned trackables must match the XrTrackableTypeANDROID of the trackableTracker.

Get trackable plane

The xrGetTrackablePlaneANDROID function is defined as:

XrResult xrGetTrackablePlaneANDROID(
    XrTrackableTrackerANDROID                   trackableTracker,
    const XrTrackableGetInfoANDROID*            getInfo,
    XrTrackablePlaneANDROID*                    planeOutput);

Parameter Descriptions

The xrGetTrackablePlaneANDROID function returns details about the trackable plane, such as its geometry, orientation, and tracking state.

The plane information is resolved and relative to the base space at the time of the call to xrGetTrackablePlaneANDROID using XrTrackableGetInfoANDROID::time, XrTrackableGetInfoANDROID::baseSpace.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_VALIDATION_FAILURE
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_TIME_INVALID

The XrTrackableGetInfoANDROID structure is defined as: 

typedef struct XrTrackableGetInfoANDROID {
    XrStructureType       type;
    void*                 next;
    XrTrackableANDROID    trackable;
    XrSpace               baseSpace;
    XrTime                time;
} XrTrackableGetInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • trackable is the XrTrackableANDROID plane to query.
  • baseSpace the plane pose will be relative to this XrSpace at time.
  • time is the XrTime at which to evaluate the coordinates relative to the baseSpace.

The XrTrackableGetInfoANDROID structure provides query options when passed to xrGetTrackablePlaneANDROID. The trackable must correspond to the trackableTracker used in xrGetTrackablePlaneANDROID.

XR_ERROR_MISMATCHING_TRACKABLE_TYPE_ANDROID will be returned if the trackable type of the trackable is not XR_TRACKABLE_TYPE_PLANE_ANDROID.

Valid Usage (Implicit)

The XrTrackablePlaneANDROID structure is defined as: 

typedef struct XrTrackablePlaneANDROID {
    XrStructureType           type;
    void*                     next;
    XrTrackingStateANDROID    trackingState;
    XrPosef                   centerPose;
    XrExtent2Df               extents;
    XrPlaneTypeANDROID        planeType;
    XrPlaneLabelANDROID       planeLabel;
    XrTrackableANDROID        subsumedByPlane;
    XrTime                    lastUpdatedTime;
    uint32_t                  vertexCapacityInput;
    uint32_t*                 vertexCountOutput;
    XrVector2f*               vertices;
} XrTrackablePlaneANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • trackingState is the XrTrackingStateANDROID of the plane.
  • centerPose is an XrPosef defining the position and orientation of the plane within the reference frame of the corresponding XrTrackableGetInfoANDROID::baseSpace. An identity orientation here represents a coordinate axes with +Y parallel to the plane's normal.
  • extents is the XrExtent2Df dimension of the plane.
  • planeType is the XrPlaneTypeANDROID that the runtime has determined for this plane.
  • planeLabel is the XrPlaneLabelANDROID that the runtime has determined for this plane.
  • subsumedByPlane is the XrTrackableANDROID of the plane that subsumes this plane (XR_NULL_TRACKABLE_ANDROID if none exists).
  • lastUpdatedTime is the XrTime of the last update of the plane.
  • vertexCapacityInput is the capacity of the vertices array, or 0 to indicate a request to retrieve the required capacity.
  • vertexCountOutput is a pointer to the count of vertices written, or a pointer to the required capacity in the case that vertices is insufficient.
  • vertices is a pointer to an array of XrVector2f. It can be NULL if vertexCapacityInput is 0. The vertices are in counter-clockwise order. The polygon may be concave and must not be self-intersecting.
  • See the Buffer Size Parameters section for a detailed description of retrieving the required vertices size.

Valid Usage (Implicit)

The XrTrackingStateANDROID enum describes the tracking state of an XrTrackableANDROID. 

typedef enum XrTrackingStateANDROID {
    XR_TRACKING_STATE_PAUSED_ANDROID = 0,
    XR_TRACKING_STATE_STOPPED_ANDROID = 1,
    XR_TRACKING_STATE_TRACKING_ANDROID = 2
} XrTrackingStateANDROID;

XrTrackingStateANDROID

Description

XR_TRACKING_STATE_PAUSED_ANDROID

Indicates that the trackable or anchor tracking is paused but may be resumed in the future.

XR_TRACKING_STATE_STOPPED_ANDROID

Tracking has stopped on this Trackable and will never be resumed.

XR_TRACKING_STATE_TRACKING_ANDROID

The object is tracked and its pose is current.

The XrPlaneTypeANDROID enum is the type of an XrTrackableANDROID plane. 

typedef enum XrPlaneTypeANDROID {
    XR_PLANE_TYPE_HORIZONTAL_DOWNWARD_FACING_ANDROID = 0,
    XR_PLANE_TYPE_HORIZONTAL_UPWARD_FACING_ANDROID = 1,
    XR_PLANE_TYPE_VERTICAL_ANDROID = 2,
    XR_PLANE_TYPE_ARBITRARY_ANDROID = 3
} XrPlaneTypeANDROID;

The XrPlaneLabelANDROID enum is a label for a XrTrackableANDROID plane. 

typedef enum XrPlaneLabelANDROID {
    XR_PLANE_LABEL_UNKNOWN_ANDROID = 0,
    XR_PLANE_LABEL_WALL_ANDROID = 1,
    XR_PLANE_LABEL_FLOOR_ANDROID = 2,
    XR_PLANE_LABEL_CEILING_ANDROID = 3,
    XR_PLANE_LABEL_TABLE_ANDROID = 4
} XrPlaneLabelANDROID;

Create anchor space

XrResult xrCreateAnchorSpaceANDROID(
    XrSession                                   session,
    const XrAnchorSpaceCreateInfoANDROID*       createInfo,
    XrSpace*                                    anchorOutput);

Parameter Descriptions

  • session is the XrSession that creates the anchor space.
  • createInfo is a pointer to an XrAnchorSpaceCreateInfoANDROID structure containing parameters to be used to create the anchor space.
  • anchorOutput is a pointer to a handle in which the created XrSpace is returned.

At any point of time both the position and direction of the anchor is tracked or untracked. This means that both XR_SPACE_LOCATION_POSITION_TRACKED_BIT and XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT must be set or both must be cleared when the application calls xrLocateSpace or xrLocateSpaces for anchorOutput.

The application must eventually free the returned XrSpace using xrDestroySpace.

  • XR_ERROR_FEATURE_UNSUPPORTED must be returned if the system does not support anchors.
  • XR_ERROR_TRACKABLE_TYPE_NOT_SUPPORTED_ANDROID must be returned if the specific anchor attachment is not supported.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_TRACKABLE_TYPE_NOT_SUPPORTED_ANDROID
  • XR_ERROR_VALIDATION_FAILURE
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_POSE_INVALID
  • XR_ERROR_TIME_INVALID
  • XR_ERROR_OUT_OF_MEMORY

The XrAnchorSpaceCreateInfoANDROID structure is defined as: 

typedef struct XrAnchorSpaceCreateInfoANDROID {
    XrStructureType       type;
    void*                 next;
    XrSpace               space;
    XrTime                time;
    XrPosef               pose;
    XrTrackableANDROID    trackable;
} XrAnchorSpaceCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • space is the XrSpace on which the anchor will be created.
  • time is the XrTime of the anchor creation.
  • pose is the XrPosef of the anchor.
  • trackable is the XrTrackableANDROID on which the anchor will be attached. It may be XR_NULL_TRACKABLE_ANDROID to create an spatial anchor.

Valid Usage (Implicit)

Example code for getting all trackables

The following example code demonstrates how to get all trackables of a given type.

XrSession session; // previously initialized

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateTrackableTrackerANDROID xrCreateTrackableTrackerANDROID; // previously initialized
PFN_xrGetAllTrackablesANDROID xrGetAllTrackablesANDROID; // previously initialized
PFN_xrDestroyTrackableTrackerANDROID xrDestroyTrackableTrackerANDROID; // previously initialized

XrTrackableTrackerCreateInfoANDROID createInfo{XR_TYPE_TRACKABLE_TRACKER_CREATE_INFO_ANDROID};
createInfo.trackableType = XR_TRACKABLE_TYPE_PLANE_ANDROID;
XrTrackableTrackerANDROID planeTrackableTracker;
XrResult result = xrCreateTrackableTrackerANDROID(
  session,
  &createInfo,
  &planeTrackableTracker);
if (result != XR_SUCCESS) { /* Handle failures. */ }

uint32_t trackableCountOutput = 0;
std::vector<XrTrackableANDROID> allPlaneTrackables;

// Query the number of trackables available.
result = xrGetAllTrackablesANDROID(
  planeTrackableTracker,
  0,
  &trackableCountOutput,
  nullptr
);

if (result == XR_SUCCESS) {
  allPlaneTrackables.resize(trackableCountOutput, XR_NULL_HANDLE);

  // Fetch the actual trackable handles in the appropriately resized array.
  result = xrGetAllTrackablesANDROID(
    planeTrackableTracker,
    trackableCountOutput,
    &trackableCountOutput,
    allPlaneTrackables.data());

  if (result == XR_SUCCESS) {
    for (XrTrackableANDROID trackable : allPlaneTrackables) {
      // You now have all trackables of the specified type.
    }
  }
}

// Release trackable tracker.
result = xrDestroyTrackableTrackerANDROID(planeTrackableTracker);

Example code for getting trackable plane

The following example code demonstrates how to get a trackable plane from an existing XrTrackableANDROID, obtained from a hit result XR_ANDROID_raycast or xrGetTrackablesANDROID.

XrTrackableTrackerANDROID planeTracker; // previously created

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrGetTrackablePlaneANDROID xrGetTrackablePlaneANDROID; // previously initialized

XrTime updateTime; // Time used for the current frame's simulation update.
XrSpace appSpace; // Space created for XR_REFERENCE_SPACE_TYPE_LOCAL.
XrTrackableANDROID planeTrackable; // Acquired from a hit result or getTrackables().

XrTrackableGetInfoANDROID planeGetInfo;
planeGetInfo.type = XR_TYPE_TRACKABLE_GET_INFO_ANDROID;
planeGetInfo.next = nullptr;
planeGetInfo.trackable = planeTrackable;
planeGetInfo.space = appSpace;
planeGetInfo.time = updateTime;

XrTrackablePlaneANDROID plane = { XR_TYPE_TRACKABLE_PLANE_ANDROID };
result = xrGetTrackablePlaneANDROID(
  planeTracker,
  &planeGetInfo,
  &plane
);

if (result == XR_SUCCESS) {
  // Plane tracking state, center pose, extents, type now available in plane.
}

Example code for creating anchor space

The following example code demonstrates how to create an anchor space attached to a trackable.

XrSession session; // Created at app startup.
XrTime updateTime; // Time used for the current frame's simulation update.
XrSpace appSpace; // Space created for XR_REFERENCE_SPACE_TYPE_LOCAL.
XrTrackableANDROID planeTrackable; // Acquired from a hit result or getTrackables().

// Create an anchor at (2, 2, 2) world-coordinates.
XrAnchorSpaceCreateInfoANDROID spatialAnchorCreateInfo;
spatialAnchorCreateInfo.type = XR_TYPE_ANCHOR_SPACE_CREATE_INFO_ANDROID;
spatialAnchorCreateInfo.next = nullptr;
spatialAnchorCreateInfo.space = appSpace;
spatialAnchorCreateInfo.time = updateTime;
spatialAnchorCreateInfo.pose = { { 0, 0, 0, 1 }, { 2, 2, 2 } };

XrSpace spatialAnchor = XR_NULL_HANDLE;
XrResult result = xrCreateAnchorSpaceANDROID(
  session,
  &spatialAnchorCreateInfo,
  &spatialAnchor
);

// Create an anchor attached to a trackable.
XrTrackablePlane plane = ...;
XrAnchorSpaceCreateInfoANDROID trackableAnchorCreateInfo;
trackableAnchorCreateInfo.type = XR_TYPE_ANCHOR_SPACE_CREATE_INFO_ANDROID;
trackableAnchorCreateInfo.next = nullptr;
trackableAnchorCreateInfo.space = appState;
trackableAnchorCreateInfo.pose = plane.centerPose;
trackableAnchorCreateInfo.trackable = planeTrackable;

XrSpace trackableAnchor = XR_NULL_HANDLE;
XrResult result = xrCreateAnchorSpaceANDROID(
  session,
  &trackableAnchorCreateInfo,
  &trackableAnchor
);
while (true) {
  // app update loop
  // ...

  // Get the current location of the anchor's space w.r.t the world.
  XrSpaceLocation anchorLocation = { XR_TYPE_SPACE_LOCATION };
  result = xrLocateSpace(trackableAnchor, appSpace, updateTime, &anchorLocation);

  if (anchor.trackingState == XR_TRACKING_STATE_TRACKING_ANDROID) {
    // Update anchor pose.
    doDrawingForAnchor(anchorLocation.pose);
  } else {
    // ...
  }
}

// Cleanup - destroy the space, detatch the anchor so its no longer tracked by the
// runtime and then release all resources held by it.
xrDestroySpace(spatialAnchor);
xrDestroySpace(trackableAnchor);

New Base Types

New Object Types

New Enum Constants

XrStructureType enumeration is extended with:

  • XR_TYPE_TRACKABLE_GET_INFO_ANDROID
  • XR_TYPE_ANCHOR_SPACE_CREATE_INFO_ANDROID
  • XR_TYPE_TRACKABLE_PLANE_ANDROID
  • XR_TYPE_TRACKABLE_TRACKER_CREATE_INFO_ANDROID

XrObjectType enumeration is extended with:

  • XR_OBJECT_TYPE_TRACKABLE_TRACKER_ANDROID

XrResult enumeration is extended with:

  • XR_ERROR_MISMATCHING_TRACKABLE_TYPE_ANDROID
  • XR_ERROR_TRACKABLE_TYPE_NOT_SUPPORTED_ANDROID

New Enums

New Structures

New Functions

Issues

Version History

  • Revision 1, 2024-09-27 (Kenny Vercaemer)
    • Initial extension description.