XR_ANDROID_spatial_object_tracking

Name String

XR_ANDROID_spatial_object_tracking

Extension Type

Instance extension

Registered Extension Number

786

Revision

2

Ratification Status

Not ratified

Extension and Version Dependencies

XR_EXT_spatial_entity

Last Modified Date

2025-06-25

IP Status

No known IP claims.

Contributors

Kyle Chen, Google
Nihav Jain, Google
Levana Chen, Google
Spencer Quin, Google

Overview

This extension builds on XR_EXT_spatial_entity and defines a spatial capability for tracking a set of predefined physical objects without requiring an input dataset from the application. For example, the system can track keyboards, mice, and other objects in the environment, and then report the poses and extents of these objects to the application.

Permissions

Android applications must have the android.permission.SCENE_UNDERSTANDING_COARSE permission listed in their manifest as this extension depends on the geometry of the environment. The android.permission.SCENE_UNDERSTANDING_COARSE permission is considered a dangerous permission.

(protection level: dangerous)

Runtime Support

If the runtime supports object tracking, it must indicate this by enumerating XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID in xrEnumerateSpatialCapabilitiesEXT .

Configuration

The XrSpatialCapabilityConfigurationObjectTrackingANDROID structure is defined as:

typedef struct XrSpatialCapabilityConfigurationObjectTrackingANDROID {
    XrStructureType                               type;
    const void*                                   next;
    XrSpatialCapabilityEXT                        capability;
    uint32_t                                      enabledComponentCount;
    const XrSpatialComponentTypeEXT*              enabledComponents;
    uint32_t                                      activeSemanticLabelCount;
    const XrSpatialObjectSemanticLabelANDROID*    activeSemanticLabels;
} XrSpatialCapabilityConfigurationObjectTrackingANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain.
  • capability is an XrSpatialCapabilityEXT and must be XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID .
  • enabledComponentCount is a uint32_t describing the count of elements in the enabledComponents array. It must be greater than 0.
  • enabledComponents is a pointer to an array of XrSpatialComponentTypeEXT .
  • activeSemanticLabelCount is a uint32_t describing the count of elements in the activeSemanticLabels array. It can be 0, which means all semantic labels are active, or it can be greater than 0, in which case the activeSemanticLabels array is used to determine which semantic labels are active.
  • activeSemanticLabels is a pointer to the array of XrSpatialObjectSemanticLabelANDROID indicating the active objects in tracking. It must be null if activeSemanticLabelCount is 0. And it must be non-null if activeSemanticLabelCount is greater than 0, which means that it is used to determine which semantic labels are active.

Applications can enable the XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID spatial capability by adding a pointer to the XrSpatialCapabilityConfigurationObjectTrackingANDROID structure in XrSpatialContextCreateInfoEXT :: capabilityConfigs .

The runtime must return XR_ERROR_VALIDATION_FAILURE if capability is not XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID .

Applications can specify the desired semantic labels in activeSemanticLabels which they are in favor to track. If there is no semantic labels specified in activeSemanticLabels and activeSemanticLabelCount is 0, all semantic labels are active to be tracked by default.

Valid Usage (Implicit)

Guaranteed Components

A runtime that supports XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID must support the following spatial components as guaranteed components of the entities discovered by this capability and must enumerate them in xrEnumerateSpatialCapabilityComponentTypesEXT -

  • XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT
  • XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID An object tracking capability is meaningless without a semantic label component. While this extension defines the XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID as one of the possible semantic label components for runtimes to support, other extensions may define new component types to provide different classes of semantic labels that describe a tracked object.

Bounded 3D Component

The bounded 3D component provides the center and extents of the object represented by the entity it is on. See Bounded 3D for more details.

Object Semantic Label Component

Component Data

typedef enum XrSpatialObjectSemanticLabelANDROID {
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_ANDROID = 0,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_KEYBOARD_ANDROID = 1,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MOUSE_ANDROID = 2,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_LAPTOP_BASE_ANDROID = 3,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSpatialObjectSemanticLabelANDROID;

The XrSpatialObjectSemanticLabelANDROID enumeration describes a set of semantic labels for a object.

Enum Description

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_ANDROID

The runtime was unable to classify this entity.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_KEYBOARD_ANDROID

The entity is a keyboard.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MOUSE_ANDROID

The entity is a mouse.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_LAPTOP_BASE_ANDROID

The entity is a laptop.

Spatial object orientation

The orientation of the objects is shown in the following image (with xyz = rgb) originating at the bottom center of the objects.

The 3D orientation of the mouse is aligned to the 3D orientation of the supporting plane. The extents enclose the keyboard and mouse completely and the base of the laptop.

XR ANDROID spatial object orientation

Figure 34. Example object orientation.

Component list struct to query data

The XrSpatialComponentObjectSemanticLabelListANDROID structure is defined as:

typedef struct XrSpatialComponentObjectSemanticLabelListANDROID {
    XrStructureType                         type;
    void*                                   next;
    uint32_t                                semanticLabelCount;
    XrSpatialObjectSemanticLabelANDROID*    semanticLabels;
} XrSpatialComponentObjectSemanticLabelListANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • semanticLabelCount is a uint32_t describing the count of elements in the semanticLabels array.
  • semanticLabels is an array of XrSpatialObjectSemanticLabelANDROID .

The application can query the semantic label component of the spatial entities in an XrSpatialSnapshotEXT by adding XR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID in XrSpatialComponentDataQueryConditionEXT :: componentTypes and adding XrSpatialComponentObjectSemanticLabelListANDROID to the next pointer chain of XrSpatialComponentDataQueryResultEXT .

The runtime must return XR_ERROR_VALIDATION_FAILURE from xrQuerySpatialComponentDataEXT if XrSpatialComponentObjectSemanticLabelListANDROID is in the next chain of XrSpatialComponentDataQueryResultEXT :: next but XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID is not included in XrSpatialComponentDataQueryConditionEXT :: componentTypes .

The runtime must return XR_ERROR_SIZE_INSUFFICIENT from xrQuerySpatialComponentDataEXT if semanticLabelCount is less than XrSpatialComponentDataQueryResultEXT :: entityIdCountOutput .

Valid Usage (Implicit)

Configuration

If XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID is enumerated in XrSpatialCapabilityComponentTypesEXT :: componentTypes for some capability, application can enable it by including the enum in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: enabledComponents list of the XrSpatialCapabilityConfigurationBaseHeaderEXT derived structure of the capability that supports this component.

This component does not require any special configuration to be included in the next chain of XrSpatialCapabilityConfigurationBaseHeaderEXT .

Example code

Configure Object Tracking Capability

The following example code demonstrates how to configure object tracking capability when creating a spatial context.

// Check if object tracking capability is supported
uint32_t capabilityCount = 0;
CHK_XR(xrEnumerateSpatialCapabilitiesEXT(instance, systemId, 0, &capabilityCount, nullptr));
std::vector<XrSpatialCapabilityEXT> capabilities(capabilityCount);
CHK_XR(xrEnumerateSpatialCapabilitiesEXT(instance, systemId, capabilityCount, &capabilityCount, capabilities.data()));

if (std::find(capabilities.begin(), capabilities.end(), XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID) == capabilities.end()) {
  return;
}

// Enumerate supported components for object tracking capability
XrSpatialCapabilityComponentTypesEXT objectComponents{
  .type = XR_TYPE_SPATIAL_CAPABILITY_COMPONENT_TYPES_EXT,
};
CHK_XR(xrEnumerateSpatialCapabilityComponentTypesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID, &objectComponents));
std::vector<XrSpatialComponentTypeEXT> objectCapabilityComponents(objectComponents.componentTypeCountOutput);
objectComponents.componentTypeCapacityInput = objectCapabilityComponents.size();
objectComponents.componentTypes = objectCapabilityComponents.data();
CHK_XR(xrEnumerateSpatialCapabilityComponentTypesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID, &objectComponents));

// Create a spatial context
XrSpatialContextEXT spatialContext{XR_NULL_HANDLE};

// Enable the 2 guaranteed components (bounded 3D, semantic label) of the object tracking capability
std::vector<XrSpatialComponentTypeEXT> enabledComponents = {
  XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT,
  XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID,
};

// Specify some semantic labels to be active and tracked
std::vector<XrSpatialObjectSemanticLabelANDROID> activeSemanticLabels =
{
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_KEYBOARD_ANDROID,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MOUSE_ANDROID,
};

XrSpatialCapabilityConfigurationObjectTrackingANDROID objectTrackingConfig{
  .type = XR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_ANDROID,
  .capability = XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID,
  .enabledComponentCount = static_cast<uint32_t>(enabledComponents.size()),
  .enabledComponents = enabledComponents.data(),
  // Specify the count of active semantic labels and the pointer to the array of
  // active semantic labels.
  // If the `activeSemanticLabelCount` is 0 and `activeSemanticLabels` is NULL,
  // all labels will be set to active by default.
  .activeSemanticLabelCount = static_cast<uint32_t>(activeSemanticLabels.size()),
  .activeSemanticLabels = activeSemanticLabels.data(),
};

std::array<XrSpatialCapabilityConfigurationBaseHeaderEXT*, 1> capabilityConfigs = {
  reinterpret_cast<XrSpatialCapabilityConfigurationBaseHeaderEXT*>(&objectTrackingConfig),
};

XrSpatialContextCreateInfoEXT spatialContextCreateInfo{
  .type = XR_TYPE_SPATIAL_CONTEXT_CREATE_INFO_EXT,
  .capabilityConfigCount = capabilityConfigs.size(),
  .capabilityConfigs = capabilityConfigs.data(),
};
XrFutureEXT createContextFuture {XR_NULL_FUTURE_EXT};
CHK_XR(xrCreateSpatialContextAsyncEXT(session, &spatialContextCreateInfo, &createContextFuture));

waitUntilReady(createContextFuture);

XrCreateSpatialContextCompletionEXT completion{
  .type = XR_TYPE_CREATE_SPATIAL_CONTEXT_COMPLETION_EXT,
};
CHK_XR(xrCreateSpatialContextCompleteEXT(session, createContextFuture, &completion));
if (completion.futureResult != XR_SUCCESS) {
  return;
}

spatialContext = completion.spatialContext;

// ...
// Discover entities with the spatial context
// ...

CHK_XR(xrDestroySpatialContextEXT(spatialContext));

Discover Spatial Entities & Query Component Data

The following example code demonstrates how to discover spatial entities for a context configured with XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID and query its component data.

XrFutureEXT future = XR_NULL_FUTURE_EXT;

// We want to look for entities that have the following components.
std::vector<XrSpatialComponentTypeEXT> snapshotComponents = {
  XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT,
  XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID,
};

auto discoverSpatialEntities = [&](XrSpatialContextEXT spatialContext, XrTime time) {
  XrSpatialDiscoverySnapshotCreateInfoEXT snapshotCreateInfo{
    .type = XR_TYPE_SPATIAL_DISCOVERY_SNAPSHOT_CREATE_INFO_EXT,
    .componentTypeCount = static_cast<uint32_t>(snapshotComponents.size()),
    .componentTypes = snapshotComponents.data(),
  };
  CHK_XR(xrCreateSpatialDiscoverySnapshotAsyncEXT(spatialContext, &snapshotCreateInfo, &future));

  waitUntilReady(future);

  XrCreateSpatialDiscoverySnapshotCompletionInfoEXT completionInfo{
    .type = XR_TYPE_CREATE_SPATIAL_DISCOVERY_SNAPSHOT_COMPLETION_INFO_EXT,
    .baseSpace = localSpace,
    .time = time,
    .future = future,
  };

  XrCreateSpatialDiscoverySnapshotCompletionEXT completion{
    .type = XR_TYPE_CREATE_SPATIAL_DISCOVERY_SNAPSHOT_COMPLETION_EXT,
  };
  CHK_XR(xrCreateSpatialDiscoverySnapshotCompleteEXT(spatialContext, &completionInfo, &completion));
  if (completion.futureResult == XR_SUCCESS) {

    // Query for the semantic label component data
    XrSpatialComponentDataQueryConditionEXT queryCond{
      .type = XR_TYPE_SPATIAL_COMPONENT_DATA_QUERY_CONDITION_EXT,
      .componentTypeCount = static_cast<uint32_t>(snapshotComponents.size()),
      .componentTypes = snapshotComponents.data(),
    };

    XrSpatialComponentDataQueryResultEXT queryResult{
      .type = XR_TYPE_SPATIAL_COMPONENT_DATA_QUERY_RESULT_EXT,
    };
    CHK_XR(xrQuerySpatialComponentDataEXT(completion.snapshot, &queryCond, &queryResult));

    std::vector<XrSpatialEntityIdEXT> entityIds(queryResult.entityIdCountOutput);
    std::vector<XrSpatialEntityTrackingStateEXT> entityStates(queryResult.entityIdCountOutput);
    queryResult.entityIdCapacityInput = entityIds.size();
    queryResult.entityIds = entityIds.data();
    queryResult.entityStateCapacityInput = entityStates.size();
    queryResult.entityStates = entityStates.data();

    std::vector<XrBoxf> bounded3D(queryResult.entityIdCountOutput);
    XrSpatialComponentBounded3DListEXT bounded3DList{
      .type = XR_TYPE_SPATIAL_COMPONENT_BOUNDED_3D_LIST_EXT,
      .boundCount = static_cast<uint32_t>(bounded3D.size()),
      .bounds = bounded3D.data(),
    };
    queryResult.next = &bounded3DList;

    std::vector<XrSpatialObjectSemanticLabelANDROID> semanticLabels(queryResult.entityIdCountOutput);
    XrSpatialComponentObjectSemanticLabelListANDROID semanticLabelsList{
      .type = XR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID,
      .semanticLabelCount = static_cast<uint32_t>(semanticLabels.size()),
      .semanticLabels = semanticLabels.data(),
    };
    queryResult.next = &semanticLabelsList;

    CHK_XR(xrQuerySpatialComponentDataEXT(completion.snapshot, &queryCond, &queryResult));

    for (int32_t i = 0; i < queryResult.entityIdCountOutput; ++i) {
      if (entityStates[i] != XR_SPATIAL_ENTITY_TRACKING_STATE_TRACKING_EXT) {
        continue;
      }

      // 3D bounds for entity entityIds[i] is bounded3D[i].extents centered on bounded3D[i].center.

      // semantic label for entity entityIds[i] is semanticLabels[i].

    }

    CHK_XR(xrDestroySpatialSnapshotEXT(completion.snapshot));
  }
};

while (1) {
  // ...
  // For every frame in frame loop
  // ...

  XrFrameState frameState;  // previously returned from xrWaitFrame
  const XrTime time = frameState.predictedDisplayTime;

  // Poll for the XR_TYPE_EVENT_DATA_SPATIAL_DISCOVERY_RECOMMENDED_EXT event
  XrEventDataBuffer event = {
    .type = XR_TYPE_EVENT_DATA_BUFFER,
  };
  XrResult result = xrPollEvent(instance, &event);
  if (result == XR_SUCCESS) {
    if (event.type == XR_TYPE_EVENT_DATA_SPATIAL_DISCOVERY_RECOMMENDED_EXT) {
      const XrEventDataSpatialDiscoveryRecommendedEXT& eventdata =
          *reinterpret_cast<XrEventDataSpatialDiscoveryRecommendedEXT*>(&event);
      // Discover spatial entities for the context that we received the "discovery
      // recommended" event for.
      discoverSpatialEntities(eventdata.spatialContext, time);
    }
  }

  // ...
  // Finish frame loop
  // ...
}

New Structures

New Enums

New Enum Constants

  • XR_ANDROID_SPATIAL_OBJECT_TRACKING_EXTENSION_NAME
  • XR_ANDROID_spatial_object_tracking_SPEC_VERSION
  • Extending XrSpatialCapabilityEXT :

    • XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID
  • Extending XrSpatialComponentTypeEXT :

    • XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID
  • Extending XrStructureType :

    • XR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_ANDROID
    • XR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID

Issues

Version History

  • Revision 1, 2025-06-26 (Kyle Chen)

    • Initial extension description.
  • Revision 2, 2025-12-04 (Kyle Chen)

    • Update the example format to new style.