XR_ANDROID_spatial_object_tracking
Name String
XR_ANDROID_spatial_object_tracking
Extension Type
Instance extension
Registered Extension Number
786
Revision
2
Ratification Status
Not ratified
Extension and Version Dependencies
XR_EXT_spatial_entity
Last Modified Date
2025-06-25
IP Status
No known IP claims.
Contributors
Kyle Chen, Google
Nihav Jain, Google
Levana Chen, Google
Spencer Quin, Google
Overview
This extension builds on XR_EXT_spatial_entity and defines a spatial capability for tracking a set of predefined physical objects without requiring an input dataset from the application. For example, the system can track keyboards, mice, and other objects in the environment, and then report the poses and extents of these objects to the application.
Permissions
Android applications must have the android.permission.SCENE_UNDERSTANDING_COARSE permission listed in their manifest as this extension depends on the geometry of the environment. The android.permission.SCENE_UNDERSTANDING_COARSE permission is considered a dangerous permission.
(protection level: dangerous)
Runtime Support
If the runtime supports object tracking, it must indicate this by enumerating XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID in xrEnumerateSpatialCapabilitiesEXT .
Configuration
The XrSpatialCapabilityConfigurationObjectTrackingANDROID structure is defined as:
typedef struct XrSpatialCapabilityConfigurationObjectTrackingANDROID {
XrStructureType type;
const void* next;
XrSpatialCapabilityEXT capability;
uint32_t enabledComponentCount;
const XrSpatialComponentTypeEXT* enabledComponents;
uint32_t activeSemanticLabelCount;
const XrSpatialObjectSemanticLabelANDROID* activeSemanticLabels;
} XrSpatialCapabilityConfigurationObjectTrackingANDROID;
Member Descriptions
typeis the XrStructureType of this structure.nextisNULLor a pointer to the next structure in a structure chain.capabilityis an XrSpatialCapabilityEXT and must beXR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID.enabledComponentCountis auint32_tdescribing the count of elements in theenabledComponentsarray. It must be greater than 0.enabledComponentsis a pointer to an array of XrSpatialComponentTypeEXT .activeSemanticLabelCountis auint32_tdescribing the count of elements in theactiveSemanticLabelsarray. It can be 0, which means all semantic labels are active, or it can be greater than 0, in which case theactiveSemanticLabelsarray is used to determine which semantic labels are active.activeSemanticLabelsis a pointer to the array of XrSpatialObjectSemanticLabelANDROID indicating the active objects in tracking. It must be null ifactiveSemanticLabelCountis 0. And it must be non-null ifactiveSemanticLabelCountis greater than 0, which means that it is used to determine which semantic labels are active.
Applications can enable the XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID spatial capability by adding a pointer to the XrSpatialCapabilityConfigurationObjectTrackingANDROID structure in XrSpatialContextCreateInfoEXT :: capabilityConfigs .
The runtime must return XR_ERROR_VALIDATION_FAILURE if capability is not XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID .
Applications can specify the desired semantic labels in activeSemanticLabels which they are in favor to track. If there is no semantic labels specified in activeSemanticLabels and activeSemanticLabelCount is 0, all semantic labels are active to be tracked by default.
Valid Usage (Implicit)
- The
XR_ANDROID_spatial_object_trackingextension must be enabled prior to using XrSpatialCapabilityConfigurationObjectTrackingANDROID -
typemust beXR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_ANDROID -
nextmust beNULLor a valid pointer to the next structure in a structure chain -
capabilitymust be a valid XrSpatialCapabilityEXT value -
enabledComponentsmust be a pointer to an array ofenabledComponentCountvalid XrSpatialComponentTypeEXT values -
activeSemanticLabelsmust be a pointer to an array ofactiveSemanticLabelCountvalid XrSpatialObjectSemanticLabelANDROID values - The
enabledComponentCountparameter must be greater than0 - The
activeSemanticLabelCountparameter must be greater than0
Guaranteed Components
A runtime that supports XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID must support the following spatial components as guaranteed components of the entities discovered by this capability and must enumerate them in xrEnumerateSpatialCapabilityComponentTypesEXT -
XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXTXR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROIDAn object tracking capability is meaningless without a semantic label component. While this extension defines theXR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROIDas one of the possible semantic label components for runtimes to support, other extensions may define new component types to provide different classes of semantic labels that describe a tracked object.
Bounded 3D Component
The bounded 3D component provides the center and extents of the object represented by the entity it is on. See Bounded 3D for more details.
Object Semantic Label Component
Component Data
typedef enum XrSpatialObjectSemanticLabelANDROID {
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_ANDROID = 0,
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_KEYBOARD_ANDROID = 1,
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MOUSE_ANDROID = 2,
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_LAPTOP_BASE_ANDROID = 3,
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSpatialObjectSemanticLabelANDROID;
The XrSpatialObjectSemanticLabelANDROID enumeration describes a set of semantic labels for a object.
Enum Description
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_ANDROID
The runtime was unable to classify this entity.
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_KEYBOARD_ANDROID
The entity is a keyboard.
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MOUSE_ANDROID
The entity is a mouse.
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_LAPTOP_BASE_ANDROID
The entity is a laptop.
Spatial object orientation
The orientation of the objects is shown in the following image (with xyz = rgb) originating at the bottom center of the objects.
The 3D orientation of the mouse is aligned to the 3D orientation of the supporting plane. The extents enclose the keyboard and mouse completely and the base of the laptop.
Figure 34. Example object orientation.
Component list struct to query data
The XrSpatialComponentObjectSemanticLabelListANDROID structure is defined as:
typedef struct XrSpatialComponentObjectSemanticLabelListANDROID {
XrStructureType type;
void* next;
uint32_t semanticLabelCount;
XrSpatialObjectSemanticLabelANDROID* semanticLabels;
} XrSpatialComponentObjectSemanticLabelListANDROID;
Member Descriptions
typeis the XrStructureType of this structure.nextisNULLor a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.semanticLabelCountis auint32_tdescribing the count of elements in thesemanticLabelsarray.semanticLabelsis an array of XrSpatialObjectSemanticLabelANDROID .
The application can query the semantic label component of the spatial entities in an XrSpatialSnapshotEXT by adding XR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID in XrSpatialComponentDataQueryConditionEXT :: componentTypes and adding XrSpatialComponentObjectSemanticLabelListANDROID to the next pointer chain of XrSpatialComponentDataQueryResultEXT .
The runtime must return XR_ERROR_VALIDATION_FAILURE from xrQuerySpatialComponentDataEXT if XrSpatialComponentObjectSemanticLabelListANDROID is in the next chain of XrSpatialComponentDataQueryResultEXT :: next but XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID is not included in XrSpatialComponentDataQueryConditionEXT :: componentTypes .
The runtime must return XR_ERROR_SIZE_INSUFFICIENT from xrQuerySpatialComponentDataEXT if semanticLabelCount is less than XrSpatialComponentDataQueryResultEXT :: entityIdCountOutput .
Valid Usage (Implicit)
- The
XR_ANDROID_spatial_object_trackingextension must be enabled prior to using XrSpatialComponentObjectSemanticLabelListANDROID -
typemust beXR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID -
nextmust beNULLor a valid pointer to the next structure in a structure chain -
semanticLabelsmust be a pointer to an array ofsemanticLabelCountXrSpatialObjectSemanticLabelANDROID values - The
semanticLabelCountparameter must be greater than0
Configuration
If XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID is enumerated in XrSpatialCapabilityComponentTypesEXT :: componentTypes for some capability, application can enable it by including the enum in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: enabledComponents list of the XrSpatialCapabilityConfigurationBaseHeaderEXT derived structure of the capability that supports this component.
This component does not require any special configuration to be included in the next chain of XrSpatialCapabilityConfigurationBaseHeaderEXT .
Example code
Configure Object Tracking Capability
The following example code demonstrates how to configure object tracking capability when creating a spatial context.
// Check if object tracking capability is supported
uint32_t capabilityCount = 0;
CHK_XR(xrEnumerateSpatialCapabilitiesEXT(instance, systemId, 0, &capabilityCount, nullptr));
std::vector<XrSpatialCapabilityEXT> capabilities(capabilityCount);
CHK_XR(xrEnumerateSpatialCapabilitiesEXT(instance, systemId, capabilityCount, &capabilityCount, capabilities.data()));
if (std::find(capabilities.begin(), capabilities.end(), XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID) == capabilities.end()) {
return;
}
// Enumerate supported components for object tracking capability
XrSpatialCapabilityComponentTypesEXT objectComponents{
.type = XR_TYPE_SPATIAL_CAPABILITY_COMPONENT_TYPES_EXT,
};
CHK_XR(xrEnumerateSpatialCapabilityComponentTypesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID, &objectComponents));
std::vector<XrSpatialComponentTypeEXT> objectCapabilityComponents(objectComponents.componentTypeCountOutput);
objectComponents.componentTypeCapacityInput = objectCapabilityComponents.size();
objectComponents.componentTypes = objectCapabilityComponents.data();
CHK_XR(xrEnumerateSpatialCapabilityComponentTypesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID, &objectComponents));
// Create a spatial context
XrSpatialContextEXT spatialContext{XR_NULL_HANDLE};
// Enable the 2 guaranteed components (bounded 3D, semantic label) of the object tracking capability
std::vector<XrSpatialComponentTypeEXT> enabledComponents = {
XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT,
XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID,
};
// Specify some semantic labels to be active and tracked
std::vector<XrSpatialObjectSemanticLabelANDROID> activeSemanticLabels =
{
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_KEYBOARD_ANDROID,
XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MOUSE_ANDROID,
};
XrSpatialCapabilityConfigurationObjectTrackingANDROID objectTrackingConfig{
.type = XR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_ANDROID,
.capability = XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID,
.enabledComponentCount = static_cast<uint32_t>(enabledComponents.size()),
.enabledComponents = enabledComponents.data(),
// Specify the count of active semantic labels and the pointer to the array of
// active semantic labels.
// If the `activeSemanticLabelCount` is 0 and `activeSemanticLabels` is NULL,
// all labels will be set to active by default.
.activeSemanticLabelCount = static_cast<uint32_t>(activeSemanticLabels.size()),
.activeSemanticLabels = activeSemanticLabels.data(),
};
std::array<XrSpatialCapabilityConfigurationBaseHeaderEXT*, 1> capabilityConfigs = {
reinterpret_cast<XrSpatialCapabilityConfigurationBaseHeaderEXT*>(&objectTrackingConfig),
};
XrSpatialContextCreateInfoEXT spatialContextCreateInfo{
.type = XR_TYPE_SPATIAL_CONTEXT_CREATE_INFO_EXT,
.capabilityConfigCount = capabilityConfigs.size(),
.capabilityConfigs = capabilityConfigs.data(),
};
XrFutureEXT createContextFuture {XR_NULL_FUTURE_EXT};
CHK_XR(xrCreateSpatialContextAsyncEXT(session, &spatialContextCreateInfo, &createContextFuture));
waitUntilReady(createContextFuture);
XrCreateSpatialContextCompletionEXT completion{
.type = XR_TYPE_CREATE_SPATIAL_CONTEXT_COMPLETION_EXT,
};
CHK_XR(xrCreateSpatialContextCompleteEXT(session, createContextFuture, &completion));
if (completion.futureResult != XR_SUCCESS) {
return;
}
spatialContext = completion.spatialContext;
// ...
// Discover entities with the spatial context
// ...
CHK_XR(xrDestroySpatialContextEXT(spatialContext));
Discover Spatial Entities & Query Component Data
The following example code demonstrates how to discover spatial entities for a context configured with XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID and query its component data.
XrFutureEXT future = XR_NULL_FUTURE_EXT;
// We want to look for entities that have the following components.
std::vector<XrSpatialComponentTypeEXT> snapshotComponents = {
XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT,
XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID,
};
auto discoverSpatialEntities = [&](XrSpatialContextEXT spatialContext, XrTime time) {
XrSpatialDiscoverySnapshotCreateInfoEXT snapshotCreateInfo{
.type = XR_TYPE_SPATIAL_DISCOVERY_SNAPSHOT_CREATE_INFO_EXT,
.componentTypeCount = static_cast<uint32_t>(snapshotComponents.size()),
.componentTypes = snapshotComponents.data(),
};
CHK_XR(xrCreateSpatialDiscoverySnapshotAsyncEXT(spatialContext, &snapshotCreateInfo, &future));
waitUntilReady(future);
XrCreateSpatialDiscoverySnapshotCompletionInfoEXT completionInfo{
.type = XR_TYPE_CREATE_SPATIAL_DISCOVERY_SNAPSHOT_COMPLETION_INFO_EXT,
.baseSpace = localSpace,
.time = time,
.future = future,
};
XrCreateSpatialDiscoverySnapshotCompletionEXT completion{
.type = XR_TYPE_CREATE_SPATIAL_DISCOVERY_SNAPSHOT_COMPLETION_EXT,
};
CHK_XR(xrCreateSpatialDiscoverySnapshotCompleteEXT(spatialContext, &completionInfo, &completion));
if (completion.futureResult == XR_SUCCESS) {
// Query for the semantic label component data
XrSpatialComponentDataQueryConditionEXT queryCond{
.type = XR_TYPE_SPATIAL_COMPONENT_DATA_QUERY_CONDITION_EXT,
.componentTypeCount = static_cast<uint32_t>(snapshotComponents.size()),
.componentTypes = snapshotComponents.data(),
};
XrSpatialComponentDataQueryResultEXT queryResult{
.type = XR_TYPE_SPATIAL_COMPONENT_DATA_QUERY_RESULT_EXT,
};
CHK_XR(xrQuerySpatialComponentDataEXT(completion.snapshot, &queryCond, &queryResult));
std::vector<XrSpatialEntityIdEXT> entityIds(queryResult.entityIdCountOutput);
std::vector<XrSpatialEntityTrackingStateEXT> entityStates(queryResult.entityIdCountOutput);
queryResult.entityIdCapacityInput = entityIds.size();
queryResult.entityIds = entityIds.data();
queryResult.entityStateCapacityInput = entityStates.size();
queryResult.entityStates = entityStates.data();
std::vector<XrBoxf> bounded3D(queryResult.entityIdCountOutput);
XrSpatialComponentBounded3DListEXT bounded3DList{
.type = XR_TYPE_SPATIAL_COMPONENT_BOUNDED_3D_LIST_EXT,
.boundCount = static_cast<uint32_t>(bounded3D.size()),
.bounds = bounded3D.data(),
};
queryResult.next = &bounded3DList;
std::vector<XrSpatialObjectSemanticLabelANDROID> semanticLabels(queryResult.entityIdCountOutput);
XrSpatialComponentObjectSemanticLabelListANDROID semanticLabelsList{
.type = XR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID,
.semanticLabelCount = static_cast<uint32_t>(semanticLabels.size()),
.semanticLabels = semanticLabels.data(),
};
queryResult.next = &semanticLabelsList;
CHK_XR(xrQuerySpatialComponentDataEXT(completion.snapshot, &queryCond, &queryResult));
for (int32_t i = 0; i < queryResult.entityIdCountOutput; ++i) {
if (entityStates[i] != XR_SPATIAL_ENTITY_TRACKING_STATE_TRACKING_EXT) {
continue;
}
// 3D bounds for entity entityIds[i] is bounded3D[i].extents centered on bounded3D[i].center.
// semantic label for entity entityIds[i] is semanticLabels[i].
}
CHK_XR(xrDestroySpatialSnapshotEXT(completion.snapshot));
}
};
while (1) {
// ...
// For every frame in frame loop
// ...
XrFrameState frameState; // previously returned from xrWaitFrame
const XrTime time = frameState.predictedDisplayTime;
// Poll for the XR_TYPE_EVENT_DATA_SPATIAL_DISCOVERY_RECOMMENDED_EXT event
XrEventDataBuffer event = {
.type = XR_TYPE_EVENT_DATA_BUFFER,
};
XrResult result = xrPollEvent(instance, &event);
if (result == XR_SUCCESS) {
if (event.type == XR_TYPE_EVENT_DATA_SPATIAL_DISCOVERY_RECOMMENDED_EXT) {
const XrEventDataSpatialDiscoveryRecommendedEXT& eventdata =
*reinterpret_cast<XrEventDataSpatialDiscoveryRecommendedEXT*>(&event);
// Discover spatial entities for the context that we received the "discovery
// recommended" event for.
discoverSpatialEntities(eventdata.spatialContext, time);
}
}
// ...
// Finish frame loop
// ...
}
New Structures
- XrSpatialCapabilityConfigurationObjectTrackingANDROID
Extending XrSpatialComponentDataQueryResultEXT :
New Enums
New Enum Constants
XR_ANDROID_SPATIAL_OBJECT_TRACKING_EXTENSION_NAMEXR_ANDROID_spatial_object_tracking_SPEC_VERSIONExtending XrSpatialCapabilityEXT :
XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_ANDROID
Extending XrSpatialComponentTypeEXT :
XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_ANDROID
Extending XrStructureType :
XR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_ANDROIDXR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_ANDROID
Issues
Version History
Revision 1, 2025-06-26 (Kyle Chen)
- Initial extension description.
Revision 2, 2025-12-04 (Kyle Chen)
- Update the example format to new style.