XR_EXT_spatial_object_tracking

Name String

XR_EXT_spatial_object_tracking

Extension Type

Instance extension

Registered Extension Number

745

Revision

1

Ratification Status

Not ratified

Extension and Version Dependencies

XR_EXT_spatial_entity

IP Status

No known IP claims.

Contributors

Nihav Jain, Google
Natalie Fleury, Meta
Yuichi Taguchi, Meta
Ron Bessems, Meta
Yin Li, Microsoft
Jimmy Alamparambil, ByteDance
Zhipeng Liu, ByteDance
Jun Yan, ByteDance
Andy Biar, Unity Technologies
Rebecca Nordhauser, Unity Technologies

Overview

This extension builds on XR_EXT_spatial_entity and defines a spatial capability for tracking a set of predefined objects without requiring an input dataset from the application.

Runtime Support

If the runtime supports object tracking, it must indicate this by enumerating XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT in xrEnumerateSpatialCapabilitiesEXT .

Configuration

The XrSpatialCapabilityConfigurationObjectTrackingEXT structure is defined as:

typedef struct XrSpatialCapabilityConfigurationObjectTrackingEXT {
    XrStructureType                     type;
    const void*                         next;
    XrSpatialCapabilityEXT              capability;
    uint32_t                            enabledComponentCount;
    const XrSpatialComponentTypeEXT*    enabledComponents;
} XrSpatialCapabilityConfigurationObjectTrackingEXT;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain.
  • capability is an XrSpatialCapabilityEXT and must be XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT .
  • enabledComponentCount is a uint32_t describing the count of elements in the enabledComponents array.
  • enabledComponents is a pointer to an array of XrSpatialComponentTypeEXT .

Applications can enable the XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT spatial capability by adding a pointer to the XrSpatialCapabilityConfigurationObjectTrackingEXT structure in XrSpatialContextCreateInfoEXT :: capabilityConfigs .

The runtime must return XR_ERROR_VALIDATION_FAILURE if capability is not XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT .

Valid Usage (Implicit)

Guaranteed Components

A runtime that supports XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT , must support the following spatial components as guaranteed components of the entities discovered by this capability and must enumerate them in xrEnumerateSpatialCapabilityComponentTypesEXT :

  • XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT
  • At least one component type that provides a set of semantic label for the tracked objects. An object tracking capability is meaningless without a semantic label component. While this extension defines XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT and XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT as possible semantic label components for runtimes to support, other extensions may define new component types to provide different classes of semantic labels that describe a tracked object. See [ext_spatial_semantic_labels_guidelines] for details about the semantic label sets and component types design. Runtimes may not support either of XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT or XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT for the object tracking capability but must support at least one component type that provides a semantic label.

Bounded 3D Component

The bounded 3D component provides the center and extents of the object represented by the entity it is on. See Bounded 3D for more details.

Optional Components

A runtime that supports XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT , may support spatial components in addition to the ones listed in the Guaranteed Components section. The application can use xrEnumerateSpatialCapabilityComponentTypesEXT to get the full list of components that a runtime supports and can then configure the ones it is interested in when creating the spatial context.

Object Semantic Label

Component data

typedef enum XrSpatialObjectSemanticLabelEXT {
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_EXT = 1,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_SOFA_EXT = 2,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_BED_EXT = 3,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_CHAIR_EXT = 4,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_TABLE_EXT = 5,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_LAMP_EXT = 6,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_PLANT_EXT = 7,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_DISPLAY_SCREEN_EXT = 8,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_WALL_ART_EXT = 9,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_CABINET_EXT = 10,
    XR_SPATIAL_OBJECT_SEMANTIC_LABEL_MAX_ENUM_EXT = 0x7FFFFFFF
} XrSpatialObjectSemanticLabelEXT;

The XrSpatialObjectSemanticLabelEXT enumeration describes a set of semantic labels for objects.

For entities with the XR_SPATIAL_OBJECT_SEMANTIC_LABEL_DISPLAY_SCREEN_EXT label, the +Z axis of the Bounded 3D component must roughly match the surface normal of the screen.

For entities with the XR_SPATIAL_OBJECT_SEMANTIC_LABEL_WALL_ART_EXT label, the +Z axis of the Bounded 3D component must match the surface normal of the wall the art is discovered on.

For entities with the XR_SPATIAL_OBJECT_SEMANTIC_LABEL_TABLE_EXT label, the +Z axis of the Bounded 3D component must match the surface normal of the table.

For all the other labels in XrSpatialObjectSemanticLabelEXT , the application must not make any assumptions about how the XrBoxf :: center :: orientation provided by the Bounded 3D component on the entity aligns with the physical attributes of that object.

The enumeration values have the following meanings:

Enum Description

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_EXT

The runtime was unable to classify this entity.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_SOFA_EXT

The entity is a sofa.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_BED_EXT

The entity is a bed.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_CHAIR_EXT

The entity is a chair.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_TABLE_EXT

The entity is a table.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_LAMP_EXT

The entity is a lamp.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_PLANT_EXT

The entity is a plant.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_DISPLAY_SCREEN_EXT

The entity is a display screen, like a monitor or TV.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_WALL_ART_EXT

The entity is a wall art.

XR_SPATIAL_OBJECT_SEMANTIC_LABEL_CABINET_EXT

The entity is a cabinet.

Component List Structure to Query Data

The XrSpatialComponentObjectSemanticLabelListEXT structure is defined as:

typedef struct XrSpatialComponentObjectSemanticLabelListEXT {
    XrStructureType                     type;
    void*                               next;
    uint32_t                            semanticLabelCount;
    XrSpatialObjectSemanticLabelEXT*    semanticLabels;
} XrSpatialComponentObjectSemanticLabelListEXT;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • semanticLabelCount is a uint32_t describing the count of elements in the semanticLabels array.
  • semanticLabels is an array of XrSpatialObjectSemanticLabelEXT .

To query the object semantic label component of the spatial entities in an XrSpatialSnapshotEXT , include XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT in XrSpatialComponentDataQueryConditionEXT :: componentTypes and add XrSpatialComponentObjectSemanticLabelListEXT to the XrSpatialComponentDataQueryResultEXT :: next chain.

The runtime must return XR_ERROR_VALIDATION_FAILURE from xrQuerySpatialComponentDataEXT if XrSpatialComponentObjectSemanticLabelListEXT is in the XrSpatialComponentDataQueryResultEXT :: next chain but XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT is not included in XrSpatialComponentDataQueryConditionEXT :: componentTypes .

The runtime must return XR_ERROR_SIZE_INSUFFICIENT from xrQuerySpatialComponentDataEXT if semanticLabelCount is less than XrSpatialComponentDataQueryResultEXT :: entityIdCountOutput .

Valid Usage (Implicit)

Configuration

If XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT is enumerated in XrSpatialCapabilityComponentTypesEXT :: componentTypes for some capability, an application can enable it by including the enumerant in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: enabledComponents list of the XrSpatialCapabilityConfigurationBaseHeaderEXT derived structure of the capability that supports this component.

This component does not require any special configuration to be included in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: next chain but does offer an optional configuration for applications to provide.

The XrSpatialFeatureObjectSemanticLabelSubsetEXT structure is defined as:

typedef struct XrSpatialFeatureObjectSemanticLabelSubsetEXT {
    XrStructureType                           type;
    const void*                               next;
    uint32_t                                  semanticLabelCount;
    const XrSpatialObjectSemanticLabelEXT*    semanticLabels;
} XrSpatialFeatureObjectSemanticLabelSubsetEXT;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain.
  • semanticLabelCount is a uint32_t describing the count of elements in the semanticLabels array.
  • semanticLabels is an array of XrSpatialObjectSemanticLabelEXT .

If the runtime enumerates XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT from xrEnumerateSpatialCapabilityComponentTypesEXT for a given capability, it must also enumerate XR_SPATIAL_CAPABILITY_FEATURE_OBJECT_SEMANTIC_LABEL_SUBSET_EXT from xrEnumerateSpatialCapabilityFeaturesEXT for the same capability. If an application enables XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT when configuring that capability, it can chain XrSpatialFeatureObjectSemanticLabelSubsetEXT to the corresponding configuration structure of that capability to provide the runtime with a subset of labels defined in XrSpatialObjectSemanticLabelEXT . The runtime may use this subset of labels to optimize its tracking pipelines.

The runtime must return XR_ERROR_VALIDATION_FAILURE from xrCreateSpatialContextAsyncEXT if semanticLabels contains any label that is not defined in XrSpatialObjectSemanticLabelEXT or if semanticLabels contains XR_SPATIAL_OBJECT_SEMANTIC_LABEL_UNCATEGORIZED_EXT .

The runtime must return XR_ERROR_SPATIAL_COMPONENT_NOT_ENABLED_EXT from xrCreateSpatialContextAsyncEXT if the application provides a valid list of labels in semanticLabels but XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT is not included in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: enabledComponents array of the configuration structure that XrSpatialFeatureObjectSemanticLabelSubsetEXT is chained to.

If the application configures a capability with a valid XrSpatialFeatureObjectSemanticLabelSubsetEXT , the runtime must not include entities that have semantic labels which are not listed in semanticLabels in any discovery snapshot created for the configured spatial context.

Valid Usage (Implicit)

Keyboard Semantic Label

Component data

typedef enum XrSpatialKeyboardSemanticLabelEXT {
    XR_SPATIAL_KEYBOARD_SEMANTIC_LABEL_UNCATEGORIZED_EXT = 1,
    XR_SPATIAL_KEYBOARD_SEMANTIC_LABEL_KEYBOARD_EXT = 2,
    XR_SPATIAL_KEYBOARD_SEMANTIC_LABEL_MAX_ENUM_EXT = 0x7FFFFFFF
} XrSpatialKeyboardSemanticLabelEXT;

The XrSpatialKeyboardSemanticLabelEXT enumeration describes a semantic label for a keyboard.

The extents and orientation of XR_SPATIAL_KEYBOARD_SEMANTIC_LABEL_KEYBOARD_EXT represented by the Bounded 3D component on the same entity is shown in the following image (with xyz = rgb) originating at the center of the object.

ext spatial object tracking keyboard orientation

Figure 30. Keyboard orientation.

Enum Description

XR_SPATIAL_KEYBOARD_SEMANTIC_LABEL_UNCATEGORIZED_EXT

The runtime was unable to classify this entity.

XR_SPATIAL_KEYBOARD_SEMANTIC_LABEL_KEYBOARD_EXT

The entity is a keyboard.

Component List Struct to Query Data

The XrSpatialComponentKeyboardSemanticLabelListEXT structure is defined as:

typedef struct XrSpatialComponentKeyboardSemanticLabelListEXT {
    XrStructureType                       type;
    void*                                 next;
    uint32_t                              semanticLabelCount;
    XrSpatialKeyboardSemanticLabelEXT*    semanticLabels;
} XrSpatialComponentKeyboardSemanticLabelListEXT;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • semanticLabelCount is a uint32_t describing the count of elements in the semanticLabels array.
  • semanticLabels is an array of XrSpatialKeyboardSemanticLabelEXT .

To query the keyboard semantic label component of the spatial entities in an XrSpatialSnapshotEXT , include XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT in XrSpatialComponentDataQueryConditionEXT :: componentTypes and add XrSpatialComponentKeyboardSemanticLabelListEXT to the XrSpatialComponentDataQueryResultEXT :: next chain.

The runtime must return XR_ERROR_VALIDATION_FAILURE from xrQuerySpatialComponentDataEXT if XrSpatialComponentKeyboardSemanticLabelListEXT is in the XrSpatialComponentDataQueryResultEXT :: next chain but XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT is not included in XrSpatialComponentDataQueryConditionEXT :: componentTypes .

The runtime must return XR_ERROR_SIZE_INSUFFICIENT from xrQuerySpatialComponentDataEXT if semanticLabelCount is less than XrSpatialComponentDataQueryResultEXT :: entityIdCountOutput .

Valid Usage (Implicit)

Configuration

If XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT is enumerated in XrSpatialCapabilityComponentTypesEXT :: componentTypes for some capability, an application can enable it by including the enumerant in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: enabledComponents list of the XrSpatialCapabilityConfigurationBaseHeaderEXT derived structure of the capability that supports this component.

This component does not require any special configuration to be included in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: next chain.

Mouse Semantic Label

Component data

typedef enum XrSpatialMouseSemanticLabelEXT {
    XR_SPATIAL_MOUSE_SEMANTIC_LABEL_UNCATEGORIZED_EXT = 1,
    XR_SPATIAL_MOUSE_SEMANTIC_LABEL_MOUSE_EXT = 2,
    XR_SPATIAL_MOUSE_SEMANTIC_LABEL_MAX_ENUM_EXT = 0x7FFFFFFF
} XrSpatialMouseSemanticLabelEXT;

The XrSpatialMouseSemanticLabelEXT enumeration describes a semantic label for a mouse.

Enum Description

XR_SPATIAL_MOUSE_SEMANTIC_LABEL_UNCATEGORIZED_EXT

The runtime was unable to classify this entity.

XR_SPATIAL_MOUSE_SEMANTIC_LABEL_MOUSE_EXT

The entity is a mouse.

Component List Struct to Query Data

The XrSpatialComponentMouseSemanticLabelListEXT structure is defined as:

typedef struct XrSpatialComponentMouseSemanticLabelListEXT {
    XrStructureType                    type;
    void*                              next;
    uint32_t                           semanticLabelCount;
    XrSpatialMouseSemanticLabelEXT*    semanticLabels;
} XrSpatialComponentMouseSemanticLabelListEXT;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • semanticLabelCount is a uint32_t describing the count of elements in the semanticLabels array.
  • semanticLabels is an array of XrSpatialMouseSemanticLabelEXT .

To query the mouse semantic label component of the spatial entities in an XrSpatialSnapshotEXT , include XR_SPATIAL_COMPONENT_TYPE_MOUSE_SEMANTIC_LABEL_EXT in XrSpatialComponentDataQueryConditionEXT :: componentTypes and add XrSpatialComponentMouseSemanticLabelListEXT to the XrSpatialComponentDataQueryResultEXT :: next chain.

The runtime must return XR_ERROR_VALIDATION_FAILURE from xrQuerySpatialComponentDataEXT if XrSpatialComponentMouseSemanticLabelListEXT is in the XrSpatialComponentDataQueryResultEXT :: next chain but XR_SPATIAL_COMPONENT_TYPE_MOUSE_SEMANTIC_LABEL_EXT is not included in XrSpatialComponentDataQueryConditionEXT :: componentTypes .

The runtime must return XR_ERROR_SIZE_INSUFFICIENT from xrQuerySpatialComponentDataEXT if semanticLabelCount is less than XrSpatialComponentDataQueryResultEXT :: entityIdCountOutput .

Valid Usage (Implicit)

Configuration

If XR_SPATIAL_COMPONENT_TYPE_MOUSE_SEMANTIC_LABEL_EXT is enumerated in XrSpatialCapabilityComponentTypesEXT :: componentTypes for some capability, an application can enable it by including the enumerant in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: enabledComponents list of the XrSpatialCapabilityConfigurationBaseHeaderEXT derived structure of the capability that supports this component.

This component does not require any special configuration to be included in the XrSpatialCapabilityConfigurationBaseHeaderEXT :: next chain.

Example code

Configure Object Tracking Capability

The following example code demonstrates how to configure object tracking capability when creating a spatial context.

// Check if object tracking capability is supported
uint32_t capabilityCount;
CHK_XR(xrEnumerateSpatialCapabilitiesEXT(instance, systemId, 0, &capabilityCount, nullptr));
std::vector<XrSpatialCapabilityEXT> capabilities(capabilityCount);
CHK_XR(xrEnumerateSpatialCapabilitiesEXT(instance, systemId, capabilityCount, &capabilityCount, capabilities.data()));

if (std::find(capabilities.begin(), capabilities.end(), XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT) == capabilities.end()) {
  return;
}

// Enumerate supported components for object tracking capability
XrSpatialCapabilityComponentTypesEXT objectComponents{XR_TYPE_SPATIAL_CAPABILITY_COMPONENT_TYPES_EXT};
CHK_XR(xrEnumerateSpatialCapabilityComponentTypesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT, &objectComponents));
std::vector<XrSpatialComponentTypeEXT> objectCapabilityComponents(objectComponents.componentTypeCountOutput);
objectComponents.componentTypeCapacityInput = objectCapabilityComponents.size();
objectComponents.componentTypes = objectCapabilityComponents.data();
CHK_XR(xrEnumerateSpatialCapabilityComponentTypesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT, &objectComponents));

// Check if the object & keyboard semantic labels component is supported
const auto supportsComponent = [&objectCapabilityComponents](XrSpatialComponentTypeEXT component) {
  return std::find(objectCapabilityComponents.begin(), objectCapabilityComponents.end(), component) != objectCapabilityComponents.end();
};

const bool supportsObjectSemanticLabel = supportsComponent(XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT);
const bool supportsKeyboardSemanticLabel = supportsComponent(XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT);

// Enumerate supported features for object tracking capability
uint32_t featureCount;
CHK_XR(xrEnumerateSpatialCapabilityFeaturesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT, 0, &featureCount, nullptr));
std::vector<XrSpatialCapabilityFeatureEXT> features(featureCount);
CHK_XR(xrEnumerateSpatialCapabilityFeaturesEXT(instance, systemId, XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT, featureCount, &featureCount, features.data()));

const bool supportsObjectSemanticLabelSubsetFeature = std::find(features.begin(), features.end(), XR_SPATIAL_CAPABILITY_FEATURE_OBJECT_SEMANTIC_LABEL_SUBSET_EXT) != features.end();

// Create a spatial context
XrSpatialContextEXT spatialContext{};

// Enable the guaranteed components of the object tracking capability
std::vector<XrSpatialComponentTypeEXT> enabledComponents = {
  XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT,
};
// Optionally enable the supported semantic labels
if (supportsObjectSemanticLabel) {
  enabledComponents.push_back(XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT);
}
if (supportsKeyboardSemanticLabel) {
  enabledComponents.push_back(XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT);
}

// We only want to track sofa and table.
std::vector<XrSpatialObjectSemanticLabelEXT> objectSemanticLabelSubset = {
  XR_SPATIAL_OBJECT_SEMANTIC_LABEL_SOFA_EXT,
  XR_SPATIAL_OBJECT_SEMANTIC_LABEL_TABLE_EXT
};
XrSpatialFeatureObjectSemanticLabelSubsetEXT subsetFeature{XR_TYPE_SPATIAL_FEATURE_OBJECT_SEMANTIC_LABEL_SUBSET_EXT};
subsetFeature.semanticLabelCount = objectSemanticLabelSubset.size();
subsetFeature.semanticLabels = objectSemanticLabelSubset.data();

XrSpatialCapabilityConfigurationObjectTrackingEXT objectTrackingConfig{XR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_EXT};
objectTrackingConfig.capability = XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT;
objectTrackingConfig.enabledComponentCount = enabledComponents.size();
objectTrackingConfig.enabledComponents = enabledComponents.data();
if (supportsObjectSemanticLabelSubsetFeature) {
  objectTrackingConfig.next = &subsetFeature;
}

std::array<XrSpatialCapabilityConfigurationBaseHeaderEXT*, 1> capabilityConfigs = {
  reinterpret_cast<XrSpatialCapabilityConfigurationBaseHeaderEXT*>(&objectTrackingConfig),
};

XrSpatialContextCreateInfoEXT spatialContextCreateInfo{XR_TYPE_SPATIAL_CONTEXT_CREATE_INFO_EXT};
spatialContextCreateInfo.capabilityConfigCount = capabilityConfigs.size();
spatialContextCreateInfo.capabilityConfigs = capabilityConfigs.data();
XrFutureEXT createContextFuture;
CHK_XR(xrCreateSpatialContextAsyncEXT(session, &spatialContextCreateInfo, &createContextFuture));

waitUntilReady(createContextFuture);

XrCreateSpatialContextCompletionEXT completion{XR_TYPE_CREATE_SPATIAL_CONTEXT_COMPLETION_EXT};
CHK_XR(xrCreateSpatialContextCompleteEXT(session, createContextFuture, &completion));
if (completion.futureResult != XR_SUCCESS) {
  return;
}

spatialContext = completion.spatialContext;

// ...
// Discovery entities with the spatial context
// ...

CHK_XR(xrDestroySpatialContextEXT(spatialContext));

Discover Spatial Entities & Query Component Data

The following example code demonstrates how to discover spatial entities for a context configured with XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT and query its component data.

XrFutureEXT future = XR_NULL_FUTURE_EXT;

// We want to look for entities that have the following components.
std::vector<XrSpatialComponentTypeEXT> snapshotComponents = {
  XR_SPATIAL_COMPONENT_TYPE_BOUNDED_3D_EXT,
};
if (supportsKeyboardSemanticLabel) {
  snapshotComponents.push_back(XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT);
}

auto discoverSpatialEntities = [&](XrSpatialContextEXT spatialContext, XrTime time) {
  XrSpatialDiscoverySnapshotCreateInfoEXT snapshotCreateInfo{XR_TYPE_SPATIAL_DISCOVERY_SNAPSHOT_CREATE_INFO_EXT};
  snapshotCreateInfo.componentTypeCount = snapshotComponents.size();
  snapshotCreateInfo.componentTypes = snapshotComponents.data();
  CHK_XR(xrCreateSpatialDiscoverySnapshotAsyncEXT(spatialContext, &snapshotCreateInfo, &future));

  waitUntilReady(future);

  XrCreateSpatialDiscoverySnapshotCompletionInfoEXT completionInfo{XR_TYPE_CREATE_SPATIAL_DISCOVERY_SNAPSHOT_COMPLETION_INFO_EXT};
  completionInfo.baseSpace = localSpace;
  completionInfo.time = time;
  completionInfo.future = future;

  XrCreateSpatialDiscoverySnapshotCompletionEXT completion{XR_TYPE_CREATE_SPATIAL_DISCOVERY_SNAPSHOT_COMPLETION_EXT};
  CHK_XR(xrCreateSpatialDiscoverySnapshotCompleteEXT(spatialContext, &completionInfo, &completion));
  if (completion.futureResult == XR_SUCCESS) {

    // Query for the semantic label component data
    XrSpatialComponentDataQueryConditionEXT queryCond{XR_TYPE_SPATIAL_COMPONENT_DATA_QUERY_CONDITION_EXT};
    queryCond.componentTypeCount = snapshotComponents.size();
    queryCond.componentTypes = snapshotComponents.data();

    XrSpatialComponentDataQueryResultEXT queryResult{XR_TYPE_SPATIAL_COMPONENT_DATA_QUERY_RESULT_EXT};
    CHK_XR(xrQuerySpatialComponentDataEXT(completion.snapshot, &queryCond, &queryResult));

    std::vector<XrSpatialEntityIdEXT> entityIds(queryResult.entityIdCountOutput);
    std::vector<XrSpatialEntityTrackingStateEXT> entityStates(queryResult.entityIdCountOutput);
    queryResult.entityIdCapacityInput = entityIds.size();
    queryResult.entityIds = entityIds.data();
    queryResult.entityStateCapacityInput = entityStates.size();
    queryResult.entityStates = entityStates.data();

    std::vector<XrBoxf> bounded3D(queryResult.entityIdCountOutput);
    XrSpatialComponentBounded3DListEXT bounded3DList{XR_TYPE_SPATIAL_COMPONENT_BOUNDED_3D_LIST_EXT};
    bounded3DList.boundCount = bounded3D.size();
    bounded3DList.bounds = bounded3D.data();
    queryResult.next = &bounded3DList;

    std::vector<XrSpatialKeyboardSemanticLabelEXT> semanticLabels;
    XrSpatialComponentKeyboardSemanticLabelListEXT semanticLabelsList{XR_TYPE_SPATIAL_COMPONENT_KEYBOARD_SEMANTIC_LABEL_LIST_EXT};
    if (supportsKeyboardSemanticLabel) {
      semanticLabels.resize(queryResult.entityIdCountOutput);
      semanticLabelsList.semanticLabelCount = semanticLabels.size();
      semanticLabelsList.semanticLabels = semanticLabels.data();
      semanticLabelsList.next = queryResult.next;
      queryResult.next = &semanticLabelsList;
    }

    CHK_XR(xrQuerySpatialComponentDataEXT(completion.snapshot, &queryCond, &queryResult));

    for (int32_t i = 0; i < queryResult.entityIdCountOutput; ++i) {
      if (entityStates[i] != XR_SPATIAL_ENTITY_TRACKING_STATE_TRACKING_EXT) {
        continue;
      }

      // 3D bounds for entity entityIds[i] is bounded3D[i].extents centered on bounded3D[i].center.

      if (supportsKeyboardSemanticLabel) {
        // semantic label for entity entityIds[i] is semanticLabels[i].
      }
    }

    CHK_XR(xrDestroySpatialSnapshotEXT(completion.snapshot));
  }
};

while (1) {
  // ...
  // For every frame in frame loop
  // ...

  XrFrameState frameState;  // previously returned from xrWaitFrame
  const XrTime time = frameState.predictedDisplayTime;

  // Poll for the XR_TYPE_EVENT_DATA_SPATIAL_DISCOVERY_RECOMMENDED_EXT event
  XrEventDataBuffer event = {XR_TYPE_EVENT_DATA_BUFFER};
  XrResult result = xrPollEvent(instance, &event);
  if (result == XR_SUCCESS) {
      switch (event.type) {
          case XR_TYPE_EVENT_DATA_SPATIAL_DISCOVERY_RECOMMENDED_EXT: {
              const XrEventDataSpatialDiscoveryRecommendedEXT& eventdata =
                  *reinterpret_cast<XrEventDataSpatialDiscoveryRecommendedEXT*>(&event);
              // Discover spatial entities for the context that we received the "discovery
              // recommended" event for.
              discoverSpatialEntities(eventdata.spatialContext, time);
              break;
          }
      }
  }

  // ...
  // Finish frame loop
  // ...
}

New Structures

New Enums

New Enum Constants

  • XR_EXT_SPATIAL_OBJECT_TRACKING_EXTENSION_NAME
  • XR_EXT_spatial_object_tracking_SPEC_VERSION

  • Extending XrSpatialCapabilityEXT :

    • XR_SPATIAL_CAPABILITY_OBJECT_TRACKING_EXT

  • Extending XrSpatialCapabilityFeatureEXT :

    • XR_SPATIAL_CAPABILITY_FEATURE_OBJECT_SEMANTIC_LABEL_SUBSET_EXT

  • Extending XrSpatialComponentTypeEXT :

    • XR_SPATIAL_COMPONENT_TYPE_KEYBOARD_SEMANTIC_LABEL_EXT
    • XR_SPATIAL_COMPONENT_TYPE_MOUSE_SEMANTIC_LABEL_EXT
    • XR_SPATIAL_COMPONENT_TYPE_OBJECT_SEMANTIC_LABEL_EXT

  • Extending XrStructureType :

    • XR_TYPE_SPATIAL_CAPABILITY_CONFIGURATION_OBJECT_TRACKING_EXT
    • XR_TYPE_SPATIAL_COMPONENT_KEYBOARD_SEMANTIC_LABEL_LIST_EXT
    • XR_TYPE_SPATIAL_COMPONENT_MOUSE_SEMANTIC_LABEL_LIST_EXT
    • XR_TYPE_SPATIAL_COMPONENT_OBJECT_SEMANTIC_LABEL_LIST_EXT
    • XR_TYPE_SPATIAL_FEATURE_OBJECT_SEMANTIC_LABEL_SUBSET_EXT

Issues

Version History

  • Revision 1, 2025-06-03 (Nihav Jain, Google)

    • Initial extension description