XR_ANDROID_scene_meshing OpenXR extension

Name String XR_ANDROID_scene_meshing

Extension Type Instance extension

Registered Extension Number 464

Revision 3

Extension and Version Dependencies OpenXR 1.0

Last Modified Date 2025-05-15

IP Status No known IP claims.

Contributors Spencer Quin, Google Jared Finder, Google Antonio Fontan, Google Cairn Overturf, Google Nihav Jain, Google Salar Khan, Google Sebastian Klose, Google Jürgen Sturm, Google Vinny DaSilva, Google Ricardo Campbell, Google

Overview

This extension is intended to provide mesh data for the meshes that approximately represents the physical objects in your environment. It can be used for visualizing your scene in an immersive application and allowing virtual objects to interact with the physical objects such as collisions.

Scene meshing data can be sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications that store or transfer scene meshing data always ask the user for active and specific acceptance to do so.

Permissions

Android applications must have the android.permission.SCENE_UNDERSTANDING_FINE permission listed in their manifest. The android.permission.SCENE_UNDERSTANDING_FINE permission is considered a sensitive permission as it allows the runtime to study the user environment.

The application must request the permission at runtime to use these functions:

(protection level: dangerous)

Inspect system capability

An application can inspect whether the system is capable of scene meshing by chaining an XrSystemSceneMeshingPropertiesANDROID structure to the XrSystemProperties when calling xrGetSystemProperties.

XrSystemSceneMeshingPropertiesANDROID

typedef struct XrSystemSceneMeshingPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsSceneMeshing;
} XrSystemSceneMeshingPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsSceneMeshing is an XrBool32, indicating if the system supports scene meshing.

If supportsSceneMeshing is XR_FALSE, the system does not support scene meshing. The application should avoid using scene meshing functionality when supportsSceneMeshing is XR_FALSE, as calls to xrCreateSceneMeshingTrackerANDROID will fail.

If supportsSceneMeshing is XR_TRUE, the system supports scene meshing.

Valid Usage (Implicit)

XrSceneMeshSemanticLabelSetANDROID

The XrSceneMeshSemanticLabelSetANDROID enumeration describes the semantic label sets for scene meshing. Each value in this enum represents another enum that contains the semantic labels. For example, the XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID value represents the set XrSceneMeshSemanticLabelANDROID.

typedef enum XrSceneMeshSemanticLabelSetANDROID {
    XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID = 0,
    XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID = 1,
    XR_SCENE_MESH_SEMANTIC_LABEL_SET_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSceneMeshSemanticLabelSetANDROID;

xrEnumerateSupportedSemanticLabelSetsANDROID

The application can get semantic label sets supported by the system using xrEnumerateSupportedSemanticLabelSetsANDROID function.

XrResult xrEnumerateSupportedSemanticLabelSetsANDROID(
    XrInstance                                  instance,
    XrSystemId                                  systemId,
    uint32_t                                    supportedSemanticLabelSetsInputCapacity,
    uint32_t*                                   supportedSemanticLabelSetsOutputCount,
    XrSceneMeshSemanticLabelSetANDROID*         supportedSemanticLabelSets);

Parameter Descriptions

  • instance is an XrInstance created by the application.
  • systemId is an XrSystemId retried from xrGetSystem.
  • supportedSemanticLabelSetsInputCapacity is the length of supportedSemanticLabelSets.
  • supportedSemanticLabelSetsOutputCount is the number of elements in supportedSemanticLabelSets modified by the runtime starting from the beginning of the array.
  • supportedSemanticLabelSets is an array of XrSceneMeshSemanticLabelSetANDROID to which the supported semantic label sets are written by the runtime.

It is expected that every system will at least support XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID because this enum value represents no semantic label set and can be used in the case the application does not need vertex semantics.

Create a scene meshing tracker handle

XrSceneMeshingTrackerANDROID

XR_DEFINE_HANDLE(XrSceneMeshingTrackerANDROID)

The XrSceneMeshingTrackerANDROID handle represents a scene meshing tracker for scene meshing and managing the related resources.

This handle can be used to create a scene mesh snapshot using xrCreateSceneMeshSnapshotANDROID in this extension.

xrCreateSceneMeshingTrackerANDROID

An application can create an XrSceneMeshingTrackerANDROID handle using xrCreateSceneMeshingTrackerANDROID function.

XrResult xrCreateSceneMeshingTrackerANDROID(
    XrSession                                   session,
    const XrSceneMeshingTrackerCreateInfoANDROID* createInfo,
    XrSceneMeshingTrackerANDROID*               tracker);

Parameter Descriptions

If the system does not support scene meshing, xrCreateSceneMeshingTrackerANDROID will return XR_ERROR_FEATURE_UNSUPPORTED. The application can check for system support by calling xrGetSystemProperties with the XrSystemSceneMeshingPropertiesANDROID structure.

Only semantic label sets that are supported by the system can be used in XrSceneMeshingTrackerCreateInfoANDROID::semanticLabelSet when creating a scene meshing tracker. The function xrEnumerateSupportedSemanticLabelSetsANDROID can be used to get the list of supported semantic label sets.

If the application requests an unsupported semanticLabelSet, xrCreateSceneMeshingTrackerANDROID will return XR_ERROR_FEATURE_UNSUPPORTED.

The xrCreateSceneMeshingTrackerANDROID handle owns all the resources for scene meshing. Once finished with scene meshing experience, the application must destroy the handle via the xrDestroySceneMeshingTrackerANDROID function.

XrSceneMeshingTrackerCreateInfoANDROID

The XrSceneMeshingTrackerCreateInfoANDROID structure describes the information to create an XrSceneMeshingTrackerANDROID handle.

typedef struct XrSceneMeshingTrackerCreateInfoANDROID {
    XrStructureType                       type;
    const void*                           next;
    XrSceneMeshSemanticLabelSetANDROID    semanticLabelSet;
    XrBool32                              enableNormals;
} XrSceneMeshingTrackerCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • semanticLabelSet is an XrSceneMeshSemanticLabelSetANDROID used to specify the semantic label set to be used for the scene meshing. If this is set to XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID, the runtime will ignore the XrSceneSubmeshDataANDROID::vertexSemantics buffer field.
  • enableNormals is an XrBool32 used to specify whether to include vertex normals for the mesh vertices in the scene mesh buffer when getting the scene mesh data.

xrDestroySceneMeshingTrackerANDROID

xrDestroySceneMeshingTrackerANDROID function releases the tracker and the underlying resources when finished with scene meshing experiences.

XrResult xrDestroySceneMeshingTrackerANDROID(
    XrSceneMeshingTrackerANDROID          tracker);

Parameter Descriptions

Create a scene mesh snapshot handle

XrSceneMeshSnapshotANDROID

XR_DEFINE_HANDLE(XrSceneMeshSnapshotANDROID)

The XrSceneMeshSnapshotANDROID handle represents a scene mesh snapshot. This is created using xrCreateSceneMeshSnapshotANDROID which essentially grabs a snapshot of the scene mesh data at the time of the function call.

This handle can be used to retrieve submesh information and data using xrGetAllSubmeshStatesANDROID and xrGetSubmeshDataANDROID.

XrSceneMeshSnapshotCreateInfoANDROID

The XrSceneMeshSnapshotCreateInfoANDROID structure describes the information to create an XrSceneMeshSnapshotANDROID handle

typedef struct XrSceneMeshSnapshotCreateInfoANDROID {
    XrStructureType    type;
    const void*        next;
    XrSpace            baseSpace;
    XrTime             time;
    XrBoxf             boundingBox;
} XrSceneMeshSnapshotCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • baseSpace is an XrSpace used to describe the reference space in which the scene submeshes pose info should be represented.
  • time is the XrTime that describes the time when the scene mesh will be processed or displayed.
  • boundingBox is an XrBoxf describing the bounding box within which to acquire the scene mesh.

XrSceneMeshTrackingStateANDROID

The XrSceneMeshTrackingStateANDROID enumeration describes the tracking state for scene meshing tracker. Each value in this enum represents a state of the scene meshing tracker. This enum wrapped in XrSceneMeshSnapshotCreationResultANDROID structure which is returned from xrCreateSceneMeshSnapshotANDROID.

typedef enum XrSceneMeshTrackingStateANDROID {
    XR_SCENE_MESH_TRACKING_STATE_INITIALIZING_ANDROID = 0,
    XR_SCENE_MESH_TRACKING_STATE_TRACKING_ANDROID = 1,
    XR_SCENE_MESH_TRACKING_STATE_WAITING_ANDROID = 2,
    XR_SCENE_MESH_TRACKING_STATE_ERROR_ANDROID = 3,
    XR_SCENE_MESH_TRACKING_STATE_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSceneMeshTrackingStateANDROID;

XrSceneMeshSnapshotCreationResultANDROID

The XrSceneMeshSnapshotCreationResultANDROID structure stores the result of a scene mesh snapshot creation returned from xrCreateSceneMeshSnapshotANDROID. The result includes the snapshot handle and the tracking state of the scene meshing tracker at the time of the snapshot creation.

typedef struct XrSceneMeshSnapshotCreationResultANDROID {
    XrStructureType                  type;
    void*                            next;
    XrSceneMeshSnapshotANDROID       snapshot;
    XrSceneMeshTrackingStateANDROID    trackingState;
} XrSceneMeshSnapshotCreationResultANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • snapshot is an XrSceneMeshSnapshotANDROID handle created by the runtime.
  • trackingState is an XrSceneMeshTrackingStateANDROID that describes the state of the scene meshing tracker at the time of the snapshot creation.

xrCreateSceneMeshSnapshotANDROID

The application can use the xrCreateSceneMeshSnapshotANDROID function to create a scene mesh snapshot from the scene meshing tracker. This function returns an XrSceneMeshSnapshotANDROID handle along with XrSceneMeshTrackingStateANDROID wrapped in an XrSceneMeshSnapshotCreationResultANDROID structure. The handle is technically a snapshot of the scene mesh data at the time of creation. This handle can be used to query the scene mesh info and data using xrGetAllSubmeshStatesANDROID and xrGetSubmeshDataANDROID respectively.

XrResult xrCreateSceneMeshSnapshotANDROID(
    XrSceneMeshingTrackerANDROID                tracker,
    const XrSceneMeshSnapshotCreateInfoANDROID* createInfo,
    XrSceneMeshSnapshotCreationResultANDROID*   outSnapshotCreationResult);

Parameter Descriptions

Acquiring scene mesh data from the scene mesh snapshot

XrSceneSubmeshStateANDROID

XrSceneSubmeshStateANDROID describes a submesh. It contains the basic information about the submesh (id, pose, bounds, last updated time).

typedef struct XrSceneSubmeshStateANDROID {
    XrStructureType    type;
    void*              next;
    XrUuid             submeshId;
    XrTime             lastUpdatedTime;
    XrPosef            submeshPoseInBaseSpace;
    XrExtent3Df        bounds;
} XrSceneSubmeshStateANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • submeshId is an XrUuid identifying the submesh.
  • lastUpdatedTime is an XrTime representing the time when the submesh was last updated.
  • submeshPoseInBaseSpace is an XrPosef representing the pose of the submesh in the base space used to create the scene mesh snapshot. This is also the pose of the center of the submesh bounding box.
  • bounds is an XrExtent3Df describing the dimensions of the bounding box that encloses the submesh. This bounds is in the base space that was used to create the scene mesh handle.

xrGetAllSubmeshStatesANDROID

The application can use the xrGetAllSubmeshStatesANDROID function to get the state of all submeshes in the scene mesh. This contains basic information information to allow the application to select submeshes for which the application wants data for. This function can be used in a 2-call idiom.

XrResult xrGetAllSubmeshStatesANDROID(
    XrSceneMeshSnapshotANDROID            snapshot,
    uint32_t                              submeshStateCapacityInput,
    uint32_t*                             submeshStateCountOutput,
    XrSceneSubmeshStateANDROID*           submeshStates);

Parameter Descriptions

  • snapshot is an XrSceneMeshSnapshotANDROID handle previously created with xrCreateSceneMeshSnapshotANDROID.
  • submeshStateCapacityInput is the length of submeshStates.
  • submeshStateCountOutput is the number of elements in submeshStates that are modified by the runtime starting from the beginning of the array.
  • submeshStates is an array of XrSceneSubmeshStateANDROID structures to which the runtime will output the state of all submeshes in the scene mesh.

XrSceneSubmeshDataANDROID

XrSceneSubmeshDataANDROID contains the triangle data for a submesh along with the submesh id. This structure is used in xrGetSubmeshDataANDROID to get the data of the submesh. The application must set the submesh id of this struct and the buffer pointers to allocated buffers so that the returned data for the submesh can be populated in the allocated buffers. The application can use xrGetSubmeshDataANDROID as a 2-call idiom where the first call will will require the application to set the submesh id and get the sizes of the buffers required for the submesh data, and then the application can allocate the buffers and get the data in the second call.

typedef struct XrSceneSubmeshDataANDROID {
    XrStructureType    type;
    const void*        next;
    XrUuid             submeshId;
    uint32_t           vertexCapacityInput;
    uint32_t           vertexCountOutput;
    XrVector3f*        vertexPositions;
    XrVector3f*        vertexNormals;
    uint8_t*           vertexSemantics;
    uint32_t           indexCapacityInput;
    uint32_t           indexCountOutput;
    uint32_t*          indices;
} XrSceneSubmeshDataANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • submeshId is an XrUuid identifying the submesh.
  • vertexCapacityInput is the length of vertexPositions, vertexNormals, and vertexSemantics.
  • vertexCountOutput is the number of elements in vertexPositions, vertexNormals, and vertexSemantics that have been modified by the runtime starting from the beginning of the array.
  • vertexPositions is an array of XrVector3f to which the runtime will output the vertex positions of the submesh.
  • vertexNormals is an array of XrVector3f to which the runtime will output the vertex normals of the submesh. This field can be left NULL if the application created a tracker with normals disabled.
  • vertexSemantics is an array of uint8_t to which the runtime will output the vertex semantics of the submesh. This field can be left NULL if the application created a tracker with XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID semantic label set.
  • indexCapacityInput is the length of indices.
  • indexCountOutput is the number of elements in indices that have been modified by the runtime starting from the beginning of the array.
  • indices is an array of uint32_t to which the runtime will output the indices of the submesh.

xrGetSubmeshDataANDROID

The application can use the xrGetSubmeshDataANDROID function to get the data for a selected list of submeshes. This function can be used in a 2-call idiom. In first call, the application must set a valid submesh id (i.e: Retrievable from xrGetAllSubmeshStatesANDROID with the same scene mesh snapshot) and zero capacity for each element in the selected list of submeshes to get the sizes of the buffers required for the submesh data. In second call, the application must set the buffer pointers to an allocated buffer for each element in the selected list of submeshes along with the capacity of the buffers to get the data of the submeshes.

XrResult xrGetSubmeshDataANDROID(
    XrSceneMeshSnapshotANDROID            snapshot,
    uint32_t                              submeshDataCount,
    XrSceneSubmeshDataANDROID*            inoutSubmeshData);

Parameter Descriptions

XrSceneMeshSemanticLabelANDROID

The XrSceneMeshSemanticLabelANDROID enumeration is the default set of semantic labels that can be used to label mesh vertices to describe which physical environment surface the mesh vertex is on. This enum set is represented by the XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID value in XrSceneMeshSemanticLabelSetANDROID.

typedef enum XrSceneMeshSemanticLabelANDROID {
    XR_SCENE_MESH_SEMANTIC_LABEL_OTHER_ANDROID = 0,
    XR_SCENE_MESH_SEMANTIC_LABEL_FLOOR_ANDROID = 1,
    XR_SCENE_MESH_SEMANTIC_LABEL_CEILING_ANDROID = 2,
    XR_SCENE_MESH_SEMANTIC_LABEL_WALL_ANDROID = 3,
    XR_SCENE_MESH_SEMANTIC_LABEL_TABLE_ANDROID = 4,
    XR_SCENE_MESH_SEMANTIC_LABEL_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSceneMeshSemanticLabelANDROID;

Destroy a scene mesh snapshot

xrDestroySceneMeshSnapshotANDROID

The application can use the xrDestroySceneMeshSnapshotANDROID function to destroy a scene mesh snapshot. Once the handle is destroyed, the application can no longer use it to get the submesh info or data. The handle is automatically destroyed when the tracker is destroyed because the tracker handle is the parent of the scene mesh snapshot handle.

XrResult xrDestroySceneMeshSnapshotANDROID(
    XrSceneMeshSnapshotANDROID            snapshot);

Parameter Descriptions

Example code for scene meshing

Following example code demonstrates how to access scene mesh data for rendering.

XrInstance instance;  // Created at app startup
XrSystemId systemId;  // Received from xrGetSystem() at app startup
XrSession session;    // Created at app startup
XrSpace appPlaySpace; // Created at app startup

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateSceneMeshingTrackerANDROID xrCreateSceneMeshingTrackerANDROID;
PFN_xrDestroySceneMeshingTrackerANDROID xrDestroySceneMeshingTrackerANDROID;
PFN_xrEnumerateSupportedSemanticLabelSetsANDROID xrEnumerateSupportedSemanticLabelSetsANDROID;
PFN_xrCreateSceneMeshSnapshotANDROID xrCreateSceneMeshSnapshotANDROID;
PFN_xrDestroySceneMeshSnapshotANDROID xrDestroySceneMeshSnapshotANDROID;
PFN_xrGetAllSubmeshStatesANDROID xrGetAllSubmeshStatesANDROID;
PFN_xrGetSubmeshDataANDROID xrGetSubmeshDataANDROID;

// Inspect system capability
XrSystemSceneMeshingPropertiesANDROID sceneMeshingProps = {
  .type = XR_TYPE_SYSTEM_SCENE_MESHING_PROPERTIES_ANDROID,
};
XrSystemProperties sysProps = {
  .type = XR_TYPE_SYSTEM_PROPERTIES,
  .next = &sceneMeshingProps
};
CHK_XR(xrGetSystemProperties(instance, systemId, &sysProps));
if (!sceneMeshingProps.supportsSceneMeshing) {
    // scene meshing is not supported.
    return;
}

uint32_t supportedsemanticLabelSetsCount = 0;
xrEnumerateSupportedSemanticLabelSetsANDROID(
  instance, systemId, 0, &supportedsemanticLabelSetsCount, nullptr);
std::vector<XrSceneMeshSemanticLabelSetANDROID> supportedSemanticLabelSets(
  supportedsemanticLabelSetsCount);
xrEnumerateSupportedSemanticLabelSetsANDROID(
  instance, systemId, supportedsemanticLabelSetsCount,
  &supportedsemanticLabelSetsCount, supportedSemanticLabelSets.data());

XrSceneMeshSemanticLabelSetANDROID semanticLabelSet = XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID;
// Check if system supports XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID
if (std::find(supportedSemanticLabelSets.begin(), supportedSemanticLabelSets.end(),
              XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID) !=
    supportedSemanticLabelSets.end()) {
  semanticLabelSet = XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID;
}

XrSceneMeshingTrackerCreateInfoANDROID trackerCreateInfo = {
  .type = XR_TYPE_SCENE_MESHING_TRACKER_CREATE_INFO_ANDROID,
  .semanticLabelSet = semanticLabelSet,
  .enableNormals = XR_TRUE
};

XrSceneMeshingTrackerANDROID tracker = XR_NULL_HANDLE;
CHK_XR(xrCreateSceneMeshingTrackerANDROID(session, &trackerCreateInfo, &tracker));

XrSceneMeshSnapshotCreationResultANDROID snapshotResult = {
  .type = XR_TYPE_SCENE_MESH_SNAPSHOT_CREATION_RESULT_ANDROID,
  .snapshot = XR_NULL_HANDLE
};
XrSceneMeshSnapshotANDROID& snapshot = snapshotResult.snapshot;
XrSceneMeshSnapshotCreateInfoANDROID createInfo = {
  .type = XR_TYPE_SCENE_MESH_SNAPSHOT_CREATE_INFO_ANDROID
};

// app update loop
while (true) {
  // ...
  // For every frame in frame loop
  // ...

  XrFrameState frameState; // previously returned from xrWaitFrame
  const XrTime time = frameState.predictedDisplayTime;
  XrBoxf box; // populated with the bounding box of the camera frustum

  // ...

  createInfo.baseSpace = appPlaySpace;
  createInfo.time = time;
  createInfo.boundingBox = box;

  // Grab the scene mesh snapshot. This way the app can: know all of the submesh infos,
  // choose the appropriate submeshes for which to get data, allocate the necessary
  // buffer for those submeshes, and then get the data.
  CHK_XR(xrCreateSceneMeshSnapshotANDROID(tracker, &createInfo, &snapshotResult));

  // check the tracking state
  if (snapshotResult.trackingState == XR_SCENE_MESH_TRACKING_STATE_ERROR_ANDROID) {
    // unrecoverable error. Exit the app.
    if (snapshot != XR_NULL_HANDLE) {
      CHK_XR(xrDestroySceneMeshSnapshotANDROID(snapshot));
      snapshot = XR_NULL_HANDLE;
    }
    break;
  } else if (snapshotResult.trackingState != XR_SCENE_MESH_TRACKING_STATE_TRACKING_ANDROID) {
    // The tracker is not tracking. Try again later.
    if (snapshot != XR_NULL_HANDLE) {
      CHK_XR(xrDestroySceneMeshSnapshotANDROID(snapshot));
      snapshot = XR_NULL_HANDLE;
    }
    continue;
  }

  // 2-call idiom for getting submesh states
  std::vector<XrSceneSubmeshStateANDROID> states;
  uint32_t submeshCount = 0;
  CHK_XR(xrGetAllSubmeshStatesANDROID(snapshot, 0, &submeshCount, nullptr));
  states.resize(submeshCount);
  for (XrSceneSubmeshStateANDROID& state : states) {
    state = {.type = XR_TYPE_SCENE_SUBMESH_STATE_ANDROID};
  }
  CHK_XR(xrGetAllSubmeshStatesANDROID(snapshot, submeshCount, &submeshCount, states.data()));

  // To store the poses for the selected submeshes
  std::vector<XrPosef> submeshesPoses;
  submeshesPoses.reserve(submeshCount);

  std::vector<XrSceneSubmeshDataANDROID> submeshes;
  submeshes.reserve(submeshCount);
  // Iterate the states and determine which submeshes the app wants data for
  for (XrSceneSubmeshStateANDROID& state : states) {
    // Modify this bool as necessary by looking at states. Maybe state.lastUpdatedTime
    // is below a threshold or maybe the bounding box does not intersect with camera view
    // frustum (The submesh will be culled).
    bool needed;
    if (needed) {
      // Add the selected submesh to the submeshes list. Set the capacity
      // zero for now since the size of the buffer will be determined later.
      XrSceneSubmeshDataANDROID submesh = {
        .type = XR_TYPE_SCENE_SUBMESH_DATA_ANDROID,
        .submeshId = state.submeshId,
        .vertexCapacityInput = 0,
        .vertexCountOutput = 0,
        .vertexPositions = nullptr,
        .vertexNormals = nullptr,
        .vertexSemantics = nullptr,
        .indexCapacityInput = 0,
        .indexCountOutput = 0,
        .indices = nullptr,
      };
      submeshes.push_back(submesh);
      submeshesPoses.push_back(state.submeshPoseInBaseSpace);
    }
  }

  // Grab the data for the selected submeshes using the 2-call idiom.
  CHK_XR(xrGetSubmeshDataANDROID(snapshot, submeshes.size(), submeshes.data()));
  for (XrSceneSubmeshDataANDROID& submesh : submeshes) {
    submesh.vertexCapacityInput = submesh.vertexCountOutput;
    submesh.vertexCountOutput = 0;
    submesh.vertexPositions = new XrVector3f[submesh.vertexCountOutput];
    submesh.vertexNormals = new XrVector3f[submesh.vertexCountOutput];
    submesh.vertexSemantics = new uint8_t[submesh.vertexCountOutput];
    submesh.indexCapacityInput = submesh.indexCountOutput;
    submesh.indexCountOutput = 0;
    submesh.indices = new uint32_t[submesh.indexCountOutput];
  }
  CHK_XR(xrGetSubmeshDataANDROID(snapshot, submeshes.size(), submeshes.data()));

  // Destroy the scene mesh snapshot since we have finally grabbed the submeshes data. In
  // next iteration app can: create a new one to get latest mesh data
  CHK_XR(xrDestroySceneMeshSnapshotANDROID(snapshot));
  snapshot = XR_NULL_HANDLE;

  // Iterate the submeshes and get the vertex positions, vertex normals and
  // vertex semantics data for the selected submeshes
  for (uint32_t i = 0; i < submeshes.size(); i++) {
    XrSceneSubmeshDataANDROID& data = submeshes[i];
    XrVector3f* vertexPositions = data.vertexPositions;
    XrVector3f* vertexNormals = data.vertexNormals;
    XrSceneMeshSemanticLabelANDROID* vertexSemantics =
      reinterpret_cast<XrSceneMeshSemanticLabelANDROID*>(data.vertexSemantics);
    XrPosef pose = submeshesPoses[i];

    // *** Use the data as per needs ***

    // Release the allocated memory for the data buffers when done using
    delete [] data.vertexPositions;
    delete [] data.vertexNormals;
    delete [] data.vertexSemantics;
    delete [] data.indices;
  }

  // ...
  // Finish frame loop
  // ...
}

CHK_XR(xrDestroySceneMeshingTrackerANDROID(tracker));

New Object Types

New Enum Constants

XrObjectType enumeration is extended with:

  • XR_OBJECT_TYPE_SCENE_MESHING_TRACKER_ANDROID
  • XR_OBJECT_TYPE_SCENE_MESH_SNAPSHOT_ANDROID

XrStructureType enumeration is extended with:

  • XR_TYPE_SYSTEM_SCENE_MESHING_PROPERTIES_ANDROID
  • XR_TYPE_SCENE_MESHING_TRACKER_CREATE_INFO_ANDROID
  • XR_TYPE_SCENE_MESH_SNAPSHOT_CREATE_INFO_ANDROID
  • XR_TYPE_SCENE_MESH_SNAPSHOT_CREATION_RESULT_ANDROID
  • XR_TYPE_SCENE_SUBMESH_STATE_ANDROID
  • XR_TYPE_SCENE_SUBMESH_DATA_ANDROID

New Enums

New Structures

New Functions

Issues

Version History

  • Revision 3, 2025-05-15 (Salar Khan) ** Initial extension description

OpenXR™ and the OpenXR logo are trademarks owned by The Khronos Group Inc. and are registered as a trademark in China, the European Union, Japan and the United Kingdom.