XR_ANDROID_depth_texture

Name String

XR_ANDROID_depth_texture

Extension Type

Instance extension

Registered Extension Number

703

Revision

1

Ratification Status

Not ratified

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-11

IP Status

No known IP claims.

Contributors

Sushant Kulkarni, Google
Cairn Overturf, Google
Spencer Quin, Google
Levana Chen, Google

Overview

This extension allows the application to request depth maps of the real-world environment around the headset and query supported depth resolutions at creation.

This extension is intended to expose raw and smooth depth for occlusion, hit tests and other specific tasks that leverage accurate scene geometry, for example, counterfeit face detection.

Note

The depth maps are generated by the runtime and shared with the application using XrDepthSwapchainANDROID . The runtime must not change the depth image resolutions returned via xrEnumerateDepthResolutionsANDROID throughout the course of the application’s lifetime.

Permissions

This extension expose a downsampled depth texture to mitigate Personally Identifiable Information (PII) concerns. Android applications must have the android.permission.SCENE_UNDERSTANDING_FINE permission listed in their manifest as this extension exposes the geometry of the environment. The android.permission.SCENE_UNDERSTANDING_FINE permission is considered a dangerous permission. The application must request the permission at runtime to use these functions:

(protection level: dangerous)

Inspect system capability

The XrSystemDepthTrackingPropertiesANDROID structure is defined as:

typedef struct XrSystemDepthTrackingPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsDepthTracking;
} XrSystemDepthTrackingPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsDepthTracking is an XrBool32 indicating if the current system supports depth tracking.

An application can inspect whether the system is capable of depth tracking by extending the XrSystemProperties with XrSystemDepthTrackingPropertiesANDROID structure when calling xrGetSystemProperties .

If and only if a runtime returns XR_FALSE for supportsDepthTracking , the runtime must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateDepthSwapchainANDROID .

Valid Usage (Implicit)

Query depth resolutions

The xrEnumerateDepthResolutionsANDROID function is defined as:

XrResult xrEnumerateDepthResolutionsANDROID(
    XrSession                                   session,
    uint32_t                                    resolutionCapacityInput,
    uint32_t*                                   resolutionCountOutput,
    XrDepthCameraResolutionANDROID*             resolutions);

Parameter Descriptions

  • session is the XrSession that enumerates the supported depth resolutions.
  • resolutionCapacityInput is the capacity of the resolutions , or 0 to retrieve the required capacity.
  • resolutionCountOutput is a pointer to the count of uint64_t resolutions written, or a pointer to the required capacity in the case that resolutionCapacityInput is insufficient.
  • resolutions is a pointer to an array of XrDepthCameraResolutionANDROID , but can be NULL if resolutionCapacityInput is 0 .
  • See the Buffer Size Parameters section for a detailed description of retrieving the required resolutions size.

xrEnumerateDepthResolutionsANDROID enumerates the depth resolutions supported by the current session. Depth resolutions should be in order from highest to lowest runtime preference. The application should use the highest preference that it supports for optimal performance and quality.

Runtimes must always return identical buffer contents from this enumeration for the lifetime of the session.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_VALIDATION_FAILURE

The XrDepthCameraResolutionANDROID enum describes the supported depth resolutions when creating an XrDepthSwapchainANDROID .

typedef enum XrDepthCameraResolutionANDROID {
    XR_DEPTH_CAMERA_RESOLUTION_80x80_ANDROID = 0,
    XR_DEPTH_CAMERA_RESOLUTION_160x160_ANDROID = 1,
    XR_DEPTH_CAMERA_RESOLUTION_320x320_ANDROID = 2,
    XR_DEPTH_CAMERA_RESOLUTION_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrDepthCameraResolutionANDROID;

Enumerant Descriptions

  • XR_DEPTH_CAMERA_RESOLUTION_80x80_ANDROID — The resolution of the depth and confidence images is 80x80.
  • XR_DEPTH_CAMERA_RESOLUTION_160x160_ANDROID — The resolution of the depth and confidence images is 160x160.
  • XR_DEPTH_CAMERA_RESOLUTION_320x320_ANDROID — The resolution of the depth and confidence images is 320x320.

Create a depth swapchain

XR_DEFINE_HANDLE(XrDepthSwapchainANDROID)

An XrDepthSwapchainANDROID is a depth swapchain handle.

The xrCreateDepthSwapchainANDROID function is defined as:

XrResult xrCreateDepthSwapchainANDROID(
    XrSession                                   session,
    const XrDepthSwapchainCreateInfoANDROID*    createInfo,
    XrDepthSwapchainANDROID*                    swapchain);

Parameter Descriptions

The application can use xrCreateDepthSwapchainANDROID function to create a depth swapchain which manages both depth and confidence images.

  • The runtime must return XR_ERROR_FEATURE_UNSUPPORTED if the system does not support depth tracking.
  • The runtime must return XR_ERROR_PERMISSION_INSUFFICIENT if the required permissions have not been granted to the calling app.
  • The runtime must return XR_ERROR_VALIDATION_FAILURE if the depth resolution specified in the XrDepthSwapchainCreateInfoANDROID is unsupported.
  • The runtime should only create depth confidence images when the corresponding bits are set for the createInfo at swapchain creation.

The returned depth swapchain handle may be subsequently used in API calls. The XrDepthSwapchainANDROID handle must be eventually freed via the xrDestroyDepthSwapchainANDROID function.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FEATURE_UNSUPPORTED
  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_PERMISSION_INSUFFICIENT
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST

The XrDepthSwapchainCreateInfoANDROID structure is defined as:

typedef struct XrDepthSwapchainCreateInfoANDROID {
    XrStructureType                       type;
    const void*                           next;
    XrDepthCameraResolutionANDROID        resolution;
    XrDepthSwapchainCreateFlagsANDROID    createFlags;
} XrDepthSwapchainCreateInfoANDROID;

Member Descriptions

The XrDepthSwapchainCreateInfoANDROID structure provides creation options for the XrDepthSwapchainANDROID when passed to xrCreateDepthSwapchainANDROID .

Valid Usage (Implicit)

The XrDepthSwapchainCreateFlagsANDROID specifies creation options for XrDepthSwapchainANDROID .

typedef XrFlags64 XrDepthSwapchainCreateFlagsANDROID;

Valid bits for XrDepthSwapchainCreateFlagsANDROID are defined by XrDepthSwapchainCreateFlagBitsANDROID , which is specified as:

// Flag bits for XrDepthSwapchainCreateFlagsANDROID
static const XrDepthSwapchainCreateFlagsANDROID XR_DEPTH_SWAPCHAIN_CREATE_SMOOTH_DEPTH_IMAGE_BIT_ANDROID = 0x00000001;
static const XrDepthSwapchainCreateFlagsANDROID XR_DEPTH_SWAPCHAIN_CREATE_SMOOTH_CONFIDENCE_IMAGE_BIT_ANDROID = 0x00000002;
static const XrDepthSwapchainCreateFlagsANDROID XR_DEPTH_SWAPCHAIN_CREATE_RAW_DEPTH_IMAGE_BIT_ANDROID = 0x00000004;
static const XrDepthSwapchainCreateFlagsANDROID XR_DEPTH_SWAPCHAIN_CREATE_RAW_CONFIDENCE_IMAGE_BIT_ANDROID = 0x00000008;

Flag Descriptions

  • XR_DEPTH_SWAPCHAIN_CREATE_SMOOTH_DEPTH_IMAGE_BIT_ANDROID — Indicates the swapchain will provide smooth depth imagess.
  • XR_DEPTH_SWAPCHAIN_CREATE_SMOOTH_CONFIDENCE_IMAGE_BIT_ANDROID — Indicates the swapchain will provide smooth depth confidence imagess.
  • XR_DEPTH_SWAPCHAIN_CREATE_RAW_DEPTH_IMAGE_BIT_ANDROID — Indicates the swapchain will provide raw depth imagess.
  • XR_DEPTH_SWAPCHAIN_CREATE_RAW_CONFIDENCE_IMAGE_BIT_ANDROID — Indicates the swapchain will provide raw depth confidence imagess.

The xrDestroyDepthSwapchainANDROID function is defined as:

XrResult xrDestroyDepthSwapchainANDROID(
    XrDepthSwapchainANDROID                     swapchain);

Parameter Descriptions

The xrDestroyDepthSwapchainANDROID function destroys the depth swapchain. After this call the runtime may free all related memory and resources.

Valid Usage (Implicit)

Thread Safety

  • Access to swapchain , and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID

Access depth textures

The xrEnumerateDepthSwapchainImagesANDROID function is defined as:

XrResult xrEnumerateDepthSwapchainImagesANDROID(
    XrDepthSwapchainANDROID                     depthSwapchain,
    uint32_t                                    depthImageCapacityInput,
    uint32_t*                                   depthImageCountOutput,
    XrDepthSwapchainImageANDROID*               depthImages);

Parameter Descriptions

  • depthSwapchain is the XrDepthSwapchainANDROID to get images from.
  • depthImageCapacityInput is the capacity of the depthImages array, or 0 to indicate a request to retrieve the required capacity.
  • depthImageCountOutput is a pointer to the count of depthImages written, or a pointer to the required capacity in the case that depthImageCapacityInput is insufficient.
  • depthImages is a pointer to an array of XrDepthSwapchainImageANDROID structures. It can be NULL if depthImageCapacityInput is 0.
  • See the Buffer Size Parameters section for a detailed description of retrieving the required depthImages size.

xrEnumerateDepthSwapchainImagesANDROID fills an array of XrDepthSwapchainImageANDROID structures. The resources must be constant and valid for the lifetime of the XrDepthSwapchainANDROID . This function behaves analogously to xrEnumerateSwapchainImages .

The runtime must always return identical buffer contents from this enumeration for the lifetime of the swapchain.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_VALIDATION_FAILURE

The XrDepthSwapchainImageANDROID structure is defined as:

typedef struct XrDepthSwapchainImageANDROID {
    XrStructureType    type;
    void*              next;
    const float*       rawDepthImage;
    const uint8_t*     rawDepthConfidenceImage;
    const float*       smoothDepthImage;
    const uint8_t*     smoothDepthConfidenceImage;
} XrDepthSwapchainImageANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • rawDepthImage is NULL or the pointer to raw depth images for both left and right views allocated by the runtime. The values have units of meters. Special values: 0.0 indicates an invalid or empty depth pixel in the raw depth, Inf indicates known depth that is effectively infinitely far away,
  • rawDepthConfidenceImage is NULL or the pointer to raw depth confidence images for both left and right views allocated by the runtime.
  • smoothDepthImage is NULL or the pointer to smooth depth images for both left and right views allocated by the runtime. The values have units of meters. Special values: 0.0 indicates an invalid or empty depth pixel in the smooth depth, Inf indicates known depth that is effectively infinitely far away.
  • smoothDepthConfidenceImage is NULL or the pointer to smooth depth confidence images for both left and right views allocated by the runtime.

The image buffer pointers will only be valid during the life of the swapchain. Once the swapchain is destroyed, the pointers will be considered dangling.

XrDepthSwapchainImageANDROID represents the depth images from a readable XrDepthSwapchainANDROID , allocated as described in the XrDepthSwapchainCreateInfoANDROID :: resolution and XrDepthSwapchainCreateInfoANDROID :: createFlags while calling xrCreateDepthSwapchainANDROID . For each depth image:

Valid Usage (Implicit)

The xrAcquireDepthSwapchainImagesANDROID function is defined as:

XrResult xrAcquireDepthSwapchainImagesANDROID(
    XrDepthSwapchainANDROID                     depthSwapchain,
    const XrDepthAcquireInfoANDROID*            acquireInfo,
    XrDepthAcquireResultANDROID*                acquireResult);

Parameter Descriptions

Applications can use xrAcquireDepthSwapchainImagesANDROID function to acquire the latest available swapchain image index, i.e. XrDepthAcquireResultANDROID :: acquiredIndex , into the XrDepthSwapchainImageANDROID array enumerated by xrEnumerateDepthSwapchainImagesANDROID . The returned XrDepthAcquireResultANDROID also contains other information such as the field of view and pose that are necessary to interpret the depth data. It is safe to read from the acquired slot in the image array until the next call to xrAcquireDepthSwapchainImagesANDROID .

There must be no more than one call to xrAcquireDepthSwapchainImagesANDROID between any pair of corresponding xrBeginFrame and xrEndFrame calls in a session.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_CALL_ORDER_INVALID
  • XR_ERROR_DEPTH_NOT_AVAILABLE_ANDROID
  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_TIME_INVALID
  • XR_ERROR_VALIDATION_FAILURE

The XrDepthAcquireInfoANDROID structure is defined as:

typedef struct XrDepthAcquireInfoANDROID {
    XrStructureType    type;
    const void*        next;
    XrSpace            space;
    XrTime             displayTime;
} XrDepthAcquireInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • space is an XrSpace defining the reference frame of the returned pose in XrDepthAcquireResultANDROID :: views .
  • displayTime is an XrTime specifying the time used to compute the pose for the returned pose in XrDepthAcquireResultANDROID :: views . Applications should pass their predicted display time for the current frame.

Valid Usage (Implicit)

The XrDepthAcquireResultANDROID structure is defined as:

typedef struct XrDepthAcquireResultANDROID {
    XrStructureType       type;
    const void*           next;
    uint32_t              acquiredIndex;
    XrTime                exposureTimestamp;
    XrDepthViewANDROID    views[2];
} XrDepthAcquireResultANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • acquiredIndex is the index of the acquired texture into the XrDepthSwapchainImageANDROID array enumerated by xrEnumerateDepthSwapchainImagesANDROID .
  • exposureTimestamp is the XrTime specifying the time at which the depth map was captured.
  • views is an array of two XrDepthViewANDROID , one for each eye, where index 0 is left eye and index 1 is the right eye.

Valid Usage (Implicit)

The XrDepthViewANDROID structure is defined as:

typedef struct XrDepthViewANDROID {
    XrStructureType    type;
    const void*        next;
    XrFovf             fov;
    XrPosef            pose;
} XrDepthViewANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • fov is an XrFovf specifying the field of view used to generate this view. The view is never flipped horizontally nor vertically.
  • pose is an XrPosef specifying the pose from which the depth map was rendered. The reference frame is specified in XrDepthAcquireInfoANDROID .

Valid Usage (Implicit)

Example code for depth tracking

The following example code demonstrates how to acquire depth images and use depth values to map depth image coordinates to stage space.

XrInstance instance;  // previously initialized
XrSystemId systemId;  // previously initialized
XrSession session; // previously initialized
XrSpace stageSpace; // space created for XR_REFERENCE_SPACE_TYPE_STAGE.

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateDepthSwapchainANDROID xrCreateDepthSwapchainANDROID; // previously initialized
PFN_xrDestroyDepthSwapchainANDROID xrDestroyDepthSwapchainANDROID; // previously initialized
PFN_xrEnumerateDepthSwapchainImagesANDROID xrEnumerateDepthSwapchainImagesANDROID; // previously initialized
PFN_xrEnumerateDepthResolutionsANDROID xrEnumerateDepthResolutionsANDROID; // previously initialized
PFN_xrAcquireDepthSwapchainImagesANDROID xrAcquireDepthSwapchainImagesANDROID; // previously initialized

float tanf(float);

// Inspect system capability
XrSystemProperties properties{XR_TYPE_SYSTEM_PROPERTIES};
XrSystemDepthTrackingPropertiesANDROID depthTrackingProperties{XR_TYPE_SYSTEM_DEPTH_TRACKING_PROPERTIES_ANDROID};
properties.next = &depthTrackingProperties;
CHK_XR(xrGetSystemProperties(instance, systemId, &properties));
if (!depthTrackingProperties.supportsDepthTracking) {
  // depth tracking is not supported.
  return;
}

// Query the supported depth resolution.
XrDepthCameraResolutionANDROID supportedDepthResolution;
uint32_t supportedResolutionCount = 0;
CHK_XR(xrEnumerateDepthResolutionsANDROID(
    session, 1, &supportedResolutionCount, &supportedDepthResolution));

// Define metadata to access the raw and smooth depth along with confidences.
XrDepthSwapchainCreateInfoANDROID swapchainCreateInfo = {
  .type = XR_TYPE_DEPTH_SWAPCHAIN_CREATE_INFO_ANDROID,
  .next = nullptr,
  // Use the resolution supported by the runtime.
  .resolution = supportedDepthResolution,
  .createFlags =
    XR_DEPTH_SWAPCHAIN_CREATE_SMOOTH_DEPTH_IMAGE_BIT_ANDROID |
    XR_DEPTH_SWAPCHAIN_CREATE_SMOOTH_CONFIDENCE_IMAGE_BIT_ANDROID |
    XR_DEPTH_SWAPCHAIN_CREATE_RAW_DEPTH_IMAGE_BIT_ANDROID |
    XR_DEPTH_SWAPCHAIN_CREATE_RAW_CONFIDENCE_IMAGE_BIT_ANDROID,
};

XrDepthSwapchainANDROID depthSwapchain;
CHK_XR(xrCreateDepthSwapchainANDROID(
    session, &swapchainCreateInfo, &depthSwapchain));

// Enumerate depth images.
uint32_t imageCountOutput = 0;
CHK_XR(xrEnumerateDepthSwapchainImagesANDROID(
    depthSwapchain, 0, &imageCountOutput, nullptr));
std::vector<XrDepthSwapchainImageANDROID> depthImages(imageCountOutput);
for (int i = 0; i < imageCountOutput; i++) {
  depthImages[i].type = XR_TYPE_DEPTH_SWAPCHAIN_IMAGE_ANDROID;
}
CHK_XR(xrEnumerateDepthSwapchainImagesANDROID(
  depthSwapchain, imageCountOutput, &imageCountOutput, depthImages.data()));

while (1) {
    // ...
    // For every frame in frame loop
    // ...

    XrFrameState frameState;  // previously returned from xrWaitFrame
    const XrTime time = frameState.predictedDisplayTime;

    XrDepthAcquireInfoANDROID acquireInfo = {
        .type = XR_TYPE_DEPTH_ACQUIRE_INFO_ANDROID,
        .space = stageSpace,
        .displayTime = time
    };
    XrDepthAcquireResultANDROID acquireResult = {
        .type = XR_TYPE_DEPTH_ACQUIRE_RESULT_ANDROID,
    };
    CHK_XR(xrAcquireDepthSwapchainImagesANDROID(
        depthSwapchain, &acquireInfo, &acquireResult));

    // Each value in a depth image corresponds to a point in the real world.
    // The sample code below shows how to find the stageSpace position of
    // the point corresponding to a particular value in the depth image.

    // For this sample code, assume we are using a right handed coordinate system
    // with +X to the right, +Y up and -Z forward.

    XrDepthSwapchainImageANDROID *image =
        &depthImages[acquireResult.acquiredIndex];

    // Assume supported resolution is XR_DEPTH_CAMERA_RESOLUTION_160x160_ANDROID.
    const int imageResolution = 160;
    int imageY = 80; // value in [0, imageResolution)
    int imageX = 80; // value in [0, imageResolution)

    // Get depth value from left eye.
    // A right depth value is obtained with the following expression:
    // depthR = image->rawDepthImage[imageResolution*imageResolution+i*imageResolution+j]
    float depthL = image->rawDepthImage[imageY*imageResolution + imageX];
    XrDepthViewANDROID viewL = acquireResult.views[0];

    float tanL = tanf(viewL.fov.angleLeft);
    float tanR = tanf(viewL.fov.angleRight);
    float tanU = tanf(viewL.fov.angleUp);
    float tanD = tanf(viewL.fov.angleDown);

    float s = (imageX + 0.5f) / (float)imageResolution;
    float t = (imageY + 0.5f) / (float)imageResolution;

    // Calculate the depth camera space position of the point
    // corresponding to this depth value.
    XrVector3f posInCameraSpace;
    posInCameraSpace.z = -depthL;
    posInCameraSpace.x = (tanL + (tanR - tanL)*s)*depthL;
    posInCameraSpace.y = (tanD + (tanU - tanD)*t)*depthL;

    XrPosef depthCameraPoseL = viewL.pose;
    // Transform posInCameraSpace by depthCameraPoseL

    // ...
    // Finish frame loop
    // ...
}

New Object Types

New Commands

New Structures

New Enums

New Bitmasks

New Enum Constants

  • XR_ANDROID_DEPTH_TEXTURE_EXTENSION_NAME
  • XR_ANDROID_depth_texture_SPEC_VERSION

  • Extending XrObjectType :

    • XR_OBJECT_TYPE_DEPTH_SWAPCHAIN_ANDROID

  • Extending XrResult :

    • XR_ERROR_DEPTH_NOT_AVAILABLE_ANDROID

  • Extending XrStructureType :

    • XR_TYPE_DEPTH_ACQUIRE_INFO_ANDROID
    • XR_TYPE_DEPTH_ACQUIRE_RESULT_ANDROID
    • XR_TYPE_DEPTH_SWAPCHAIN_CREATE_INFO_ANDROID
    • XR_TYPE_DEPTH_SWAPCHAIN_IMAGE_ANDROID
    • XR_TYPE_DEPTH_VIEW_ANDROID
    • XR_TYPE_SYSTEM_DEPTH_TRACKING_PROPERTIES_ANDROID

Issues

Version History

  • Revision 1, 2024-09-09 (Levana Chen)

    • Initial extension description