XR_ANDROID_light_estimation

Name String

XR_ANDROID_light_estimation

Extension Type

Instance extension

Registered Extension Number

701

Revision

1

Ratification Status

Not ratified

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2025-03-13

IP Status

No known IP claims.

Contributors

Jared Finder, Google
Cairn Overturf, Google
Spencer Quin, Google
Levana Chen, Google
Nihav Jain, Google
Salar Khan, Google
Scott Chung, Google

Overview

This extension allows the application to request data representing the lighting of the real-world environment around the headset. The application can use this information to light virtual objects under the same conditions as the real-world scene they’re placed in.

Permissions

Android applications must have the android.permission.SCENE_UNDERSTANDING_COARSE permission listed in their manifest as this extension exposes lighting information about the environment. The android.permission.SCENE_UNDERSTANDING_COARSE permission is considered a dangerous permission. The application must request the permission at runtime to use these functions:

(protection level: dangerous)

Inspect system capability

The XrSystemLightEstimationPropertiesANDROID structure is defined as:

typedef struct XrSystemLightEstimationPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsLightEstimation;
} XrSystemLightEstimationPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsLightEstimation is an XrBool32 , indicating if the current system supports light estimation.

An application can inspect whether the system is capable of supporting light estimation by extending the XrSystemProperties with XrSystemLightEstimationPropertiesANDROID structure when calling xrGetSystemProperties .

If and only if a runtime returns XR_FALSE for supportsLightEstimation , the runtime must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateLightEstimatorANDROID .

Valid Usage (Implicit)

Create a light estimator handle

XR_DEFINE_HANDLE(XrLightEstimatorANDROID)

The light estimation data is generated by the runtime and shared with the application using the XrLightEstimatorANDROID handle.

This handle can be used to access light estimation information using other functions in this extension.

The xrCreateLightEstimatorANDROID function is defined as:

XrResult xrCreateLightEstimatorANDROID(
    XrSession                                   session,
    XrLightEstimatorCreateInfoANDROID*          createInfo,
    XrLightEstimatorANDROID*                    outHandle);

Parameter Descriptions

The application uses the xrCreateLightEstimatorANDROID function to create a light estimator.

  • The runtime must return XR_ERROR_FEATURE_UNSUPPORTED if the system does not support light estimation.
  • The runtime must return XR_ERROR_PERMISSION_INSUFFICIENT if the required permissions have not been granted to the calling app.

If an application wants to indicate to the runtime that it is done accessing light estimation data it must destroy the handle via xrDestroyLightEstimatorANDROID .

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FEATURE_UNSUPPORTED
  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_PERMISSION_INSUFFICIENT
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST

The XrLightEstimatorCreateInfoANDROID structure describes the information to create an XrLightEstimatorANDROID handle.

typedef struct XrLightEstimatorCreateInfoANDROID {
    XrStructureType    type;
    const void*        next;
} XrLightEstimatorCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

Valid Usage (Implicit)

The xrDestroyLightEstimatorANDROID function is defined as:

XrResult xrDestroyLightEstimatorANDROID(
    XrLightEstimatorANDROID                     estimator);

Parameter Descriptions

The xrDestroyLightEstimatorANDROID function releases the estimator and any underlying resources.

Valid Usage (Implicit)

Thread Safety

  • Access to estimator , and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID

Access light estimation data

The xrGetLightEstimateANDROID function is defined as:

XrResult xrGetLightEstimateANDROID(
    XrLightEstimatorANDROID                     estimator,
    const XrLightEstimateGetInfoANDROID*        input,
    XrLightEstimateANDROID*                     output);

Parameter Descriptions

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_TIME_INVALID
  • XR_ERROR_VALIDATION_FAILURE

The XrLightEstimateGetInfoANDROID structure is defined as:

typedef struct XrLightEstimateGetInfoANDROID {
    XrStructureType    type;
    const void*        next;
    XrSpace            space;
    XrTime             time;
} XrLightEstimateGetInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain.
  • space is the XrSpace defining the reference space in which the returned light direction and spherical harmonics are expressed in.
  • time is the XrTime that describes the time at which the application wishes to query the light estimation.

Valid Usage (Implicit)

The XrLightEstimateANDROID structure contains light estimation data:

typedef struct XrLightEstimateANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    XrTime                         lastUpdatedTime;
} XrLightEstimateANDROID;

Member Descriptions

For every light estimate struct if the state is XR_LIGHT_ESTIMATE_STATE_INVALID_ANDROID , then all other fields in the struct are arbitrary, meaning their values are undefined.

To obtain light estimate information for ambient light, spherical harmonics and the primary directional light, applications can chain instances of the following structures, XrAmbientLightANDROID , XrSphericalHarmonicsANDROID and XrDirectionalLightANDROID respectively to XrLightEstimateANDROID :: next .

Valid Usage (Implicit)

The XrAmbientLightANDROID structure contains light estimation data about the ambient light in the scene.

typedef struct XrAmbientLightANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    XrVector3f                     intensity;
    XrVector3f                     colorCorrection;
} XrAmbientLightANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. Valid structures include XrSphericalHarmonicsANDROID and XrDirectionalLightANDROID .
  • state is the XrLightEstimateStateANDROID representing the state of the light estimate.
  • intensity is an XrVector3f representing the intensity of the ambient light. Each component of the vector corresponds to the red, green, and blue channels respectively.

  • colorCorrection is an XrVector3f with values in gamma space. If rendering in gamma color space, multiply them component-wise against the final calculated color after rendering. If rendering in linear space, first convert the values to linear space by rising to the power 2.2, then multiply component-wise against the final calculated color after rendering.

    The purpose of pname:colorCorrection is to make a scene appear natural and
blend with the real world.

Valid Usage (Implicit)

The XrSphericalHarmonicsANDROID structure contains spherical harmonics representing the lighting of the scene.

typedef struct XrSphericalHarmonicsANDROID {
    XrStructureType                    type;
    void*                              next;
    XrLightEstimateStateANDROID        state;
    XrSphericalHarmonicsKindANDROID    kind;
    float                              coefficients[9][3];
} XrSphericalHarmonicsANDROID;

Member Descriptions

The coefficients are to be used in a special function which takes a direction as an input and outputs the light color coming from that direction. More detail available on this paper .

Valid Usage (Implicit)

The XrSphericalHarmonicsKindANDROID enumeration identifies to the runtime, what type of spherical harmonics the application is requesting.

typedef enum XrSphericalHarmonicsKindANDROID {
    XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID = 0,
    XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID = 1,
    XR_SPHERICAL_HARMONICS_KIND_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSphericalHarmonicsKindANDROID;

The enumeration values have the following meanings:

Enum Description

XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID

The spherical harmonic coefficients represent the environment light radiance function, excluding the contribution from the main light.

XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID

The spherical harmonic coefficients represent the environment light radiance function, including the contribution from the main light.

The XrDirectionalLightANDROID structure contains light estimation data.

typedef struct XrDirectionalLightANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    XrVector3f                     intensity;
    XrVector3f                     direction;
} XrDirectionalLightANDROID;

Member Descriptions

Valid Usage (Implicit)

The XrLightEstimateStateANDROID enumeration indicates the state of the light estimate returned by the runtime.

typedef enum XrLightEstimateStateANDROID {
    XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID = 0,
    XR_LIGHT_ESTIMATE_STATE_INVALID_ANDROID = 1,
    XR_LIGHT_ESTIMATE_STATE_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrLightEstimateStateANDROID;

The enumeration values have the following meanings:

Enum Description

XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID

The light estimate is valid

XR_LIGHT_ESTIMATE_STATE_INVALID_ANDROID

The light estimate is invalid

Example code for light estimation

The following example code demonstrates how to get all possible light estimation quantities from the runtime

XrSession session;  // Created at app startup
XrSpace appSpace;   // Created previously.
PFN_xrCreateLightEstimatorANDROID xrCreateLightEstimatorANDROID; // Created previously.
PFN_xrDestroyLightEstimatorANDROID xrDestroyLightEstimatorANDROID; // Created previously.
PFN_xrGetLightEstimateANDROID xrGetLightEstimateANDROID; // Created previously.

XrLightEstimatorANDROID estimator;
XrLightEstimatorCreateInfoANDROID createInfo = {
    .type = XR_TYPE_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID};
CHK_XR(xrCreateLightEstimatorANDROID(session, &createInfo, &estimator));

// Every frame
XrTime updateTime;  // Time used for the current frame's simulation update.

XrLightEstimateGetInfoANDROID info = {
    .type = XR_TYPE_LIGHT_ESTIMATE_GET_INFO_ANDROID,
    .space = appSpace,
    .time = updateTime,
};

XrDirectionalLightANDROID directionalLight = {
    .type = XR_TYPE_DIRECTIONAL_LIGHT_ANDROID,
};

XrSphericalHarmonicsANDROID sphericalHarmonics = {
    .type = XR_TYPE_SPHERICAL_HARMONICS_ANDROID,
    .next = &directionalLight,
};

// Querying both TOTAL or AMBIENT spherical harmonics in one call will result in an error because chaining two structs with the same type is not allowed
bool useTotalSH;
if (useTotalSH) {
  sphericalHarmonics.kind = XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID;
} else {
  sphericalHarmonics.kind = XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID;
}

XrAmbientLightANDROID ambientLight = {
    .type = XR_TYPE_AMBIENT_LIGHT_ANDROID,
    .next = &sphericalHarmonics,
};

XrLightEstimateANDROID estimate = {
    .type = XR_TYPE_LIGHT_ESTIMATE_ANDROID,
    .next = &ambientLight,
};

XrResult result = xrGetLightEstimateANDROID(estimator, &info, &estimate);
if (result == XR_SUCCESS &&
    estimate.state == XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID) {
  // use directionalLight, totalSh, ambientSh, ambientLight if each
  // struct has a valid state field
}

// When you want to disable light estimation
CHK_XR(xrDestroyLightEstimatorANDROID(estimator));

New Object Types

New Commands

New Structures

New Enums

New Enum Constants

  • XR_ANDROID_LIGHT_ESTIMATION_EXTENSION_NAME
  • XR_ANDROID_light_estimation_SPEC_VERSION

  • Extending XrObjectType :

    • XR_OBJECT_TYPE_LIGHT_ESTIMATOR_ANDROID

  • Extending XrStructureType :

    • XR_TYPE_AMBIENT_LIGHT_ANDROID
    • XR_TYPE_DIRECTIONAL_LIGHT_ANDROID
    • XR_TYPE_LIGHT_ESTIMATE_ANDROID
    • XR_TYPE_LIGHT_ESTIMATE_GET_INFO_ANDROID
    • XR_TYPE_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID
    • XR_TYPE_SPHERICAL_HARMONICS_ANDROID
    • XR_TYPE_SYSTEM_LIGHT_ESTIMATION_PROPERTIES_ANDROID

Issues

Version History

  • Revision 1, 2024-09-16 (Cairn Overturf)

    • Initial extension description