Work with ARCore for Jetpack XR

ARCore for Jetpack XR allows apps to work with basic concepts of augmented reality (AR), using low-level scene understanding primitives and motion tracking. Use ARCore for Jetpack XR when building AR experiences and you need to use planar data or anchor content to a fixed location in space.

Understand a Session's lifecycle

All objects tracked by ARCore for Jetpack XR must be accessed through a Session. Similar to an Activity's lifecycle, Session objects also have a lifecycle that must be maintained according to your app's usage of a Session object's features. If your app contains a single XR-enabled activity, consider handling the lifecycle of the Session using a Lifecycle-aware component.

Create a Session

A Session must be created before it can be used. Creating a session requires the user to have granted the android.permission.SCENE_UNDERSTANDING permission to your app.

To create a session:

when (val result = Session.create(owner)) {
  is SessionCreateSuccess -> {
    session = result.session
  }
  is SessionCreatePermissionsNotGranted -> {
   // Request android.permission.SCENE_UNDERSTANDING.
  }
}

See SessionCreateResult for reasons why a Session could fail to be created.

Resume a session

Resuming a session should be done when your app is ready to handle state changes from ARCore for Jetpack XR. In many cases, this is done in your Activity's onResume() callback, but your app may want to delay processing until user interaction.

The following code snippet shows an example of resuming a session.

when (val result = session.resume()) {
  is SessionResumeSuccess -> {
    // Session has been created successfully.
    // Attach any successful handlers here.
  }
  is SessionResumePermissionsNotGranted -> {
    // Request android.permission.SCENE_UNDERSTANDING.
}

See SessionResumeResult for reasons why a Session could fail to resume.

Pause a session

When your activity goes to the background, pause the Session using Session.pause(). Pausing a session temporarily stops tracking until the session is resumed, maintaining the state of the perception system.

Destroy a session

To permanently dispose of a session, use Session.destroy(). This frees resources in use by the session and destroys all session states.

Retrieve the state of perceived planes

ARCore for Jetpack XR provides the state of planes through a StateFlow that emits the state of planes. Subscribing to planes in a session notifies your app when planes are added, updated, or removed.

Plane.subscribe(session).collect { planes ->
 // Planes have changed; update plane rendering
}

A plane has the following properties:

• label: a semantic description of a given Plane. Could be a Wall, Floor, Ceiling, or Table.
• centerPose: The pose of the center of the detected plane.
• extents: The dimensions of the detected plane, in meters.
• vertices: A list of vertices of a convex polygon that approximates the plane.

Perform a hit-test against planes

A hit-test is a method of calculating the intersection of a ray with objects tracked by the session. A common application of a hit-test is to point at a table and place an object at that location. Conducting a hit-test results in a list of hit objects. In other words, a hit-test doesn't stop at the first object hit. However, often you may only be interested in the first object hit of a given type.

To perform a hit-test, use Interaction.hitTest() with a Ray:

val results = Interaction.hitTest(session, ray)
// When interested in the first Table hit:
val tableHit = results.firstOrNull {
  val trackable = it.trackable
  trackable is Plane && trackable.state.value.label == Plane.Label.Table
}

Anchor content to a fixed location in space

To give virtual objects a position in the real-world, use an Anchor. An Anchor object helps your app keep track of a fixed location in physical space.

An anchor is created using a Pose, which can be interpreted relative to an existing Trackable or not.

Create an anchor relative to a Trackable

When an anchor is created relative to a Trackable, such as a Plane, which makes the anchor follow the attached Trackable when it moves through space.

val anchor = trackable.createAnchor(pose)

Create an anchor without a Trackable

To create an anchor that isn't attached to a Trackable:

when (val result = Anchor.create(session, pose)) {
  is AnchorCreateSuccess -> // ...
  else -> // handle failure
}

Attach an entity to an anchor

To render a model at this location, create a GltfModel and set its pose to the anchor's pose. Ensure that the model is hidden when the Anchor's TrackingState is Stopped.

// renderSession is androidx.xr.core.Session
anchor.state.collect { state ->
  if (state.trackingState == TrackingState.Tracking) {
    gltfEntity.setPose(
      renderSession.perceptionSpace.transformPoseTo(state.pose, renderSession.activitySpace)
    )
  } else if (state.trackingState == TrackingState.Stopped) {
    entity.setHidden(true)
  }
}

Understand TrackingState

Each Trackable has a TrackingState that should be checked before being used. A Trackable that has a TrackableState of Tracking has its Pose actively updated by the system. A Trackable that is Paused may become Tracking in the future, whereas one that is Stopped will never become Tracking.

Persist an Anchor throughout sessions

An anchor that is not persisted disappears after a session is destroyed. By persisting an anchor, your app remembers that anchor's position in its private app data. This anchor can be retrieved in a subsequent session and is anchored in the same location in the world.

To persist an anchor, use anchor.persist() as shown here:

val uuid = anchor.persist()

Your app can retrieve the anchor by using the UUID in a future session:

when (val result = Anchor.load(session, uuid)) {
  is AnchorCreateSuccess -> // Loading was successful. The anchor is stored in result.anchor.
  else -> // handle failure
}

When you don't need an anchor anymore, call unpersist(). This removes the anchor from your app's storage and makes the given UUID unretrievable for calls to Anchor.load().

Anchor.unpersist(session, uuid)

Your app can also request a list of all anchors that have been persisted that are still present in your app's storage:

val uuids = Anchor.getPersistedAnchorUuids(session)