Add spatial video to your app

The Jetpack XR SDK supports the playback of stereoscopic side-by-side video onto flat surfaces. With stereoscopic video, each frame consists of a left-eye and a right-eye image to give viewers a sense of depth—also known as stereopsis.

You can render non-stereoscopic 2D video on Android XR apps with the standard media APIs used for Android development on other form factors.

Play side-by-side video using Jetpack SceneCore

With side-by-side video, each stereoscopic frame is presented as two images arranged horizontally adjacent to each other. Top-and-bottom video frames are arranged vertically adjacent to each other.

Side-by-side video is not a codec but rather a way of organizing stereoscopic frames, which means it can be encoded in any of the codecs supported by Android.

You can load side-by-side video using Media3 Exoplayer and then render it using the new SurfaceEntity. To create a SurfaceEntity, call SurfaceEntity.create, as shown in the following example.

val stereoSurfaceEntity = SurfaceEntity.create(
    xrSession,
    SurfaceEntity.StereoMode.SIDE_BY_SIDE,
    Pose(Vector3(0.0f, 0.0f, -1.5f)),
    SurfaceEntity.CanvasShape.Quad(1.0f, 1.0f)
)
val videoUri = Uri.Builder()
    .scheme(ContentResolver.SCHEME_ANDROID_RESOURCE)
    .path("sbs_video.mp4")
    .build()
val mediaItem = MediaItem.fromUri(videoUri)

val exoPlayer = ExoPlayer.Builder(this).build()
exoPlayer.setVideoSurface(stereoSurfaceEntity.getSurface())
exoPlayer.setMediaItem(mediaItem)
exoPlayer.prepare()
exoPlayer.play()
SpatialVideo.kt

Play MV-HEVC video using Jetpack SceneCore

The MV-HEVC codec standard is optimized and designed for stereoscopic video, allowing your app to efficiently play back immersive videos at great quality. MV-HEVC files have a primary stream, usually the left eye, and a stereo stream with the other eye.

Similar to side-by-side video, you can load it using Media3 Exoplayer and render it using the SurfaceEntity. You will want to specify whether your MV-HEVC file is left or right primary in the stereoMode parameter when calling SurfaceEntity.create.

// Create the SurfaceEntity with the StereoMode corresponding to the MV-HEVC content
val stereoSurfaceEntity = SurfaceEntity.create(
    xrSession,
    SurfaceEntity.StereoMode.MULTIVIEW_LEFT_PRIMARY,
    Pose(Vector3(0.0f, 0.0f, -1.5f)),
    SurfaceEntity.CanvasShape.Quad(1.0f, 1.0f)
)
val videoUri = Uri.Builder()
    .scheme(ContentResolver.SCHEME_ANDROID_RESOURCE)
    .path("mvhevc_video.mp4")
    .build()
val mediaItem = MediaItem.fromUri(videoUri)

val exoPlayer = ExoPlayer.Builder(this).build()
exoPlayer.setVideoSurface(stereoSurfaceEntity.getSurface())
exoPlayer.setMediaItem(mediaItem)
exoPlayer.prepare()
exoPlayer.play()
SpatialVideo.kt

Play 180-degree and 360-degree video using Jetpack SceneCore

SurfaceEntity supports playback of 180° videos on hemispherical surfaces and 360° videos on spherical surfaces. The radius parameter refers to the radial size of the respective surfaces in meters by default.

The following code shows how to set up SurfaceEntity for playback on a 180° hemisphere and a 360° sphere. When using these canvas shapes, position the surface by leveraging the user's head pose to provide an immersive experience.

// Set up the surface for playing a 180° video on a hemisphere.
val hemisphereStereoSurfaceEntity =
    SurfaceEntity.create(
        xrSession,
        SurfaceEntity.StereoMode.SIDE_BY_SIDE,
        xrSession.scene.spatialUser.head?.transformPoseTo(
            Pose.Identity,
            xrSession.scene.activitySpace
        )!!,
        SurfaceEntity.CanvasShape.Vr180Hemisphere(1.0f),
    )
// ... and use the surface for playing the media.
SpatialVideo.kt

// Set up the surface for playing a 360° video on a sphere.
val sphereStereoSurfaceEntity =
    SurfaceEntity.create(
        xrSession,
        SurfaceEntity.StereoMode.TOP_BOTTOM,
        xrSession.scene.spatialUser.head?.transformPoseTo(
            Pose.Identity,
            xrSession.scene.activitySpace
        )!!,
        SurfaceEntity.CanvasShape.Vr360Sphere(1.0f),
    )
// ... and use the surface for playing the media.
SpatialVideo.kt

Play spatial video using Jetpack Compose for XR

If you're interested in learning how to play video using Jetpack Compose for XR, learn how to Add a surface for image or video content.