# Elevate immersive experiences with 3D Engines in Jetpack Compose for XR
- Speakers: Luke Hopkins (Android XR Developer Relations Engineer), Joshua Lorett (Android Staff Engineer), Rodrigo Sanchez (Software Developer - Beyond Sports)
- Track: Android XR
- Video: https://www.youtube.com/watch?v=u8dsUtPvOlY
- Page: https://developer.android.com/events/dev-summit/talks/elevate-immersive-experiences-with-3d-engines-in-jetpack-compose-for-xr
## About the talk
Learn how to integrate 3D game engines (Godot & Unity) with Jetpack Compose into a single codebase for mobile and Android XR. We'll explore embedding 3D scenes into spatial containers, setting up Kotlin-to-engine communication —featuring an exclusive look into how the NBA built their live basketball experience on Android XR.
## Summary
- The OpenXR Spatial Containers extension allows 3D engines to share space with native Android UI. Instead of taking over the skybox, real-time engines run inside a managed 3D volume alongside Jetpack XR spatial panels.
- Developers can embed real-time engines directly into Jetpack Compose applications as AAR libraries. This approach lets native Android handle business logic and UI while the engine renders 3D components.
- Embedding Godot requires an AAR dependency and an AAPT build rule to keep hidden assets. A `GodotPlugin` subclass then handles bidirectional communication with Kotlin.
- Compose UI loops must stay decoupled from continuous engine tick loops. Blindly forwarding engine updates triggers constant recomposition and causes UI lag.
- The NBA app integrates Unity as an embedded library in a separate process using AIDL for inter-process communication, protecting the host app from crashes.
## Resources
- [Android XR developer documentation](https://developer.android.com/develop/xr)
## Topics
android-xr, jetpack-compose, spatial-containers, openxr, godot, unity
## Transcript
### Introduction to Android XR
[00:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=4s) Hello, everyone.
[00:05](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=5s) My name is Luke Hopkins.
[00:07](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=7s) I'm a developer relations engineer here on the Android XR team.
[00:11](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=11s) In today's talk, we're going to be talking about how you can embed game engines, such as Godot and Unity, into your Android project.
[00:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=18s) Later on, I'll be joined by Josh and Rodrigo from NBA and Beyond Sports, who will discuss how they've been implementing Unity as a library into their NBA application.
[00:28](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=28s) In this session, we're going to dive I'm going to dive into how to integrate Godot into your Jetpack Compose application as well, and then Josh and Rodrigo will go through their NBA application.
[00:39](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=39s) Android XR is an extension of the Android ecosystem, expanding upon a wide range of form factors to bring experiences you know and love into a new dimension.
[00:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=48s) We've made it intuitive to bring your existing XR and Android apps into this new space, and we're leveraging existing Android technologies that you're already building with today.
[00:59](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=59s) Android XR is a unified platform designed for a wide range of devices.
[01:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=64s) Android XR devices power different types of experiences.
[01:08](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=68s) We have augmented experiences for audio and Display Glasses, and then we have immersive experiences for XR headsets and wired XR glasses.
[01:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=78s) Today, we'll be discussing building immersive experiences for wired XR glasses and XR headsets.
[01:24](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=84s) Creating immersive experiences can be done via our Jetpack XR SDK or using `OpenXR` with an with a real-time engine.
[01:34](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=94s) We support Unity, Godot, and Unreal so you can build your immersive experiences for Android no matter what tool you use.
[01:42](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=102s) In this talk, I'm going to explain how you can leverage Jetpack's `SceneCore` API, alongside Godot and a new `OpenXR` extension, to make this possible.
[01:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=114s) Here you can see Godot running side-by-side with Jetpack's spatial panels running at the same time.
[02:00](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=120s) Notice how Godot is rendered as a spatial content, providing a 3D volume alongside 2D Compose UI within a spatial container.
[02:10](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=130s) This approach enables developers to run physics simulations, custom shaders using their real-time engine of choice, whilst applying native Android development practices to the rest of your application.
[02:27](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=147s) With Android XR and XREAL AURA, you too can create experiences like this.
[02:31](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=151s) And in this talk, we'll teach you how to build this experience using Godot and Android Studio, and then you hear from our partners at NBA and Beyond Sports on how they built it using Unity as a library.
### Spatial containers
[02:43](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=163s) Historically, building an XR experience with `OpenXR` has meant building an experience that renders every single pixel.
[02:50](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=170s) This means if a developer needs to display a second activity, they could not have both at once.
[02:56](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=176s) To solve this, Khronos, the OpenXR working group, in collaboration with several industry leaders, have released `OpenXR` Spatial Containers extension.
[03:05](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=185s) Think of Spatial Containers as `OpenXR` in a box.
[03:08](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=188s) Instead of your engine owning the whole skybox, it is granted a system-managed 3D volume that can live side-by-side with Jetpack's XR spatial panels, other Spatial Containers, and standard Android activities.
[03:22](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=202s) Let's say you want to make use of Android XR by displaying two spatial panels at once, whilst running an immersive game.
[03:28](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=208s) You render everything in the engine, and it makes it possible in these use cases to have the Godot editor and a spatial container running at the same time.
[03:38](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=218s) As of Spatial API level 4, Google has implemented Spatial Containers into Android XR.
[03:44](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=224s) Spatial Containers support two crucial display modes.
[03:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=228s) First is bounded mode.
[03:50](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=230s) This provides a defined 3D volume with several built-in avoidances.
[03:53](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=233s) A user can grab your spatial container, they can rotate it, position it, grab it, minimize it, and it supports stereo depth projection, giving you that magic window or that 3D pop-out effect that you have whilst clipping the container's edges too.
[04:10](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=250s) The second is unbounded mode.
[04:12](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=252s) If the user decides they want full immersion, your app can request a seamless transition from its bounded mode into full space mode, and then you can also drop back into bounded mode.
[04:24](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=264s) With the addition of Spatial Containers, Android Android XR opens the door for new and expansive ideas.
[04:30](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=270s) Let's say you want to bring your Android UI into your game, or maybe you want to create a custom shader for displaying 3D content in your application.
[04:38](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=278s) Before the introduction of Spatial Containers, it's impossible to have both Compose and a game engine running at the same time.
[04:45](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=285s) And when looking at application development in real-time engines, we've effectively lived in two different paradigms as well.
[04:52](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=292s) For native Android developers, your application runtime is a built around Jetpack Compose.
[04:56](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=296s) It's designed for rapid, reactive UI, battery efficiency, dynamic layouts, system accessibility, and seamless text input.
[05:05](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=305s) It idles when nothing happens, and it's predictable.
[05:09](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=309s) For game engine developers, the domain is real-time rendering, where tools like Godot and Unity shine.
[05:15](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=315s) They don't just wait around for user interaction.
[05:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=318s) They run a continuous tick and render loop.
[05:20](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=320s) So this can be at 60, 90, 100 frame rates per second.
[05:24](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=324s) They do physics calculations, complex shader shaders, skeletal animations, spatial audio, and all of this with incredible performance.
### Approaches for integrating 3D engines
[05:33](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=333s) So what happens if you want the best of both worlds?
[05:35](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=335s) Previously developers had to implement one of these two approaches.
[05:39](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=339s) Option A is rewriting your entire application inside a real engine like Godot or Unity.
[05:46](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=346s) Throwing away your existing codebase means losing a lot of your native Compose UI, reimplementing basic text and navigation, and writing wrappers around Android APIs such as text-to-speech.
[05:57](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=357s) Option B is building a low level 3D graphics rendering pipeline from scratch inside your native app using raw Vulkan and `OpenXR` APIs.
[06:07](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=367s) Instead of delivering product features, you're essentially building your own proprietary engine.
[06:13](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=373s) So today I can show you how you can make the most of Godot as a library and Unity as a library.
[06:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=378s) And with this, the addition of spatial containers, you can now have the best of both worlds.
[06:24](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=384s) We can achieve this by embedding a real-time engine directly inside a library inside your Jetpack application.
[06:30](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=390s) You retain the production Android app structure, the business logic, and seamless rendering of 3D spatial content inside your application.
[06:40](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=400s) This also means you can have one APK for all form factors.
[06:42](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=402s) Where you don't just want to show 3D content, you can choose not to load the library into the activity or the fragment and keep your codebase clear, and allows you to support many devices.
[06:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=414s) The philosophy is straightforward, decouple the application and make the real-time engine a library.
[07:00](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=420s) Instead of choosing between an Android app or an engine, you treat the engine as an embedded software library, also known as an AAR dependency.
[07:08](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=428s) You pull this directly into your Gradle build, just like any other library.
[07:12](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=432s) Your native Android code remains the absolute brain of the application hosting the 3D components.
[07:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=438s) Jetpack Compose XR handles what it does best, rendering native accessible fluid UI and managing app state.
[07:26](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=446s) And when a 3D element is needed, the app hosts an embedded rendering view that's rendered by the game engine itself.
[07:32](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=452s) Today, we're going to deep dive exactly how this works under the hood for Godot.
[07:36](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=456s) And I'll start by the mechanics of embedding and establishing bidirectional communication between Kotlin and GDScript.
[07:43](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=463s) And then we'll look at the new `OpenXR` extension, Spatial Container, that will really unlock the new generation of multitasking and spatial layout.
[07:51](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=471s) Let's take a quick look at the demo I showed before.
[07:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=474s) I made this using Godot and Android Studio.
[07:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=475s) And this clip shows the Jetpack Compose for XR rendering native 2D UI spatial panels for our scoreboard and notifications and highlights.
[08:05](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=485s) And on top of this, Godot is rendering a simulation of a soccer match.
### Communication between Kotlin and Godot
[08:15](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=495s) Here you can see a breakdown of how we communicate from Kotlin to GDScript.
[08:20](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=500s) To do this, Godot provides a plug-in architecture.
[08:23](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=503s) You write a lightweight Kotlin class that extends a `GodotPlugin`.
[08:27](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=507s) This class serves as an interface between the JVM and Godot's C++ runtime.
[08:33](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=513s) Because of this, interactions between your app and Godot are very very lightweight.
[08:38](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=518s) Godot's architecture supports two-way communication between Godot and Kotlin.
[08:43](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=523s) When a game event occurs in Godot, such as a goal or foul, Godot can directly invoke Kotlin functions and update Compose UI in real time.
[08:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=534s) Then when the user interacts with the media player, let's say, to pause or select different highlights, Kotlin emits a signal back to Godot, and the corresponding game engine actions happen.
[09:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=544s) Once the Godot plug-in is crossed, performance and management simply falls along the domain lines.
[09:09](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=549s) Apply the best standard practices for Android apps and the best standard practices for Godot projects.
[09:16](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=556s) Now we'll look at the architecture.
[09:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=558s) Let's look at how we wire this into Godot into an Android app.
[09:21](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=561s) The Godot engine runtime is distributed via Maven as an AAR.
[09:26](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=566s) Integrating into your build just follows standard Android practices.
[09:30](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=570s) There is one important step to be aware of straight away, and that is what I call the asset trap.
```kts
androidResources {
ignoreAssetsPattern = "!.svn:!.git:!.gitignore:!.ds_store:!*.scc:
_*:!CVS:!*~:!android:!export_presets.cfg"
}
```
[09:35](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=575s) In a Godot project, your scenes and project config- configuration files live together.
[09:40](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=580s) And when embedding Godot, the Godot project is added directly inside your assets folder.
[09:45](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=585s) So this means all the imported 3D meshes, compressed textures, and compiled textures, it all stores as a runtime artifact inside a hidden file called .godot.
[09:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=595s) By default, Android will strip out these strip out this folder when doing the build, so it ignores any directory with a dot period.
[10:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=604s) So if you sync and build your app normally, APK will build, but it will crash and load because those things have not been compiled into the APK.
[10:12](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=612s) So, to fix this, you can just explicitly override the AAPT options inside your build.gradle and allow for the leading dots to happen from Godot.
[10:20](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=620s) And once you do that, you can run your Godot scene once in the Godot editor, and from there you can build everything else from Android Studio.
[10:27](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=627s) In order to support spatial containers and the latest `OpenXR`, I recommend using Godot 4.8 and above, Godot vendor extensions 5.2, and the Khronos `OpenXR` loader 1.1.63.
```kts
dependencies {
// Godot 4.8 Android Library + Godot OpenXR Vendors plugin + Khronos OpenXR Loader
implementation("org.godotengine:godot:4.8.0.dev-SNAPSHOT")
implementation("org.godotengine:godot-openxr-vendors-androidxr:5.1.0-stable")
implementation("org.khronos.openxr:openxr_loader_for_android:1.1.63")
// Jetpack XR SDK (Compose for XR, SceneCore, Runtime)
implementation("androidx.xr.compose:compose:1.0.0-beta02")
implementation("androidx.xr.scenecore:scenecore:1.0.0")
implementation("androidx.xr.runtime:runtime:1.0.0")
```
### Rendering Godot in Jetpack Compose
[10:43](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=643s) Once the assets are packaged, how does Godot and Compose actually render?
[10:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=648s) The Godot library provides a factory method called `onInitRenderView`.
```kotlin
@Composable
fun GodotView(godotHost: GodotHost, modifier: Modifier = Modifier) {
// 1. Initialize Godot's native render surface
val renderView = remember(godotHost) {
godotHost.godot!!.onInitRenderView(godotHost)
}
// 2. Host the SurfaceView inside Jetpack Compose
AndroidView(
modifier = modifier,
factory = { renderView }
)
}
```
[10:52](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=652s) This returns a `SurfaceView` that Godot just renders to.
[10:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=655s) So, you wrap this in a Composable function, and Godot can now be sized, padded, and placed any- anywhere inside an entity panel.
```kotlin
Subspace {
SpatialPanel(
modifier = SubspaceModifier
.width(960.dp)
.height(540.dp)
.offset(z = (-200).dp)
.movable()
,
) {
GodotView(
godotHost = godotHost,
modifier = Modifier.fillMaxSize(),
)
}
}
```
[11:05](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=665s) The engine itself cannot manage the life cycle.
```kotlin
@Composable
fun GodotView(
godotHost: GodotHost,
modifier: Modifier = Modifier,
lifecycleOwner: LifecycleOwner = LocalLifecycleOwner.current,
) {
DisposableEffect(lifecycleOwner, godotHost) {
val observer = LifecycleEventObserver { _, event ->
val engine = godotHost.godot ?: return@LifecycleEventObserver
when (event) {
Lifecycle.Event.ON_START -> engine.onStart(godotHost)
….
Lifecycle.Event.ON_DESTROY -> engine.onDestroy(godotHost)
else -> {}
}
}
lifecycleOwner.lifecycle.addObserver(observer)
onDispose {
lifecycleOwner.lifecycle.removeObserver(observer)
}
}
}
```
[11:07](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=667s) So, your activity needs to implement the `GodotHost` interface.
[11:11](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=671s) This is going to just make sure that any of the life cycles that are happening within your activity, on pause, on resume, they're all going to be sent to the Godot engine itself, and now it's life-cycle aware.
[11:24](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=684s) Next, we need to add our native app and our engine to co- to communicate seamlessly.
[11:29](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=689s) So, to do this, we can use the `GodotHost` interface.
```kotlin
class MainActivity : ComponentActivity(), GodotHost {
override fun getHostPlugins(engine: Godot): Set =
setOf(SoccerGodotPlugin(engine))
}
```
[11:32](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=692s) We can define the Godot plugin, and here we can make sure when Godot starts up that the plugin's registered from the Compose side.
[11:39](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=699s) So, now the plugin's registered, let's just set up Android to communicate by emitting a signal to Godot.
[11:46](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=706s) In our Kotlin code, we're going to create an instance of the `GodotPlugin`.
[11:49](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=709s) We're going to have our activity register the plugin with the Godot library.
[11:52](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=712s) And then in Kotlin, we're creating the `GodotPlugin`, registering its name, and telling Godot all the signals that we want to use within this package.
```kotlin
class SoccerGodotPlugin(godot: Godot) : GodotPlugin(godot) {
override fun getPluginName() = "SoccerPlugin"
// 1. Signals sent from Android Compose UI -> Godot
override fun getPluginSignals() = setOf(
SignalInfo("camera_mode_changed", String::class.java),
SignalInfo("match_command", String::class.java),
)
}
```
[12:01](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=721s) Next, we have a Godot Script, where you can get the instance of this plugin.
```text
var soccer_plugin: Object = null
func _ready() -> void:
if Engine.has_singleton("SoccerPlugin"):
soccer_plugin = Engine.get_singleton("SoccerPlugin")
# Listen for signals from Jetpack Compose UI
soccer_plugin.connect("camera_mode_changed", _on_camera_mode_changed)
soccer_plugin.connect("match_command", _on_match_command)
```
[12:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=724s) We can see what's registered, and then here we can just hook up the functions to the signals that we've registered.
[12:10](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=730s) So then once you call something like `emitSignal`, you can then just pass this from your Kotlin code, and it goes straight over to Godot itself.
```kotlin
class SoccerGodotPlugin(godot: Godot) : GodotPlugin(godot) {
fun sendCommandToGodot(command: String) {
emitSignal("match_command", command)
}
}
```
[12:20](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=740s) In order for Godot to communicate to Kotlin, let's say a goal happens, we can call a Kotlin function directly from the Godot Script, just by using the plugin instance.
```text
func _on_goal_scored(minute: int, home: int, away: int) -> void:
if soccer_plugin:
soccer_plugin.onMatchEvent("GOAL", minute, home, away)
```
[12:30](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=750s) So first, you annotate the Kotlin function with `UsedByGodot`.
```kotlin
@UsedByGodot
fun onMatchEvent(type: String, minute: Int, home: Int, away: Int, title: String) {
_uiState.update { state ->
state.copy(homeScore = home, awayScore = away, matchSeconds = minute)
}
}
```
[12:34](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=754s) The engine handles the JNI marshalling under the hood.
[12:37](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=757s) And then we can call the function directly from Godot Script.
[12:41](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=761s) There's one quick tip to remember when using this, and that's `UsedByGodot` requires exact string matching.
[12:47](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=767s) So, if your Godot Script calls `onMatchEvent`, your Kotlin name needs to be exactly the same, so there's no automatic camel casing or snake-case conversion.
[12:57](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=777s) So, you just have to make sure both methods are exactly the same.
[13:01](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=781s) With the plugin connected, the most critical part of this talk is what I call the mental model.
[13:07](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=787s) This is where game engines and Jetpack Compose operate on two fundamentally, like, opposing architectures.
[13:13](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=793s) A game engine operates on a continuous frame tick.
[13:16](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=796s) Whether you've changed anything or not, the game engine tick process happens 60, 90, 100 times per second, delivering rich and smooth real-time rendering.
[13:25](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=805s) And engines like Godot make effective use of the GPU and CPU cycles.
[13:30](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=810s) Jetpack Compose, on the other hand, handles things reactively and it's state-driven.
[13:34](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=814s) And when an app isn't changing, Compose literally continues, consumes 0 CPU.
[13:43](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=823s) If you want to connect these to run two runtimes without a clear strategy, these philosophies tend to collide.
[13:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=834s) So, let's how- let- let's look at how we get around this problem by what I call like a tick leak.
[13:58](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=838s) This is where we have a 3D experience, let's say the scoreboard timer.
[14:02](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=842s) In your engine script, there's a ti- timer variable that continuously changes.
[14:06](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=846s) And if you blindly forward this variable across to Compose, let's say 90 90 times per second, you're going to enter this recom- composition loop 90 times per second.
[14:17](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=857s) So, this means Compose will re-evaluate your Composables, remeasure layouts, redraw UI trees 90 times per second.
[14:23](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=863s) And you are going to see frame drops, and you're going to see UI lag, and you're going to hit thermal throttling pretty quickly.
[14:29](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=869s) So, the golden rule is never let the continuous engine loop dictate the re-ed- reactive UI loop.
[14:35](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=875s) In this example, rather than having Godot update the scoreboard timer continuously, I just make sure Android pulls this data once every second when it needs it.
```kotlin
LaunchedEffect(uiState.phase) {
while (uiState.phase == MatchPhase.PLAYING) {
delay(1000L) // Update scoreboard timer at 1 Hz, not 90 Hz
SoccerGodotPlugin.tickLocalClock()
}
}
...
@UsedByGodot
fun onMatchEvent(eventTypeStr: String, matchMinute: Int, ...) {
// Recompose once when threshold is crossed to display the popup banner
_uiState.update { it.copy(activeBannerEvent = event, ...) }
}
```
[14:46](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=886s) Finally, always remember thread boundary.
[14:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=888s) A Godot game engine loop continually executes on its own render thread.
[14:53](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=893s) The UI, the Android UIs must execute on the Android main thread.
[14:57](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=897s) So, always ensure your Kotlin plugin dispatches events using `Dispatchers.Main` or safe-state primitives before updating your view model states.
```kotlin
private val mainScope = CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate)
mainScope.launch {
_uiState.update { current ->
current.copy(
homeScore = homeScore,
awayScore = awayScore,
matchSeconds = matchMinute.coerceIn(0, 90),
phase = updatedPhase,
events = newEventsList,
activeBannerEvent = event
)
}
```
[15:05](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=905s) By keeping high-frequency work in the engine and discrete semantics in Compose, you can get buttery-smooth 3D viewports without compromising battery life, accessibility, or performance to your native app.
[15:17](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=917s) Now, let's look at how we can add Spatial Containers to your Godot project.
[15:22](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=922s) First, let's start by setting up the app to be Full-Space, enable Spatial Containers with its bounding hints, and then set the XR loader library in the Android Manifest file.
```xml
```
[15:34](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=934s) Godot has its own XR project settings, where I can enable `OpenXR`, XR shaders, and the Spatial Containers, just via its editor.
[15:44](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=944s) And that's pretty much all you need to do to enable Spatial Containers within Godot.
[15:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=948s) So, now that you've got Godot working with Android XR, Spatial Containers, and all this developed, the best practices for using Spatial Containers, along with spatial panels and JXR, we definitely recommend checking out more information on the developer website.
[16:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=964s) We just walked through an example of how to use Spatial Containers within the Godot Engine and Android XR, and I'm sure many of you would like to hear from a developer that is actively working on Spatial Containers for their Android XR experience.
[16:16](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=976s) So, next, I'd like to invite Josh and Rodrigo from NBA, who are one of the first applicat-- one of the first partners that are using Spatial Containers within their application.
### NBA Court View overview
[16:33](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=993s) Thanks, Luke.
[16:34](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=994s) Hey, everyone.
[16:34](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=994s) My name is Josh, and I'm an Android developer at the NBA.
[16:37](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=997s) I'm here with my colleague today, Rodrigo Sanchez, a Unity software developer at Beyond Sports.
[16:43](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1003s) And today, we'd like to show you how we used Unity as a Library to make Court View for the NBA app.
[16:50](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1010s) At the NBA, we're always looking to bring new experiences to our fans.
[16:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1014s) So, together with Google and Beyond Sports, we're excited to introduce Court View.
[16:59](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1019s) Court View offers 3D recreations of NBA games, allowing League Pass subscribers to watch from new perspectives not possible with normal broadcast video.
[17:12](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1032s) Scale and re- rotate the court, placing it wherever you like in your space.
[17:16](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1036s) Move around it, and watch the game from unique angles.
[17:19](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1039s) All the while, you can use your NBA app just like normal, from viewing sync stats to watching the live broadcast.
[17:25](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1045s) How's this all set up?
[17:36](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1056s) Court View is rendered in Unity and consumed by our Android app using Unity as a Library.
[17:41](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1061s) We then produce a single Android Archive, hosted on Maven Central, that the NBA app can pull in as a single dependency.
[17:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1068s) This allowed us to split up the work between the Android side and the Unity side.
[17:51](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1071s) So each each of us can focus on what we know best, and work in parallel to speed up development overall.
[17:57](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1077s) As an example, while our team worked on full-space, experience in Compose, the Unity team could focus on the scaling and layout of the 3D court.
[18:06](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1086s) Now, I'll hand it off to Rodrigo to talk more about the Unity project and how they worked with Unity as a Library.
### Integrating Unity as a library
[18:17](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1097s) Thank you, Josh.
[18:18](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1098s) We at Beyond Sports have been working together with the NBA for some years on representing 3D visualizations with real data from matches.
[18:24](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1104s) With this project, we adapted our 3D simulation to be compatible with Android XR.
[18:29](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1109s) Our simulation is built on the Unity engine with C#.
[18:32](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1112s) We export the simulation into Android with Unity as a Library, and built a communication layer next to it using Kotlin.
[18:38](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1118s) Before bundling these two modules together as a single Maven dependency for the NBA to include in their application.
[18:45](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1125s) On the screen, you can see a short video of the same simulation that Josh showed earlier, just running on the editor.
[18:51](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1131s) To achieve this, we use the Android XR Extensions package for Unity.
[18:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1135s) Let me take you through the process.
[18:57](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1137s) So, before building in Unity, make sure that you have selected the right settings and plug-ins.
[19:03](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1143s) Get Unity's XR Plug-in Management package and Google's Android XR Extensions for Unity package.
[19:08](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1148s) Configure them in the XR Plug-in Management settings to use the `OpenXR` Android XR Extensions provider.
[19:14](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1154s) For our use case, we wanted to make use of Spatial Containers, so we enabled that feature and disabled immersive XR from the session management settings.
[19:22](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1162s) We used no XR input package, as all the input is handled natively, outside of Unity.
[19:29](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1169s) Next, in the player settings, you'll want to set your project to use a minimum API level of 24 or above, and the application entry point must be set to the newer Game Activity.
[19:38](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1178s) Mind that Activity must be left unchecked.
[19:41](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1181s) We are only interested in building for ARM64, so we select that target architecture, and in the resolution section, we enabled Run in Background, so that you can multitask with other apps, and keep the 3D simulation running on the side.
[19:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1195s) We also enable Resizeable Activity, as it is required to render dialogues, such as system permission requests.
[20:01](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1201s) Lastly, it is recommended for Android XR apps to be rendered using the Universal Render Pipeline and Vulkan Graphics API.
[20:08](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1208s) In your URP asset, HDR and post-processing should be disabled.
[20:15](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1215s) Once you have all of those steps done, you're ready to make a build.
[20:19](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1219s) Move over to the build window, and if you're using Unity 6.1 or above, you can select the Android XR build profile, which lets you separate settings if you also build for Android Mobile.
[20:28](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1228s) Before building, make sure to enable export project.
[20:31](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1231s) After Unity finishes building, you will be left with a working Android project which can be assembled in Android Studio and run on a device like any other Unity APK.
[20:41](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1241s) Once in Android Studio, you'll want to turn your exported build into a library to be used by another application.
```kts
// The Unity build
include(":unityLibrary")
project(":unityLibrary").projectDir = File("..\\Libraries\\unityBuild\\unityLibrary")
// Custom communication IPC module
include(":nativeCommunications")
project(":nativeCommunications").projectDir = File("..\\Libraries\\nativeCommunications")
// If using XR features
include(":unityLibrary:xrmanifest.androidlib")
// Used for merging libraries, exporting and publishing.
include(":packageExporter")
dependencyResolutionManagement {
repositories {
flatDir { // Required to load unity correctly
dirs("${project(":unityLibrary").projectDir}/libs")
}
}
```
[20:47](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1247s) In our project here, we have four different modules: the main APK module to test as a mock-up placeholder for NBA's wrapping application, the Unity Library module holding our exported build, a `nativeCommunications` module holding our communication layer that we built between the activities, and a `packageExporter` module, which we use to merge our AARs into a single package to deliver to Maven.
[21:08](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1268s) Merging multiple AARs can be achieved using Android Fused Library, or older community methods like the Fat AAR.
[21:15](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1275s) You can keep your structure and communication simple by keeping Unity and the main activity running in the same process.
[21:22](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1282s) We opted against this, and decided to separate the Unity activity into its own process as a safeguard in the event that if the Unity portion would crash, then the NBA app is unaffected and can safely handle closing Unity gracefully without crashing itself.
[21:36](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1296s) The decision force us to adopt a more robust inter-process communication system.
[21:42](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1302s) Since the activities are in different processes, there is no memory shared between them, and many common Android IPC methods don't work or fall short of requirements.
[21:51](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1311s) We use the Android Interface Definition Language, or AIDL, to talk between the NBA's main activity and our embedded Unity activity.
[21:58](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1318s) Then, communication between Android and Unity is also based in interfaces, making use of the built-in `AndroidJavaProxy` in Unity to register a Kotlin interface to a C# interface implementation.
[22:11](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1331s) Our communications module is primarily focused on what we call the activity bridge.
```kotlin
// Main Activity
val intent = Intent(context, AIDLBridgeService::class.java)
context.startService(intent)
context.bindService(intent, connection, 0)
fun onStartMatchClicked() {
bridge.runWithService { service ->
service.startMatch("match_id")
}
}
// AIDL Bridge Service
private val remoteBinder = object : IBridgeService.Stub() {
override fun startMatch(matchID: String?) {
addToCallbackQueue { nativeInterface?.startMatch(matchID) }
}
}
```
[22:15](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1335s) Both activities have access to the activity bridge and the utility methods within it.
[22:20](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1340s) The bridge contains a reference to a service that implements the AIDL interfaces that we defined, and it also provides an easy way for the activities to bind to the process, sorry, to the service and internally manages the lifecycle of it.
[22:32](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1352s) Starting a simulation can be done with a callback in which we pass the match ID as a parameter.
[22:36](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1356s) Then, the bridge passes it through AIDL, and then, onto the Unity app, where C# gets a command and starts configuring the scene.
```csharp
public class NativeInterfaceListener : AndroidJavaProxy {
[UsedImplicitly]
public void startMatch(string matchID) {
UnityThreadHelper.InvokeOnMain(() => _internalAPI.StartMatch(matchID));
}
}
```
[22:44](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1364s) We do need to wait for Unity to be alive and ready to press the commands that we pass onto it.
[22:49](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1369s) The Unity activity that wraps the application will be active before Unity itself is.
[22:54](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1374s) In the case of commands that are sent too early, we store them in a callback queue.
```kotlin
fun onStartMatchClickedInternal() {
if (isServiceBound) {
AIDLBridgeService.startMatch("match_id")
} else {
delayedCallbacksQueue.add(action)
}
}
```
[22:57](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1377s) After Unity is done boot loading, and initializing, and we are in a state that we consider safe, we communicate from C# that Unity is ready.
```csharp
// After bootload and initialization
using AndroidJavaClass nativeBridge = new("com.company.UnityActivity");
using AndroidJavaObject instance = nativeBridge.GetStatic("instance");
instance.Call("onUnityReady");
```
[23:06](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1386s) After which, we loop over our queue and forward all of our stored commands.
```kotlin
@Suppress("unused") // Unity implicit call
fun onUnityReady() {
bridge.activityBridgeServiceInternal?.onUnityReady()
}
delayedCallbacksQueue.forEach { it(activityBridgeService) }
```
[23:12](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1392s) This diagram illustrates better how our inter-process communication works.
[23:15](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1395s) As you can see, using the bridge between activities, we can handle those callbacks that we mentioned.
[23:20](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1400s) We set up StateFlows that are updated from Unity to monitor values such as authentication status or the current state timestamp of the match.
[23:27](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1407s) Additionally, we also manage the encryption of credentials, and we forward any errors from Unity so that the NBA can debug easier and decide how to best handle them at an application level.
[23:37](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1417s) That's it for me.
[23:38](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1418s) Now, I'll pass the stage back to Josh so he can talk more about how they integrate the Court View Unity package into the NBA app.
[23:50](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1430s) We use these Gradle dependencies to integrate Unity as a library with our Jetpack Compose app: XR Runtime to access the 3D session, Jetpack `SceneCore` to host our Unity activity and toggle spatial environments, and Jetpack Compose for XR to access spatial panels in our Compose code.
```kts
dependencies {
implementation("androidx.xr.runtime:runtime:1.0.0")
implementation("androidx.xr.scenecore:scenecore:1.0.0")
implementation("androidx.xr.compose:compose:1.0.0-beta02")
}
```
[24:10](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1450s) Next, let's see how we render Court View in Jetpack Compose.
```kotlin
// Step 1: Request Full Space
LocalSession.current?.scene?.requestFullSpace()
// Step 2: A shared Subspace for NBA app content and embedded Unity Activity.
override fun onCreate(savedInstanceState: Bundle?){
setContent {
Subspace {
SpatialPanel { AppContent() }
// Court View
val intent = courtViewFeature.intent
val controller = rememberSpatialActivityPanelController(intent)
SpatialActivityPanel(controller)
}
}
}
```
[24:14](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1454s) Before we launch Court View, we bring you into full space.
[24:16](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1456s) This is required in order to display the spatial panel holding Unity.
[24:21](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1461s) Next, we send game information over AIDL to launch Court View for a specific game.
[24:26](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1466s) Unity itself is then started.
[24:28](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1468s) Our Android archive sends this as an intent to a Unity activity, and we wrap this in a SpatialActivityPanelController.
[24:36](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1476s) From there, we just pass that controller to a `SpatialActivityPanel`.
[24:41](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1481s) This panel is what holds Court View, and it's shared in our subspace with our main spatial panel.
[24:48](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1488s) This allows us to both control both of them at once without any issues.
[24:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1495s) And that's it.
[24:55](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1495s) I'll send it back to Luke.
### Conclusion and resources
[25:03](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1503s) Thank you very much.
[25:04](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1504s) As you can see, NBA was able to accelerate Court View by splitting the Unity and Jetpack Compose development.
[25:11](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1511s) To learn more on getting started with spatial containers and using it within your own Android XR apps, check out developer documentation on developer.android.com/develop/xr.
[25:22](https://www.youtube.com/watch?v=u8dsUtPvOlY&t=1522s) Thank you very much for joining us today, and we can't wait to see what you build.