# Moving beyond 2D with custom 3D geometry in the Jetpack XR SDK

- Speakers: Egor Yusov (Software Engineer), Jay Abi-Saad (Software Engineer)
- Track: Android XR
- Video: https://www.youtube.com/watch?v=mssk4zPs0zU
- Page: https://developer.android.com/events/dev-summit/talks/moving-beyond-2d-with-custom-3d-geometry-in-the-jetpack-xr-sdk

## About the talk

Transitioning your Android applications from traditional 2D screens into spatial environments requires a new approach to constructing shapes.

In this session, we will explore the Custom Mesh API in the Jetpack XR SDK, which allows you to programmatically generate procedural 3D geometry directly at runtime. We will walk through the core workflow, from defining vertex layouts to rendering dynamic objects seamlessly within the user's physical space.

By the end of this talk, you will have the practical skills to build interactive, custom 3D geometry completely from code.

## Summary

- The glTF introspection API in Jetpack SceneCore exposes model metadata at run time. You can play embedded animations, transform subnodes, or override meshes with materials and textures.
- Jetpack SceneCore includes mesh APIs to generate custom 3D geometry in Kotlin. Four building blocks such as VertexLayout and `MeshBuffer` construct models triangle by triangle without 3D tools.
- Dynamic meshes support real-time updates as users draw paths or animate surfaces. Updating only the changed vertex data streams modifications straight to the GPU without recreating the mesh or entity.
- Gemini in Android Studio can write complex vertex animation and lighting math for the mesh APIs. One prompt generates working code that animates vertex grids and recalculates normals every frame.

## Resources

- [Android XR Emulator](https://goo.gle/xr-tools)
- [Jetpack XR SDK](https://goo.gle/jetpack-xr-sdk)

## Topics

jetpack-xr, jetpack-scenecore, android-xr, 3d-geometry, mesh-api, gltf

## Transcript

### Introduction to Android XR and SceneCore

[00:05](https://www.youtube.com/watch?v=mssk4zPs0zU&t=5s) Hey, folks.

[00:06](https://www.youtube.com/watch?v=mssk4zPs0zU&t=6s) I'm Jay.

[00:06](https://www.youtube.com/watch?v=mssk4zPs0zU&t=6s) And I'm Egor.

[00:07](https://www.youtube.com/watch?v=mssk4zPs0zU&t=7s) And we're software engineers working on Jetpack XR at Google.

[00:11](https://www.youtube.com/watch?v=mssk4zPs0zU&t=11s) With XREAL Aura right around the corner, we want to make sure you have all the tools you need to build great spatial experiences on the platform.

[00:19](https://www.youtube.com/watch?v=mssk4zPs0zU&t=19s) But best of all, you don't even need XR hardware to get started.

[00:24](https://www.youtube.com/watch?v=mssk4zPs0zU&t=24s) All the features we'll show are available today in the Android XR Emulator in Android Studio.

[00:29](https://www.youtube.com/watch?v=mssk4zPs0zU&t=29s) Our goal is to show you that with the Jetpack SceneCore library, you don't need deep 3D knowledge to build rich spatial experiences on the platform.

[00:38](https://www.youtube.com/watch?v=mssk4zPs0zU&t=38s) Everything you'll see is running in an Android app, written in idiomatic Kotlin right inside Android Studio.

[00:43](https://www.youtube.com/watch?v=mssk4zPs0zU&t=43s) We recently promoted Jetpack SceneCore to stable, and today we'll cover some cool advancements of our 3D model APIs, as well as our brand-new mesh APIs, which allow you to build custom 3D geometry on the fly.

### Working with glTF 3D models

[00:58](https://www.youtube.com/watch?v=mssk4zPs0zU&t=58s) But let's start with 3D models.

[01:00](https://www.youtube.com/watch?v=mssk4zPs0zU&t=60s) When it comes to 3D models, the first thing we need is a file format.

[01:05](https://www.youtube.com/watch?v=mssk4zPs0zU&t=65s) Open standards are a core part of the Android XR DNA.

[01:09](https://www.youtube.com/watch?v=mssk4zPs0zU&t=69s) So, naturally, we went all in with the glTF format from the Khronos Group.

[01:14](https://www.youtube.com/watch?v=mssk4zPs0zU&t=74s) Now, if you're not from the 3D world, you might be asking, "What's a glTF?"

[01:18](https://www.youtube.com/watch?v=mssk4zPs0zU&t=78s) You can think of it as a JPEG, but for 3D models.

[01:22](https://www.youtube.com/watch?v=mssk4zPs0zU&t=82s) The glTF asset structure provides data for things like animations, subnodes, and meshes.

```json
{
    "asset" : {...},
    "nodes" : [
        {
            "name" : "Right_Arm",
            "rotation" : [...],
            "scale" : [...],
            "translation" : [...],
        },
    ],
    "animations" : [
        {
            "name" : "Wave",
        },
    ],
    "meshes" : [
        {
            "name" : "Torso",
            "primitives" : [ ... ],
        },
    ],

    // Way way more!

}
```

[01:28](https://www.youtube.com/watch?v=mssk4zPs0zU&t=88s) But for the purpose of this talk, you can think of subnodes as parts of a model, each with their own position and rotation.

[01:36](https://www.youtube.com/watch?v=mssk4zPs0zU&t=96s) Meshes are the surfaces you see, and the animations are movements of those subnodes and meshes that the asset's author baked in ahead of time.

[01:44](https://www.youtube.com/watch?v=mssk4zPs0zU&t=104s) Cool.

[01:45](https://www.youtube.com/watch?v=mssk4zPs0zU&t=105s) Now, at this point, you might be asking, "Where can I get a glTF model in the first place?"

[01:50](https://www.youtube.com/watch?v=mssk4zPs0zU&t=110s) Well, on top of the many models that are out there, you can also create your own.

[01:54](https://www.youtube.com/watch?v=mssk4zPs0zU&t=114s) Now that we've covered the basics of the glTF format, let's go ahead and use it in our app.

[02:00](https://www.youtube.com/watch?v=mssk4zPs0zU&t=120s) We start with a basic Android XR project using Jetpack SceneCore.

[02:04](https://www.youtube.com/watch?v=mssk4zPs0zU&t=124s) You can see a spatial panel with an activity floating in the user's space.

[02:09](https://www.youtube.com/watch?v=mssk4zPs0zU&t=129s) You can grab it, move it, and resize it to your liking.

[02:13](https://www.youtube.com/watch?v=mssk4zPs0zU&t=133s) And although this talk is all about 3D, definitely check out the rest of the Jetpack XR SDK as well.

[02:20](https://www.youtube.com/watch?v=mssk4zPs0zU&t=140s) All right, spatial panels are fun, but we want more magic.

[02:24](https://www.youtube.com/watch?v=mssk4zPs0zU&t=144s) And what better way to add some magic than throwing in a 3D model?

[02:28](https://www.youtube.com/watch?v=mssk4zPs0zU&t=148s) Well, it just so happens that our designer made this glTF model of a car.

```kotlin
// Load assets/models/car.glb
val carModel = GltfModel.create(
    session,
    Paths.get("models", "car.glb")
)

// Create an entity in the scene
val carEntity = GltfModelEntity.create(
    session,
    carModel,
    parent = session.scene.activitySpace
)
```

[02:33](https://www.youtube.com/watch?v=mssk4zPs0zU&t=153s) They even embedded animations in it, which we'll use later on.

[02:37](https://www.youtube.com/watch?v=mssk4zPs0zU&t=157s) Using SceneCore, I can use our 3D model APIs to load it right into our scene.

### Model animations and subnode manipulation

[02:43](https://www.youtube.com/watch?v=mssk4zPs0zU&t=163s) But let's make the car feel a bit more alive.

[02:47](https://www.youtube.com/watch?v=mssk4zPs0zU&t=167s) Remember all that rich metadata inside a glTF?

[02:50](https://www.youtube.com/watch?v=mssk4zPs0zU&t=170s) With a new glTF introspection API, you can now access it, starting with animations.

```kotlin
// Access every animation in car.glb
val animations = carEntity.getAnimations()
   .associateBy { it.name }

// Open the doors...
animations["lf_door_open"]?.start()
animations["rt_door_open"]?.start()

// ...and close them again
animations["lf_door_close"]?.start()
animations["rt_door_close"]?.start()

// Same for the hood and trunk
animations["hood_open"]?.start()
animations["trunk_open"]?.start()
```

[02:56](https://www.youtube.com/watch?v=mssk4zPs0zU&t=176s) In just a few lines of code, I can individually play any glTF animations built into the car model, like opening and closing the doors, the hood, or the trunk.

[03:07](https://www.youtube.com/watch?v=mssk4zPs0zU&t=187s) But animations are not the only piece of metadata that you can access.

[03:12](https://www.youtube.com/watch?v=mssk4zPs0zU&t=192s) The glTF introspection API also exposes subnodes in the model as `GltfModelNode`, allowing you to move, rotate, and scale them to your liking.

```kotlin
// Wheel sub nodes, named by our designer
val nodes = carEntity.nodes.associateBy { it.name }
val front = listOf("lf_frontWheel_geo",
   "rt_fontWheel_geo").map { nodes[it]!! }
val rear = listOf("lf_RearWheel_geo",
   "rt_RearWheel_geo").map { nodes[it]!! }

// Update a sub node's transform at runtime
fun GltfModelNode.rotate(q: Quaternion) {
   localPose = localPose.copy(rotation = q)
}

// Each frame: steer front, spin all four
fun drive(steerDeg: Float, spinDeg: Float) {
   val steer = Quaternion.fromAxisAngle(
       Vector3.Up, steerDeg)
   val spin = Quaternion.fromAxisAngle(
       Vector3.Right, spinDeg)
   front.forEach { it.rotate(steer * spin) }
   rear.forEach { it.rotate(spin) }
}
```

[03:23](https://www.youtube.com/watch?v=mssk4zPs0zU&t=203s) For example, here, I can individually update the position and the rotation of the wheels as the car drives.

[03:29](https://www.youtube.com/watch?v=mssk4zPs0zU&t=209s) And to be clear, what you're seeing is not an embedded glTF animation.

[03:34](https://www.youtube.com/watch?v=mssk4zPs0zU&t=214s) Here, I'm individually spinning and steering the wheels all at run time.

### Applying materials and textures

[03:40](https://www.youtube.com/watch?v=mssk4zPs0zU&t=220s) One last piece of metadata that you can access is any mesh in the glTF model.

[03:45](https://www.youtube.com/watch?v=mssk4zPs0zU&t=225s) Now, Egor will dive deeper into meshes in a moment.

[03:48](https://www.youtube.com/watch?v=mssk4zPs0zU&t=228s) But for now, let's style up our car.

[03:51](https://www.youtube.com/watch?v=mssk4zPs0zU&t=231s) Using our texture API, we can load a PNG image texture, say of a metal pattern.

```kotlin
// Load a PNG with the Texture API
val metal = Texture.create(
   session, Paths.get("textures", "metal.png"))

// Create KhronosPbrMaterials
val (body, doors, hood) = List(3) {
   KhronosPbrMaterial
       .create(session, AlphaMode.OPAQUE)
}
body.setBaseColorFactor(CYAN)
doors.setBaseColorFactor(PINK)
hood.setBaseColorTexture(metal)

// Override meshes by sub node name
val nodes = carEntity.nodes.associateBy { it.name }
nodes["mainBody_geo"]?.setMaterialOverride(body)
nodes["lf_door_geo"]?.setMaterialOverride(doors)
nodes["rt_door_geo"]?.setMaterialOverride(doors)
nodes["hood_geo"]?.setMaterialOverride(hood)
```

[03:57](https://www.youtube.com/watch?v=mssk4zPs0zU&t=237s) Then, with our material API, we can plug it right into a `KhronosPbrMaterial`.

[04:02](https://www.youtube.com/watch?v=mssk4zPs0zU&t=242s) Now, if you're new to 3D, you can think of a mesh as a tabletop, a material as a tablecloth, and a texture as a pattern printed on that cloth.

[04:12](https://www.youtube.com/watch?v=mssk4zPs0zU&t=252s) But here, instead of a tabletop, we use the material objects we created to override the meshes associated with the car's hood and doors subnodes.

[04:21](https://www.youtube.com/watch?v=mssk4zPs0zU&t=261s) In just a few lines of code, we've transformed the look and feel of our car, again, all at run time.

[04:27](https://www.youtube.com/watch?v=mssk4zPs0zU&t=267s) That's some cool stuff, but here's an even better idea.

[04:31](https://www.youtube.com/watch?v=mssk4zPs0zU&t=271s) How awesome would it be to draw a race track for the car to drive on?

[04:35](https://www.youtube.com/watch?v=mssk4zPs0zU&t=275s) Here, a pre-made glTF model is not the right fit.

[04:39](https://www.youtube.com/watch?v=mssk4zPs0zU&t=279s) Instead, we need the ability to generate 3D geometry on the fly.

[04:43](https://www.youtube.com/watch?v=mssk4zPs0zU&t=283s) So, I'll hand it over to Egor to tell you all about our brand-new Jetpack SceneCore mesh APIs.

### Introduction to mesh APIs

[04:50](https://www.youtube.com/watch?v=mssk4zPs0zU&t=290s) Thanks, Jay.

[04:51](https://www.youtube.com/watch?v=mssk4zPs0zU&t=291s) Let's make it so that the user can draw their race track in the-- with their finger in their living room.

[04:57](https://www.youtube.com/watch?v=mssk4zPs0zU&t=297s) The track does not exist until it's drawn, so nobody can model it ahead of time.

[05:01](https://www.youtube.com/watch?v=mssk4zPs0zU&t=301s) We will build it in code, one triangle at a time.

[05:05](https://www.youtube.com/watch?v=mssk4zPs0zU&t=305s) A mesh starts with vertices, points in space.

```kotlin
// 4 vertices: x, y, z
val positions = floatArrayOf(
    0f, 0f, 0f,   // v0
    1f, 0f, 0f,   // v1
    0f, 1f, 0f,   // v2
    1f, 1f, 0f    // v3
)

// 2 triangles = 6 indices
val indices = intArrayOf(
    0, 1, 2,
    2, 1, 3
)
```

[05:09](https://www.youtube.com/watch?v=mssk4zPs0zU&t=309s) Each one carries attributes like a position, a normal for lighting, color, texture coordinates.

[05:15](https://www.youtube.com/watch?v=mssk4zPs0zU&t=315s) Indices connect the dots.

[05:17](https://www.youtube.com/watch?v=mssk4zPs0zU&t=317s) Every three make a triangle.

[05:19](https://www.youtube.com/watch?v=mssk4zPs0zU&t=319s) Two triangles make a square, six indices, but only four vertices, because they share two.

[05:26](https://www.youtube.com/watch?v=mssk4zPs0zU&t=326s) Keep going and you get a cube.

[05:28](https://www.youtube.com/watch?v=mssk4zPs0zU&t=328s) Every 3D model, including Jay's car, is built the same way, just with more triangles.

### Building blocks of the mesh API

[05:35](https://www.youtube.com/watch?v=mssk4zPs0zU&t=335s) Mesh API maps onto that with four building blocks: `VertexLayout`, `MeshBuffer`, `CustomMesh`, and `MeshEntity`.

[05:42](https://www.youtube.com/watch?v=mssk4zPs0zU&t=342s) Let's take them one at a time.

[05:45](https://www.youtube.com/watch?v=mssk4zPs0zU&t=345s) First, VertexLayout.

```kotlin
// How to read each vertex from bytes
val layout = VertexLayout.Builder()
    .addAttribute(POSITION, FLOAT3) // 12 B
    .addAttribute(NORMAL, FLOAT3)   // 12 B
    .addAttribute(UV0, FLOAT2)      //  8 B
    .build()              // stride: 32 B

// Offsets and stride are computed,
// or set them explicitly.

// Attributes can live in separate buffers
val split = VertexLayout.Builder()
    .addAttribute(POSITION, FLOAT3)
    .startNextBuffer()
    .addAttribute(COLOR, UBYTE4_NORM)
    .build()
```

[05:47](https://www.youtube.com/watch?v=mssk4zPs0zU&t=347s) Your vertex data is just bytes in a ByteBuffer, and every mesh packs them differently.

[05:50](https://www.youtube.com/watch?v=mssk4zPs0zU&t=350s) VertexLayout says how to read yours: which attributes, what type, and where each one sits.

[05:58](https://www.youtube.com/watch?v=mssk4zPs0zU&t=358s) Here, it's a position, a normal, and UVs, the texture coordinates, 32 bytes total.

[06:04](https://www.youtube.com/watch?v=mssk4zPs0zU&t=364s) Attributes can also live in separate buffers, so that you can update only the ones that change.

[06:10](https://www.youtube.com/watch?v=mssk4zPs0zU&t=370s) Second, `MeshBuffer`, a container for raw data.

```kotlin
// Vertex bytes + 32-bit indices
val meshBuffer = MeshBuffer.create(
    session, layout,
    vertexData = listOf(ByteBufferRegion(
        vertexBytes, 0, vertexBytes.capacity())),
    indexData = ByteBufferRegion(
        indexBytes, 0, indexBytes.capacity()))

// Copied to the GPU: reuse your
// ByteBuffers right away

// Need updates at runtime?
val dynamic = MeshBuffer.createDynamic(
    session, layout,
    vertexCount = 1000, indexCount = 3000)
```

[06:14](https://www.youtube.com/watch?v=mssk4zPs0zU&t=374s) You give it the layout, the vertex bytes, and 32-bit indices.

[06:18](https://www.youtube.com/watch?v=mssk4zPs0zU&t=378s) The data is copied to the GPU right away, so that you can reuse or release your ByteBuffers.

[06:24](https://www.youtube.com/watch?v=mssk4zPs0zU&t=384s) By default, the `MeshBuffer` is static.

[06:26](https://www.youtube.com/watch?v=mssk4zPs0zU&t=386s) It's set once and optimized for fast rendering.

[06:29](https://www.youtube.com/watch?v=mssk4zPs0zU&t=389s) And if you need to change it at runtime, `createDynamic` allocates a fixed capacity that you may update later.

[06:37](https://www.youtube.com/watch?v=mssk4zPs0zU&t=397s) Third, `CustomMesh`, a view into a `MeshBuffer`.

```kotlin
// A view: which indices to draw, and how
val road = CustomMesh
    .BuilderFromMeshBuffer(session, meshBuffer)
    // subset 0: asphalt
    .addSubset(TRIANGLES, indexOffset = 0,
        indexCount = roadCount)
    // subset 1: lane markings
    .addSubset(TRIANGLES,
        indexOffset = roadCount,
        indexCount = lineCount)
    .build()

// Shortcut: builds MeshBuffer + CustomMesh
CustomMesh.BuilderFromMeshData(session, layout)
```

[06:41](https://www.youtube.com/watch?v=mssk4zPs0zU&t=401s) It doesn't copy any data.

[06:43](https://www.youtube.com/watch?v=mssk4zPs0zU&t=403s) It says which range of indices to draw and how, triangles or triangle strips.

[06:49](https://www.youtube.com/watch?v=mssk4zPs0zU&t=409s) Here, the asphalt and the lane markings live in the same `MeshBuffer`.

[06:54](https://www.youtube.com/watch?v=mssk4zPs0zU&t=414s) Multiple custom meshes may share the same `MeshBuffer`, and when you don't need sharing, `BuilderFromMeshData` creates the mesh and the buffer in one step.

[07:04](https://www.youtube.com/watch?v=mssk4zPs0zU&t=424s) Finally, MeshEntity places the mesh in the scene.

```kotlin
// Place the mesh in the scene graph
val roadEntity = MeshEntity.create(
    session,
    mesh = road,
    // one material per subset
    materials = listOf(asphalt, paint),
    pose = Pose(Vector3(0f, 0f, -1f)),
    parent = session.scene.activitySpace)

// It's an Entity: move it, re-parent it
roadEntity.setPose(Pose(Vector3(0f, -0.5f, -1f)))
roadEntity.parent = tableAnchor

// Same mesh, another entity
val ghost = MeshEntity.create(
    session, road, listOf(glass, glass))
```

[07:08](https://www.youtube.com/watch?v=mssk4zPs0zU&t=428s) The mesh is the what.

[07:10](https://www.youtube.com/watch?v=mssk4zPs0zU&t=430s) The entity is the where.

[07:12](https://www.youtube.com/watch?v=mssk4zPs0zU&t=432s) Like the panel and Jay's car, it's an entity, so it has a position and a parent, the thing it is attached to.

[07:18](https://www.youtube.com/watch?v=mssk4zPs0zU&t=438s) Move the parent, and the road moves with it.

[07:21](https://www.youtube.com/watch?v=mssk4zPs0zU&t=441s) You also provide one material per subset, so that the same mesh may appear with different material in different entities.

### Building custom and dynamic meshes

[07:31](https://www.youtube.com/watch?v=mssk4zPs0zU&t=451s) So, now let's put it all together and build the mesh, the road mesh.

```kotlin
// Describe a vertex
val layout = VertexLayout.Builder()
    .addAttribute(POSITION, FLOAT3)
    .addAttribute(NORMAL, FLOAT3)
    .addAttribute(UV0, FLOAT2)
    .build()

// Path points -> ribbon of triangles
val (vertices, indices) = buildRoad(path)

// Build the mesh...
val roadMesh = CustomMesh
    .BuilderFromMeshData(session, layout)
    .addVertexData(vertices)
    .setIndexData(indices)
    .setTopology(TRIANGLES)
    .build()

// ...and put it in the scene
val road = MeshEntity.create(
    session, roadMesh, listOf(asphalt),
    parent = session.scene.activitySpace)
```

[07:35](https://www.youtube.com/watch?v=mssk4zPs0zU&t=455s) We define the vertex: the position, the normal, and UVs, the texture coordinates.

[07:40](https://www.youtube.com/watch?v=mssk4zPs0zU&t=460s) A small helper function turns user path into a ribbon.

[07:45](https://www.youtube.com/watch?v=mssk4zPs0zU&t=465s) For each point on the path, it adds two vertices on each side and two triangles that connect them to the next pair.

[07:52](https://www.youtube.com/watch?v=mssk4zPs0zU&t=472s) It's plain Kotlin, writing bytes into a ByteBuffer.

[07:54](https://www.youtube.com/watch?v=mssk4zPs0zU&t=474s) You don't need any kind of 3D tool.

[08:00](https://www.youtube.com/watch?v=mssk4zPs0zU&t=480s) `BuilderFromMeshData`, the shortcut we just saw, creates the mesh and the `MeshBuffer`, and computes the bounds for us.

[08:08](https://www.youtube.com/watch?v=mssk4zPs0zU&t=488s) Finally, `MeshEntity.create` places the `roadMesh` in the scene with the asphalt material.

[08:14](https://www.youtube.com/watch?v=mssk4zPs0zU&t=494s) But we want our track to grow live as the user draws that.

[08:18](https://www.youtube.com/watch?v=mssk4zPs0zU&t=498s) For that, we use a dynamic mesh.

```kotlin
// Fixed GPU capacity, allocated once
val track = CustomMesh.BuilderFromMeshData(
        session, layout,
        maxVertices = 2 * MAX_POINTS,
        maxIndices = 6 * MAX_POINTS)
    .setTopology(TRIANGLES)
    // Covers everything the track can become
    .setBounds(playArea)
    .build()

// Hand moved: upload only the new segment
val buffer = track.meshBuffer
buffer.updateVertexData(
    bufferIndex = 0,
    vertexData = segmentVertices,
    offsetInBytes = n * VERTEX_BYTES)
buffer.updateIndexData(
    segmentIndices,
    offsetInBytes = n * INDEX_BYTES)
```

[08:21](https://www.youtube.com/watch?v=mssk4zPs0zU&t=501s) The only difference, we give the builder the maximum capacity up front, so that the GPU buffers can be allocated.

[08:28](https://www.youtube.com/watch?v=mssk4zPs0zU&t=508s) Then, as the user draws, `updateVertexData` and `updateIndexData` upload only the new segments, not the entire mesh.

[08:37](https://www.youtube.com/watch?v=mssk4zPs0zU&t=517s) One gotcha: the renderer uses bounds to skip anything off-screen, so set the bounds to cover everything the track may become, like the play area here.

[08:48](https://www.youtube.com/watch?v=mssk4zPs0zU&t=528s) You may notice that the API is not the difficult part here.

[08:51](https://www.youtube.com/watch?v=mssk4zPs0zU&t=531s) The difficult part is the vertex math, the normals, texture coordinates, triangle order.

[08:57](https://www.youtube.com/watch?v=mssk4zPs0zU&t=537s) That's exactly the kind of code that AI is great at writing.

[09:01](https://www.youtube.com/watch?v=mssk4zPs0zU&t=541s) But let's say we want to put things one step farther.

[09:03](https://www.youtube.com/watch?v=mssk4zPs0zU&t=543s) Now Now, I want a checkered flag waving at the wind at the destination.

### Generating animated meshes with Gemini

[09:09](https://www.youtube.com/watch?v=mssk4zPs0zU&t=549s) Vertex animation math is the tricky part, so I asked Gemini.

[09:13](https://www.youtube.com/watch?v=mssk4zPs0zU&t=553s) In Android Studio, I gave it one prompt: "Using the Jetpack SceneCore Mesh API, build a 20x12 flag grid with checkerboard UVs, animate a wind wave every frame with updateVertexData, and recompute normals so that the lighting follows the folds."

[09:28](https://www.youtube.com/watch?v=mssk4zPs0zU&t=568s) A few seconds later, I got working code.

[09:31](https://www.youtube.com/watch?v=mssk4zPs0zU&t=571s) Let's look at what it did.

[09:34](https://www.youtube.com/watch?v=mssk4zPs0zU&t=574s) It built a flag mesh as the grid of 20 by 12 cells, two triangles per cell, and it made a smart choice with the layout.

```kotlin
// 20 x 12 cells = 273 vertices
val layout = VertexLayout.Builder()
    // buffer 0: updated every frame
    .addAttribute(POSITION, FLOAT3)
    .addAttribute(NORMAL, FLOAT3)
    .startNextBuffer()
    // buffer 1: never changes
    .addAttribute(UV0, FLOAT2)
    .build()

val flagMesh = CustomMesh.BuilderFromMeshData(
        session, layout,
        maxVertices = 273, maxIndices = 1440)
    .addVertexData(restPose)   // buffer 0
    .addVertexData(checkerUVs) // buffer 1
    .setIndexData(gridIndices)
    .setTopology(TRIANGLES)
    .setBounds(swingBounds)
    .build()
```

[09:41](https://www.youtube.com/watch?v=mssk4zPs0zU&t=581s) Positions and normals are placed in Buffer 0, and they updated every frame.

[09:46](https://www.youtube.com/watch?v=mssk4zPs0zU&t=586s) Texture coordinates are placed in Buffer 1, and they never change.

[09:50](https://www.youtube.com/watch?v=mssk4zPs0zU&t=590s) The mesh is dynamic, and the bounds are set to cover the full swing of the flag.

[09:56](https://www.youtube.com/watch?v=mssk4zPs0zU&t=596s) Every frame, each vertex gets a sine wave that grows towards the free edge, so that the pole side stays put.

```kotlin
// Gemini wrote the math
fun wave(t: Float) {
    for (i in 0 until 273) {
        val u = (i % 21) / 20f  // 0 at the pole
        setZ(i, 0.06f * u * sin(6f * u - 4f * t))
    }
    recomputeNormals()          // from neighbors
}

// Every frame: one upload (~6.5 KB)
while (isActive) withFrameNanos { ns ->
    wave(ns / 1e9f)
    flagMesh.meshBuffer.updateVertexData(
        bufferIndex = 0,
        vertexData = ByteBufferRegion(
            posNormals, 0, 273 * 24))
}
```

[10:05](https://www.youtube.com/watch?v=mssk4zPs0zU&t=605s) The normals are recomputed from the neighboring positions, which is what makes the folds catch the light.

[10:10](https://www.youtube.com/watch?v=mssk4zPs0zU&t=610s) `updateVertexData` uploads about 6.5 KB of data to the GPU each frame.

[10:16](https://www.youtube.com/watch?v=mssk4zPs0zU&t=616s) We do not recreate the mesh or the entity.

[10:19](https://www.youtube.com/watch?v=mssk4zPs0zU&t=619s) Now let's see it all together.

[10:21](https://www.youtube.com/watch?v=mssk4zPs0zU&t=621s) We place the car model on the table, then we use ARCore to draw the track.

[10:27](https://www.youtube.com/watch?v=mssk4zPs0zU&t=627s) The mesh flag we just-- the flag mesh we just built appears at the destination.

[10:32](https://www.youtube.com/watch?v=mssk4zPs0zU&t=632s) The car drives along the track, and we use glTF Introspection API to steer and spin the wheels of the car.

[10:39](https://www.youtube.com/watch?v=mssk4zPs0zU&t=639s) The mesh appears with the asphalt material, but we can also use mesh API together with the material API to change the look and feel of the road, like you can see on the video.

[10:48](https://www.youtube.com/watch?v=mssk4zPs0zU&t=648s) That is like that is some powerful stuff right at your finger fingertips.

### Summary and getting started

[10:54](https://www.youtube.com/watch?v=mssk4zPs0zU&t=654s) Today, we've covered a lot of ways to take your apps from 2D to spatial.

[10:59](https://www.youtube.com/watch?v=mssk4zPs0zU&t=659s) With the glTF Introspection API, you can now tap directly into the inner metadata of a glTF model, triggering embedded animations, transforming individual subnodes, and even swapping materials to alter its look and feel all at runtime.

[11:13](https://www.youtube.com/watch?v=mssk4zPs0zU&t=673s) Then, we saw that when pre-made 3D models weren't enough, we could leverage our brand new Mesh API to build custom geometry.

[11:21](https://www.youtube.com/watch?v=mssk4zPs0zU&t=681s) Whether you need to build static shapes from scratch or stream dynamic shapes dynamic geometry in real time, you now have the tools to do so.

[11:29](https://www.youtube.com/watch?v=mssk4zPs0zU&t=689s) And like Egor just showed, pairing those building blocks with Gemini means you don't even need to do the math alone.

[11:35](https://www.youtube.com/watch?v=mssk4zPs0zU&t=695s) Now, I'll repeat something I said earlier: everything you saw was running in an Android app, written in idiomatic Kotlin, right inside Android Studio.

[11:44](https://www.youtube.com/watch?v=mssk4zPs0zU&t=704s) I see a lot of laptops in the audience today, and I hope that Egor and I excited you enough to open up Android Studio and spin up the Android XR Emulator today.

[11:53](https://www.youtube.com/watch?v=mssk4zPs0zU&t=713s) Let your imagination run loose, and use your existing Android skills, along with what we showed today, to build amazing XR experiences.

[12:01](https://www.youtube.com/watch?v=mssk4zPs0zU&t=721s) On behalf of Egor and myself, thank you so much for joining us today, and enjoy the rest of the Android Dev Summit.