# Building audio-first experiences for intelligent eyewear with Gemini Nano 4

- Speakers: Amy Zeppenfeld (Developer Relations Engineer | Android XR), Alice Zheng (Senior Product Manager)
- Track: Android XR
- Video: https://www.youtube.com/watch?v=tW3mg0iyi_w
- Page: https://developer.android.com/events/dev-summit/talks/building-audio-first-experiences-for-intelligent-eyewear-with-gemini-nano-4

## About the talk

Discover how to build robust, hands-free audio experiences powered by on-device AI. We’ll cover everything you need: setting up your glasses activity, capturing mic and camera feeds, and using voice along with other audio cues as your ultimate UI. Through an exciting real-world showcase of the open-source XR Language Teacher built in the Google AI Edge Gallery app, learn to orchestrate Gemini Nano 4 (built on Gemma 4 and optimized for Android devices), and ML Kit, and leverage the LiteRT runtime to build fast XR experiences that hear, see, and speak naturally.

## Summary

- The Jetpack Projected library (alpha) bridges Android phones to intelligent eyewear. Adding a projected activity to your existing app projects content to display or audio-only glasses.
- ML Kit introduces alpha on-device speech libraries for audio-first interactions on select devices. It offers Text to Speech with 8 voices across 21 locales and Speech Recognition with a Gemini Nano advanced mode on Pixel 10.
- Gemini Nano 4 processes camera frames and transcribed questions together in one pass. It powers offline experiences by chaining on-device speech models locally on supported flagship phones.
- A two-pass design prioritizes real-time audio latency on smart glasses. Pass 1 streams spoken text using inline XML tags for mid-sentence locale switching, while Pass 2 extracts structured data in the background.
- The open-source XR Language Teacher implementation ships in the Google AI Edge Gallery app. Developers can adapt its code for other multimodal assistant use cases.

## Resources

- [Android XR Developer Documentation](https://goo.gle/xr-dev)
- [Android XR SDK and Developer Resources](https://developer.android.com/xr)
- [ML Kit APIs Documentation](https://developers.google.com/ml-kit)
- [Google AI Edge Offerings and Gallery App](https://developers.google.com/edge)
- [Install Android CLI](https://goo.gle/android-cli)

## Topics

intelligent-eyewear, android-xr, gemini-nano, on-device-ai, ml-kit, speech-recognition, text-to-speech

## Transcript

### Introduction

[00:06](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=6s) Hello, welcome, everybody.

[00:07](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=7s) So great to see everyone today.

[00:08](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=8s) Thanks for coming.

[00:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=10s) I'm Amy.

[00:11](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=11s) Today, I'll be joined by my colleague, Alice, and we're going to talk about how you can build incredible audio-first experiences for intelligent eyewear using Gemini Nano 4.

[00:21](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=21s) So let's go ahead and get started.

[00:26](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=26s) So when we talk about intelligent eyewear, what we're referring to is portable, lightweight glasses that are meant for, like, all-day wear.

[00:34](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=34s) This category is going to include both display and audio-only glasses, and with these devices, you're going to see some varied hardware here.

[00:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=42s) So the first thing that you'll notice is that they are going to have a touch pad, which can be used for user input, things like taps and swipes.

[00:48](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=48s) They'll also have speakers, a camera, and a microphone.

[00:52](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=52s) And then they could also have a display or not, or potentially some combination of this.

[00:58](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=58s) Today, so we're talking about audio-first and how to build an engaging, intelligent, on-the-go experiences.

[01:03](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=63s) And one of the things that's interesting with audio only is that voice is what becomes your UI here.

[01:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=70s) And on-device AI is great for developers that are looking for offline and on-the-go experiences.

### Building for Android XR

[01:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=79s) And there's great news.

[01:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=80s) So if you're already building for Android, you're already building for Android XR.

[01:24](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=84s) Existing Android apps are designed to extend well to audio glasses, delivering spoken notifications and other audio over Bluetooth.

[01:31](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=91s) And if you want to go further and differentiate your app for XR, there are lots of ways to do that.

[01:37](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=97s) So in this talk, we are talking about on-device models, but that's by no means an exhaustive list.

[01:43](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=103s) We-- there's also cloud models.

[01:44](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=104s) There's things like the Gemini Live API.

[01:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=106s) There's system agents you can integrate with, like Gemini, and all of these can help you build the experience that's best for you.

[01:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=114s) So how do you build this experience that hears, sees, and speaks naturally?

[01:59](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=119s) Today, we're going to walk you through building these robust, hands-free experiences that are powered by on-device AI.

[02:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=125s) I'm going to walk you through the basics: how to build for glasses, how to set up your activity, get the permissions you need, and then how to use ML Kit for conversational UI.

[02:13](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=133s) And then I'll hand it over to Alice, and she's going to talk you through some real examples that use Gemini Nano 4.

[02:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=139s) So let's go ahead and dive into the code.

[02:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=143s) So with glasses, the phone is what does the heavy lifting, and the glasses content is going to be projected from the phone to the glasses.

[02:31](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=151s) So there's two things that you need to bridge these devices when you're first getting set up.

[02:36](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=156s) What you're going to need first is the Jetpack Projected library.

[02:39](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=159s) You'll need to add this dependency, and this is what's going to contain the APIs to power that projected experience.

```kotlin
dependencies {
   implementation("androidx.xr.projected:projected:1.0.0-alpha13")
}
```

[02:45](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=165s) And then you're also going to need to create a new activity specifically for the glasses.

```kotlin
class ProjectedMainActivity : ComponentActivity() {
   override fun onCreate(savedInstanceState: Bundle?) {
       super.onCreate(savedInstanceState)
       setContent {
           // Glasses content
       }
   }
}
```

[02:49](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=169s) Then you're going to leverage that Jetpack Projected library both inside your existing app and inside this new activity so the two devices can communicate smoothly whenever needed.

[03:02](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=182s) So just like any normal activity you've seen, you're going to declare this in your manifest.

```xml
<activity
   android:name=".ProjectedMainActivity"
   android:exported="false"
   android:requiredDisplayCategory="android.hardware.display.category.XR_PROJECTED"
   android:label="Example activity for audio glasses">
   <intent-filter>
       <action android:name="android.intent.action.MAIN" />
       <category android:name="android.intent.category.XR_PROJECTED_LAUNCHER"/>
   </intent-filter>
</activity>
```

[03:07](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=187s) The main difference is here being that you do need to use the `XR_PROJECTED` as the `requiredDisplayCategory`.

[03:12](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=192s) That's so that the system knows that this is-- this activity is for eyewear.

[03:16](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=196s) And then you're also going to want to use this intent category here, the `XR_PROJECTED_LAUNCHER`.

[03:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=200s) What that does is that allows Gemini to discover your app so that users can launch it hands-free on the go.

[03:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=210s) All right.

[03:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=210s) So once you have that activity set up, as I mentioned, you'll use that Projected library to power that app experience.

[03:36](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=216s) This library contains all the tools that you need to build out the full glasses experience.

[03:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=220s) So let's just walk through a few of the key APIs that you'll use here.

[03:44](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=224s) So the first one I want to call out is the `isProjectedDeviceConnected` method.

```kotlin
// Continuously monitor whether the glasses are connected
val isGlassesConnected by ProjectedContext.isProjectedDeviceConnected(
   context,
   scope.coroutineContext
).collectAsStateWithLifecycle(initialValue = false)

// Launch the activity
val options = ProjectedContext.createProjectedActivityOptions(context)
val intent = Intent(context, ProjectedMainActivity::class.java)
context.startActivity(intent, options.toBundle())

// Check if the device is audio only or has a display
val projectedDeviceController = ProjectedDeviceController.create(this@ProjectedMainActivity)
isVisualUiSupported = projectedDeviceController.capabilities.contains(CAPABILITY_VISUAL_UI)
```

[03:49](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=229s) This is how you're going to continuously monitor the glasses connection.

[03:53](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=233s) You use this along with a device life cycle to know when the glasses are actually available, so not just when they're paired or powered on, but also, like, when they're being worn and they're ready for use.

[04:03](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=243s) And this is what, so, yeah.

[04:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=245s) So this is going to let you know exactly when you're ready to launch that activity.

[04:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=250s) So to do this, when you're ready to launch the activity, there's some specific options that you're going to need to set when you start the activity.

[04:17](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=257s) And those, you're going to use the `createProjectedActivityOptions` on the `ProjectedContext`.

[04:22](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=262s) And again, then you'll use that with the context on the intent there.

[04:29](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=269s) And then one more tool that you're going to want to use is a `ProjectedDeviceController`.

[04:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=273s) This is how you're going to check what type of glasses the user is using.

[04:37](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=277s) So just like I said before that intelligent eyewear includes display and audio only glasses, it-- you're going to want to see if the capabilities contain this `CAPABILITY_VISUAL_UI`, and that's going to allow you to adapt your app gracefully for different devices and tailor an experience that optimize for that device.

[04:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=297s) So as you're building out this differentiated experience, you may want to access the device's hardware, for example, leveraging the glasses' camera.

[05:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=305s) So to do this from a projected activity, you'll need to request the appropriate permissions using `ProjectedActivityCompat`.

```kotlin
// In a projected activity
private fun requestProjectedPermissions() {
   lifecycleScope.launch {
       try {
           ProjectedActivityCompat.requestPermissions(
               this@PermissionsFromProjectedActivity,
               arrayOf(Manifest.permission.CAMERA),
               // CAMERA_PERMISSION_REQUEST_CODE is a developer defined constant
               requestCode = CAMERA_PERMISSION_REQUEST_CODE,
           )
       } catch (e: IllegalStateException) {
           // Thrown when the projected system service can't be bound.
           Log.e(TAG, "Request failed: projected service unavailable.", e)
       }
   }
}
```

[05:13](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=313s) And even if your app already has the permissions that are needed for the phone, it does you do still need to request the specific permissions for the glasses' hardware, as well.

[05:21](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=321s) And whenever working with permissions, remember to keep in mind follow best practices for Android in general.

[05:26](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=326s) Declare the necessary permissions in the manifest, ask for permissions within the context that they're needed, and only request the permissions that you actually need for your app to function.

[05:37](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=337s) So, of course, we have this documented on our site on developer.android.com.

[05:41](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=341s) But we also have a brand new skill, the projected hardware skill.

[05:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=346s) You can use this with the Android CLI in Android Studio to set up your permissions workflow, and you can get access to this skill, and then all the other Android skills, as well.

```shell
android skills add projected-hardware
android skills add --all
```

### Designing audio-first experiences

[06:01](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=361s) Okay, so now that we're connected, aware of our hardware capabilities, and have the right permissions, let's talk about the tools that you have to build out that audio experience.

[06:13](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=373s) So when designing an audio first experience, we do want to make sure that we're creating a high quality experience for users.

[06:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=380s) And this could inc- include a few key interactions.

[06:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=383s) So first is using voice as output, so allowing your voice to speak naturally to your user.

[06:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=388s) Next would be using earcons to send specific audio signals to the user.

[06:32](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=392s) And then also handling input from the user, whether that's like through the touchpad using taps and swipes, or also if you want to allow the user to speak to your app.

[06:44](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=404s) So the first thing we're going to talk about is just that using voice as output.

[06:48](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=408s) So for this, I want to introduce ML Kit's new GenAI Text To Speech library.

[06:53](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=413s) This is now available in alpha, and it's on select Pixel and Samsung devices.

[06:56](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=416s) This is something we're super excited about, especially for these audio-first experiences.

[07:00](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=420s) If you've worked with Text To Speech before, you probably know that one of the pain points can be that that System Voice is not always as natural sounding as you might like.

[07:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=430s) And connecting to cloud models for other voices could leave you with potential latency or additional costs.

[07:17](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=437s) So ML Kit gives you a much more natural-sounding speech compared to that system TTS.

[07:24](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=444s) And ML Kit offers eight unique voices that you can choose from that have different characteristics to cater to your app.

[07:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=450s) And then also ML Kit offers robust language support with 21 different locales that you can choose from.

[07:39](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=459s) And on top of that, one of the huge benefits here is that after that initial download, this is going to work completely on device.

[07:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=466s) So this model will be downloaded to the system and it's going to be shared across apps, so it's not going to increase the size of your APK.

[07:52](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=472s) And not only is it available offline, which is especially important for you know glasses users that are on the go, they might not have Wi-Fi or they might be in poor coverage areas, but you're also likely again to see better performance, since you don't have to make that round trip ca- call to a server.

[08:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=490s) All right.

[08:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=490s) So let's look at the details of the library, and exactly how we can build with it.

[08:14](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=494s) So first, of course, you're just going to need to add the dependency.

```kotlin
dependencies {
   // ML Kit GenAI TTS SDK
   implementation("com.google.mlkit:genai-text-to-speech:1.0.0-alpha1")
}
```

[08:17](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=497s) So here all you need is the ML Kit's Text To Speech SDK.

[08:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=503s) And then you can get started writing your code.

[08:26](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=506s) So, the GenAI ML Kit library includes the Speech Synthesizer APIs.

```kotlin
// 1. Initializing the SpeechSynthesizer client.
val options = SpeechSynthesizerOptions.Builder(Locale.US).build()
val client = SpeechSynthesis.getClient(options)

// 2. Check and download the voice model.
val voiceInfo = client.getSupportedVoices().firstOrNull()

if (voiceInfo != null && client.checkStatus(voiceInfo) == FeatureStatus.DOWNLOADABLE) {
   client.download(voiceInfo).collect { status -> println("Download status: $status") }
} else if (voiceInfo == null) {
   // Tip: Fallback to system TTS if ML Kit voice is unavailable
}
```

[08:31](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=511s) This offers you a straightforward way to implement the voices of your choosing.

[08:36](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=516s) And the first thing step that you're going to do here is initialize your client using that speech synth- synthesizer again, setting the locale that you choose.

[08:47](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=527s) And then next, you're going to check the download status and see if and select the voice.

[08:52](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=532s) In this example, we're choosing the first voice, but I also want to call out here, notice that we're using `firstOrNull`.

[08:58](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=538s) So as I mentioned, ML Kit requires specific capable phone, so either the Samsung or Pixel devices.

[09:08](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=548s) So if you were to get a null value, that could mean that maybe the user's not using the like a compatible phone.

[09:15](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=555s) And then you just want to make sure that you have your fallback in place, whether that's going back to the System Text To Speech or like playing a recorded message or something like that.

[09:26](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=566s) Okay, and then you'll use speech synthe- `SpeechSynthesizerRequest` to construct the request, giving it the input text, and then setting the usage and the content type.

```kotlin
// 3. Construct the SpeechSynthesizerRequest
val request = SpeechSynthesizerRequest(
   input = SpeechSynthesizerRequest.Input.Text("Hello world", voiceInfo),
   audioAttributes = AudioAttributes.Builder()
       .setUsage(AudioAttributes.USAGE_MEDIA)
       .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
       .build()
)

// 4. Speak and collect status
client.synthesize(request).collect { response ->
   when (response) {
       is SpeechSynthesizerResponse.StatusUpdate -> println("Playback status: ${response.status}")
   }
}
```

[09:38](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=578s) Play the speech, monitor the status, and no- so that you know when the speech completes, and then collect any errors that could occur.

[09:47](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=587s) And then close out any resources when needed.

```kotlin
// 4. Speak and collect status
client.synthesize(request).collect { response ->
   when (response) {
       is SpeechSynthesizerResponse.StatusUpdate -> println("Playback status: ${response.status}")
   }
}

// 5. Release resources when finished (e.g., on Activity destroy)
override fun onDestroy() {
   super.onDestroy()
   client.close()
}
```

[09:52](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=592s) And so that's pretty straightforward, right?

[09:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=594s) With these steps, you can really transform the sound of your app going from something that sounds like this…

[10:01](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=601s) Hello, Android developers.

[10:03](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=603s) It's great to see you all today.

[10:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=605s) We hope you're all enjoying our presentation on audio-first experiences.

[10:09](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=609s) To something that sounds more like this.

[10:13](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=613s) Hello Android developers.

[10:15](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=615s) It's great to see you all today.

[10:16](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=616s) We hope you're all enjoying our presentation on audio-first experiences.

[10:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=623s) Much better, right?

[10:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=627s) Okay.

[10:29](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=629s) So, ML Kit's Text to Speech library, this is a super powerful tool, but you're not only limited to using these voices, you can also play earcons with this library.

```kotlin
// 1. Register Earcon resource with Builder
SpeechSynthesizerOptions.Builder(Locale.US)
   .addEarcon(
       name = "shutter_sound",
       Uri = "path/to/earcon.file".toUri()
   )
   .build()

// 2. Request Earcon playback
val request = SpeechSynthesizerRequest(
   input = SpeechSynthesizerRequest.Input.Earcon(
       name = "shutter_sound"
   ),
   audioAttributes = AudioAttributes.Builder()
       .setUsage(AudioAttributes
           .USAGE_ASSISTANCE_SONIFICATION)
       .setContentType(AudioAttributes
           .CONTENT_TYPE_SONIFICATION)
       .build()
)
```

[10:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=640s) So earcons, as the name implies, are basically the audio equivalent of icons.

[10:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=646s) This is gonna allow you to just play short sounds to communicate with your user.

[10:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=650s) And just like visual icons with earcons, the user's gonna need to know you know, what these are and what they mean.

[10:56](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=656s) So initially you may wanna pair them with voice notifications to make that clear.

[11:04](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=664s) And we've already created some standardized sounds that are available on the Android Design Hub, you can use those, you can download them straight from there and you can use those for things like app open, app close, error notifications, and muting or unmuting.

[11:18](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=678s) And but you also have the option to create your own custom earcons as well.

[11:25](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=685s) So with earcons, there are some best practices that we recommend following whether you're using the downloaded ones from the Design Hub, or you're creating your own.

[11:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=693s) So You wanna be thoughtful with the way that you use earcons, and how how you use them.

[11:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=700s) So there's just some things to keep in mind, like for starters, just try not to overuse earcons.

[11:45](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=705s) Don't replay the same sounds over and over again, this can cause what's called ear fatigue.

[11:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=710s) And remember that your earcons should be super and intuitive, also use familiar pitches and rhythms when you're implementing them.

[11:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=717s) And then, try to avoid overly complex earcons that need explanation, these might just be better suited as a voice notification.

[12:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=725s) And then always keep in mind that what might sound subtle in the studio or on your system, on your computer speakers, that that can become irritating after like re- repeated playback on your to your users.

[12:18](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=738s) So do your best not to overload your users with sounds that might confuse them or lead them to imp- ignoring important alerts.

[12:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=747s) All right.

[12:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=748s) So we now we have the ability to provide audio cues and feedback whether that's through the natural speech or using earcons.

[12:37](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=757s) But what about input?

[12:38](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=758s) So of course there's a touchpad, which or dedicated buttons also on the device, which you can use to create tap and swipe interactions.

[12:49](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=769s) But you also may wanna allow for this conversational UI, for this you can use ML Kit's GenAI Speech Recognition API.

[12:55](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=775s) Currently in alpha and available on select devices, but basic this runs in two distinct modes.

[13:01](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=781s) So there's basic mode which uses traditional on-device speech recognition, and it's available on Android devices with API level 31 and higher.

[13:11](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=791s) And then there's also the advanced mode which uses Gemini Nano, this is offered on Pixel 10 devices above and with this one you'll see better quality and better language coverage as well.

[13:22](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=802s) So `SpeechRecognizer` follows a very similar code pattern to `SpeechSynthesizer`.

[13:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=808s) You create your client, check if the model's available, download if needed, construct the request, and release your resources.

[13:34](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=814s) So let's just focus here on the main differences where you're creating your request, capturing that mic audio, and then processing the result.

[13:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=822s) Again, going to need to add your necessary dependencies.

```kotlin
dependencies {
   // ML Kit Speech Recognition
   implementation("com.google.mlkit:genai-speech-recognition:1.0.0-alpha1")
}
```

[13:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=826s) All that you need here is the Speech Recognition API.

[13:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=830s) And then you can get started creating your request.

[13:52](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=832s) So you'll use a suspend function when you're defining your recognition logic.

```kotlin
suspend fun startMyRecognition() {
   // choose locale and mode
   val options: SpeechRecognizerOptions =
       speechRecognizerOptions {
           locale = Locale.US
           preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
       }

   val speechRecognizer: SpeechRecognizer = SpeechRecognition.getClient(options)

   val request: SpeechRecognizerRequest =
       speechRecognizerRequest { audioSource = AudioSource.fromMic() }
...
```

[13:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=837s) And then specify your speech recognition options, setting the locale and the `preferredMode`.

[14:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=845s) And then this is also where you'll choose the `audioSource` and define the request itself.

[14:12](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=852s) Then you'll start the recogni- recognition and collect a response.

```kotlin
speechRecognizer.startRecognition(request).collect { response ->
   when (response) {
       is SpeechRecognizerResponse.PartialTextResponse -> {
           // Real-time: Act immediately if trigger phrase is heard
           handleResponse(response.text, speechRecognizer)
       }
       is SpeechRecognizerResponse.FinalTextResponse -> {
           // Final result and recognition
           handleResponse(response.text, speechRecognizer)
       }
       is SpeechRecognizerResponse.CompletedResponse -> {
           resetListeningUi()
       }
       is SpeechRecognizerResponse.ErrorResponse -> {
           Log.e("SpeechRec", "Error: ${response.e.errorCode}", response.e)
           resetListeningUi()
       }
...
```

[14:15](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=855s) So with `SpeechRecognizer`, you have four potential responses: partial, final, error, and complete.

[14:22](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=862s) Partial is gonna be looking at the real-time stream, so that is used for think like hotwords or keywords.

[14:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=868s) Final is taking the final audio stream after fully analyzing, updating it, and then completed is after the speech is completed, and then error collecting any errors that could occur.

[14:41](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=881s) And here's how we put it all together.

```kotlin
// Helper function to check for multiple phrases and trigger the right action
private suspend fun handleResponse(text: String, recognizer: SpeechRecognizer) {
   val savePhrases = listOf("save", "bookmark", "remember")
   if (savePhrases.any { text.contains(it) }) {

       // 1. Update app state immediately
       saveItemForLater()

       // 2. Stop the recognizer to enforce the action and prevent TTS echo
       recognizer.stopRecognition()

       // 3. Play the mute earcon for immediate, non-verbal confirmation
       playEarcon()

       // 4. Provide voice feedback
       playAudioFeedback("Saved")
   }
}
```

[14:43](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=883s) So let's say we want a user to be able to use a voice command to tell the app to save some type of content for later.

[14:49](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=889s) We could use a single explicit voice command that we've told them to use or you could allow them to use a number of keywords and phrases.

[14:55](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=895s) So here we're set- we're allowing them to use: save, bookmark, or remember.

[15:00](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=900s) As soon as `SpeechRecognizer` detects that the user has used one of those phrases, it's gonna stop the It's gonna save the content and then stop the speech recognition, play a custom earcon and use ML Kit's GenAI Text to Speech to tell the user the content has been saved.

### XR Language Teacher case study

[15:16](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=916s) Okay.

[15:17](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=917s) So, we went through a lot.

[15:18](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=918s) It might feel a bit abstract.

[15:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=920s) If you're like me, you want to see some real life examples.

[15:24](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=924s) And Alice and I want to walk you through exactly that.

[15:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=927s) She's going to walk you through the XR Language Teacher in the Google AI Edge Gallery app.

[15:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=930s) Alice's team built it, and she's going to show you exactly how it works.

[15:34](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=934s) Over to you.

[15:39](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=939s) Thanks, Amy.

[15:41](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=941s) Yeah.

[15:45](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=945s) So, I like to start with something a lot of you would recognize.

[15:48](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=948s) I'm learning Japanese, and I've spent hours in language learning apps, flashcards, streaks, the whole thing.

[15:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=954s) And then, I land in Tokyo, I walk into a market hall and realize almost nothing in front of me are the actual vocabulary that I studied.

[16:02](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=962s) There is a wooden cup-and-ball toy on a market stall that I recognize by sight, but I don't know how to speak in Japanese.

[16:08](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=968s) There's an ornament in a shop window I want to buy and cannot name.

[16:13](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=973s) That is exact moment I'm most motivated to learn a word.

[16:16](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=976s) Not on my couch looking at a language app.

[16:18](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=978s) It is that that moment, I'm looking at the thing that I want to learn it.

[16:21](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=981s) So, here's what I want.

[16:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=983s) I'd like to wear my smart glasses all day, just walk around.

[16:26](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=986s) And whenever something catches my eyes, just ask about it out loud.

[16:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=990s) And then, at night, when I go back to my hotel, everything I asked about would be just waiting for me like vocabulary cards on my phone.

[16:37](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=997s) And this is what we built.

[16:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1000s) Let's watch a short clip.

[16:43](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1003s) So, we call this the XR Language Teacher.

[16:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1010s) Hi there.

[16:51](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1011s) What are you looking to learn about today?

[16:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1014s) And you can customize your learning experience in the teacher settings.

[16:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1017s) For example, pick the language you want to learn and pick the specific teacher persona that works for you.

[17:12](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1032s) Oh, yeah.

[17:12](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1032s) And you can switch between glasses and phone mode.

[17:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1039s) How can I say these grapes in Spanish?

[17:25](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1045s) Looking at this, those delicious grapes are called uvas.

[17:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1050s) In Spanish, you can say, "Estas uvas son deliciosas," to describe them.

[17:38](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1058s) I want to say green grapes, specifically.

[17:45](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1065s) Let me take a look.

[17:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1066s) To specify the green grapes, you can say, "Uvas verdes.

[17:48](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1068s) Me gustan las uvas verdes," is a great way to tell someone you like them.

[17:58](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1078s) How do you say this in French, and how can I ask for three of them?

[18:04](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1084s) Let's check this out.

[18:06](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1086s) Those delicious looking pastries are called des madeleines.

[18:09](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1089s) In French, to ask for three, you can say, "Je voudrais trois madeleines, s'il vous plaît."

[18:18](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1098s) How to ask for change with this?

[18:25](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1105s) Let's see what we have here.

[18:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1107s) To ask for change, you can say, "Otsuri wa arimasu ka?"

[18:31](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1111s) Which means, "Do you have change?"

[18:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1113s) You can also use, "Otsuri o onegaishimasu," to politely request change.

[18:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1120s) And then, everything you learned is automatically saved as vocabulary cards for you.

[18:44](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1124s) La botella de agua.

[18:44](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1124s) Esta es una botella de agua.

[18:53](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1133s) Yeah, and you can test your memory and practice in the study tab.

[19:04](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1144s) All right.

[19:05](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1145s) Yeah.

[19:06](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1146s) So, everything you just saw ran on the device, no network in the loop at all.

[19:11](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1151s) I travel with an international data plan, which is usually fine right until the moment I'm inside a market hall or maybe underground or behind thick concrete.

[19:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1159s) The moment you want this the most is often this the moment the signal is the worst.

[19:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1163s) And I would say something, an experience that dies when the bars drop isn't real travel companion, which is why we made the XR Language Teacher work completely offline and on device.

### Chaining on-device models

[19:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1173s) So, how was this built?

[19:35](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1175s) Architecturally, Amy already showed you how to use a projected camera frame from glasses into the phone.

[19:41](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1181s) She showed you how to use ML Kit, speech recognition, to turn text into speech, and also ML Kit Text to Speech turning text back to speech.

[19:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1190s) What we still needed is a brain in the middle, something that can take a look at the camera frame and the question together, and work out what the user is actually asking about.

[19:59](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1199s) That brain is Gemini Nano 4, built on top of Gemma 4, Google's open-source model running locally on the phone.

[20:07](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1207s) So on supported flagship Android devices, you can get optimized performance by using the ML Kit APIs directly.

[20:14](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1214s) The Android system will take care of the model delivery for you, so there's nothing for you to bundle in your app.

[20:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1220s) So to put all this together, the microphone goes into speech recognition, and the camera frame and the transcript together goes into the Gemini Nano 4, and Gemini Nano 4 writes out into the speech synthesis and then comes out from the speaker.

[20:34](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1234s) Three models, chained, all running locally on your phone.

[20:39](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1239s) And here are a few reasons why Gemini Nano 4 is particularly suited for this problem.

[20:44](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1244s) First, it sees and reads in one pass.

[20:48](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1248s) Through ML Kit, Gemini Nano 4 handles text and images natively, so you're not using a detector, then a classifier, then a translator.

[20:56](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1256s) It is one model that look at the camera frame and the question together.

[21:01](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1261s) And second is its extensive language support.

[21:04](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1264s) Gemini Nano 4 was trained on over 140 languages.

[21:08](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1268s) And also, for language tutor, translation on its own is not enough.

[21:12](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1272s) You want idioms, you want formal versus casual registers, you also want example sentence a real person would say.

[21:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1279s) And lastly, it is steerable enough to have a personality.

[21:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1283s) Practicing pronunciation out loud could be intimidating.

[21:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1287s) And this model follows a system prompt well, so user can pick their own teacher persona.

[21:32](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1292s) And maybe a patient and encouraging one could make people more willing to try.

[21:38](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1298s) And here's one more thing I like to highlight about the design.

[21:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1302s) While the conversation is happening in the glasses mode, the app is quietly writing vocabulary cards in the background.

[21:49](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1309s) The word, the translation, the example sentence, and the photo from the moment you ask your question.

[21:55](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1315s) The user doesn't need to manually tap anything.

[21:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1317s) They're just walking and talking.

[21:59](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1319s) And then later that day, everything are just waiting for them on their phone, which is exactly what I wanted when I was in Tokyo.

[22:06](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1326s) So audio first does not have to mean audio only.

[22:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1330s) Glasses is a right surface for curiosity, and your phone is a right surface for the reflection afterwards.

[22:16](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1336s) Let the hands-free moment leave something behind.

[22:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1340s) Now, let's zoom in on a few practical tips for building real-time experiences with Gemini Nano 4 and ML Kit that we learned from building this demo.

### Prioritizing real-time latency with two-pass design

[22:30](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1350s) Tip 1, prioritize real-time latency with a two-pass design.

[22:35](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1355s) On audio glasses, when you ask, "What is that?"

[22:39](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1359s) Every moment of silence is noticeable.

[22:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1362s) In the example of XR Language Teacher, we need two things for each turn of the user interaction from the model: an immediate spoken answer, and also a structured vocabulary card to save to the app.

[22:53](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1373s) If we ask the model for two things together in a single pass, then every token spent on that background data generation will delay the first spoken sentence.

[23:03](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1383s) So instead, we separate what the user needs right now from what the app needs a few seconds later.

[23:10](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1390s) So we split each turn into two passes.

[23:13](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1393s) In Pass 1, we stream the camera frame and the question for the spoken reply alone, triggering text-to-speech as soon as the first sentence completes.

[23:22](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1402s) And because that TPU or NPU finish generating that short reply long before the text-to-speech finishes speaking out loud, we actually get a multi-second window where the user still listening, and the chip is already sitting idle.

[23:35](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1415s) And since Pass 1 already identified the object in its text reply, Pass 2 doesn't even need the image input.

[23:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1422s) We just pass Pass 1's reply back into that idle chip to extract and save the structured card before the teacher even finishes talking.

[23:53](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1433s) Now, let's zoom in to Pass 1, which wi-- brings me to the second tip.

[23:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1437s) Use lightweight XML tags when streaming.

[24:01](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1441s) Because Pass 1 streams directly to text-to-speech, you don't want heavy JSON object to hold up your first spoken word.

[24:09](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1449s) Instead, you want plain text that can speak immediately with lightweight, on-- inline tags whenever the app needs to take action mid-stream.

[24:17](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1457s) So let me show you the problem that led us here.

[24:20](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1460s) Remember that wooden cup-and-ball toy on the market stall I mentioned?

[24:25](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1465s) In Japanese, it's called "kendama".

[24:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1468s) Early on, the model recognized it from the camera frame and returned the right answer, "That's kendama, a wooden toy."

[24:36](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1476s) So we handed that complete sentence into the English text-to-speech session, and here is what came out.

[24:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1482s) That's ken-mi, a wooden toy.

[24:45](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1485s) That doesn't sound quite right.

[24:47](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1487s) Instead, that sentence should actually sound…

[24:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1490s) That's kendama, a wooden toy.

[24:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1494s) So why did that happen?

[24:56](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1496s) Kendama is written with hiragana, "ken", and the kanji character, "dama".

[25:01](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1501s) And the English text-to-speech engine handles the phonetic hiragana like "ken" just fine, but completely falls apart when it hits the kanji part and turning dama into something that sounds like me.

[25:14](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1514s) And this happens actually across many different languages in different ways if you don't switch locale.

[25:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1519s) For example, in Spanish, you might hear…

[25:21](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1521s) Suh-BOH-luh…

[25:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1523s) When you're actually expecting…

[25:24](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1524s) Cebolla.

[25:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1528s) And similarly, in French, you might hear…

[25:31](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1531s) PAIN oh chock-uh-lat…

[25:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1533s) When you are actually expecting…

[25:34](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1534s) Pain au chocolat.

[25:36](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1536s) Yes, a TTS session is usually designed for a single locale at a time, but a language teacher need to code-switch in every single sentence.

[25:46](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1546s) So, in pass 1, we ask the model to wrap every target language word or phrase inside the target XML tag as it streams.

[25:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1554s) And as chunks arrives, our client split on target, route each span to the matching Text To Speech locale, which could be Japanese, Spanish, French, or whatever language you're learning, and then play them back seamlessly, switching accent mid-sentence, so word like kendama comes out right.

[26:11](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1571s) So, the take away here is the model already knows which vocabulary belongs to which language, which words belongs to which language, and having it tag target spans inline while streaming, it's far cleaner than post-processing this whole thing yourself afterwards.

[26:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1587s) And here's pass 1 in code.

```kotlin
// 1. Pass 1 System + Turn Prompt (XrLanguageTeacherPersona.kt)
// System: "Always start immediately with <reply>...</reply> (strictly 1 to 3 sentences total).
//   Inside <reply>, wrap every Japanese word AND every complete Japanese example sentence
//   inside <target>...</target> tags (never output <card>)."
val pass1TurnPrompt = """
  The student is looking at the camera view and asked: "$question". Answer in English
  teaching Japanese inside <reply>...</reply>: if they refer to 'this'/'that' or point to
  an object, ground it to the focal item in the camera frame; if they ask for a phrase or
  greeting, teach that phrase.
""".trimIndent()

// 2. Stream multimodal dialogue (ImagePart + TextPart) and split <target> tags to TTS
val generativeModel = Generation.getClient()
val pass1Request = generateContentRequest(ImagePart(frame), TextPart(pass1TurnPrompt)) {
  temperature = 1.0f; topK = 1; maxOutputTokens = 1024
}
generativeModel.generateContentStream(pass1Request).collect { chunk ->
  val delta = chunk.candidates.firstOrNull()?.text.orEmpty()
  streamReplyAndSplitTargetTagsToTts(delta) // Speaks 1st complete sentence immediately
}
// Streams: "<reply>That's <target>けん玉</target> — a wooden toy. <target>けん玉をください。</target></reply>"
```

[26:29](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1589s) At the top, we use the prompt to instruct the model to reply in one to three sentence inside the reply tags and then wrap every Japanese word or example sentence inside the target tags.

[26:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1600s) Then we pass the camera `ImagePart` and the prompt `TextPart` into ML Kit's `generateContentStream`, splitting target spans into Text To Speech as each sentence arrives.

[26:53](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1613s) And that brings me to pass 2 and our third tip.

[26:57](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1617s) While using lightweight XML tags are great for a live audio stream, use structured output, which are built to return a clean, typed object when saving data to your app.

[27:08](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1628s) Because our XR Language Teacher is conversational, users can also ask follow-up questions where no new word is taught.

[27:16](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1636s) And when a word is taught, the database needs clean dictionary fields to create the vocabulary cards.

[27:23](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1643s) So instead of writing brittle regular expressions to parse pass one's transcript, we let pass 2 extract type-safe object in the background.

[27:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1653s) And to enforce the schema in pass 2, we use ML Kit's structured output API, which under the hood is using constrained decoding.

```kotlin
import com.google.mlkit.genai.prompt.Generation
import com.google.mlkit.genai.prompt.generateTypedContentRequest
import com.google.mlkit.genai.schema.annotations.Generable
import com.google.mlkit.genai.schema.annotations.Guide

// Pass 2 schema (XrLanguageTeacherGenerableSchemas.kt):
@Generable(description = "Vocabulary flashcard for the taught word or short phrase")
data class GenerableVocabCard(
  @Guide(description = TeacherResponseSchema.WORD_FIELD_DESCRIPTION) val word: String,
  @Guide(description = "Native-language meaning or translation") val meaning: String,
  @Guide(description = "Target-language example sentence") val example: String,
  @Guide(description = "Native-language translation of the example sentence") val example_translation: String,
)

@Generable(description = "Optional vocabulary card extracted from a teacher reply")
data class GenerableOptionalVocabCardWrapper(
  @Guide(description = "Vocabulary flashcard if a vocabulary term was taught, or null otherwise")
  val card: GenerableVocabCard?
)
```

[27:42](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1662s) You simply annotate a data class with `@Generable` and each field with `@Guide`, just like our vocabulary card class example here.

[27:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1670s) And by wrapping the nullable card field, pass 2 returns a populated card when a new word was taught, or null on purely conversational turns.

[28:00](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1680s) And here is pass 2 in code.

```kotlin
// 1. Pass 1 Streamed Output -> Pass 2 Extraction Prompt (no image needed!)
val replyWithTargetTags =
  "<reply>That's <target>けん玉</target> — a wooden toy. <target>けん玉をください。</target></reply>"

val pass2Prompt = """
  If the teacher reply below teaches or introduces a specific Japanese vocabulary item, extract
  only its core focal dictionary term into "card" with keys "word", "meaning", "example", and
  "example_translation". If purely conversational with no focal term taught, set "card" to null:
  $replyWithTargetTags
""".trimIndent()

// 2. Extract @Generable card on idle NPU while TTS audio plays
val pass2Request = generateContentRequest(TextPart(pass2Prompt)) {
  temperature = 1.0f; topK = 1; maxOutputTokens = 256
}
val typedRequest =
  generateTypedContentRequest(pass2Request, GenerableOptionalVocabCardWrapper::class)
val card: GenerableVocabCard? =
  generativeModel.generateContent(typedRequest).candidates.firstOrNull()?.response?.card
// Extracts: {"word": "けん玉", "meaning": "a wooden toy", "example": "けん玉をください。", ...}
```

[28:02](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1682s) At the top, we take pass 1's streamed reply text, no image needed, and build our extraction prompt.

[28:09](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1689s) Then, while the user is still listening to Text To Speech, we pass the prompt and our wrapper class into `generateTypedContentRequest` and extract a clean, tag-free vocabulary card, or null if no word was taught.

[28:24](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1704s) All right.

[28:25](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1705s) We covered a lot of things.

[28:26](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1706s) It is a reasonable thought to have at this moment, "Great, another thing that looks lovely on stage that I will never touch."

[28:33](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1713s) Not this one!

[28:34](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1714s) XR Language Teacher ships in the Google AI Edge Gallery app alongside other on-device AI experiences.

[28:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1720s) You can download it today, put it on your own phone, and point at any anything you want to learn how to say it in a different language today.

[28:48](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1728s) And if you want to know how it's built, this entire implementation is open source on GitHub.

[28:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1734s) And don't worry if not all of your users have audio or display glasses or supported flagship Android devices already.

[29:02](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1742s) Our code base also shows you examples of how this same experience can be adapted for a mobile-only experience.

[29:09](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1749s) And if you want to reach other platforms too, the Google AI Edge Gallery app code also shows you how to build this feature using the Google LiteRT runtime and MediaPipe Tasks APIs.

[29:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1759s) Or if you want to bring other open source on-device audio or language models, you can also see how to do that from the code.

### Adapting to other use cases with AI agents

[29:28](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1768s) So we open sourced XR Language Teacher to make your prototyping easier.

[29:32](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1772s) Point your agent of choice at our GitHub repo to turn this live, on-device, AI-powered experience into something else.

[29:40](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1780s) It could be a museum guide, maybe a cooking assistant looking at your counter, a repair walkthrough, accessibility narrator, a field inspection tool.

[29:50](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1790s) The model, the runtime, the open-source implementation, and the tooling are all available now so that you can get ready for the upcoming intelligent eyewear launch this fall.

[29:58](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1798s) We'd love to see what you make with it.

[30:02](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1802s) To wrap up, today, we showed powerful examples of what's possible with AI models running seamlessly on-device.

[30:09](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1809s) And here are a few ways to get started.

[30:12](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1812s) First, grab the resources on the screen.

[30:14](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1814s) Download the AI Edge Gallery app to try out the demo and check out the reference code on GitHub.

[30:19](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1819s) And scan the QR codes to access Android XR SDK, ML Kit APIs, and Google AI Edge offerings.

[30:27](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1827s) Second, start experimenting right on the phone today.

[30:32](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1832s) There are a lot of things that we covered that can already be built with a mobile first experience already.

[30:38](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1838s) And third, look at the mobile app you are already shipping and ask yourself, what core features would be better hands-free?

[30:45](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1845s) Adding a Projected Activity to your existing APK is all it takes to bring that experience to glasses, too.

[30:52](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1852s) Thank you so much for joining us today.

[30:54](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1854s) We can't wait to see what incredible audio-first experiences you build on Android and for intelligent eyewear.

[31:00](https://www.youtube.com/watch?v=tW3mg0iyi_w&t=1860s) Thank you.