# On-device intelligence: Optimized performance and quality using Android ML Kit APIs

- Speakers: David Chou (PM, AICore)
- Track: Intelligent Apps
- Video: https://www.youtube.com/watch?v=p-SZ1rWvErU
- Page: https://developer.android.com/events/dev-summit/talks/on-device-intelligence-optimized-performance-and-quality-using-android-ml-kit-apis

## About the talk

In this session, we'll explore different GenAI use cases using ML Kit APIs to understand when on-device GenAI is ideal for an app and also understand how to evaluate existing GenAI use cases to see if on-device models can satisfy their quality bar.

## Summary

- On-device AI offers major advantages in speed and privacy over cloud inference. Internal testing shows a 2x speedup compared to running models in the cloud.
- Gemini Nano memory usage dropped from 3.6 GB to 1.1 GB. Key optimizations include 2-bit quantization and a shared KV cache.
- Prefix caching and speculative decoding significantly accelerate inference. Reusing system prompts cuts processed tokens by 60%, while a smaller drafter model delivers an average 2x speedup.
- New on-device capabilities expand app interactions through speech translation and function calling. Adding thinking mode raises tool-calling accuracy on Gemini Nano 4 by 13 points out of 100.
- Multimodal embeddings combine with AppSearch to store and retrieve data through cosine similarity. Your app executes the search logic locally, protecting user privacy and data security.

## Resources

- [Prompt API Android documentation](https://developers.google.com/ml-kit/genai/prompt/android)

## Topics

on-device-ai, ml-kit, gemini-nano, aicore, function-calling, embeddings, speech-translation

## Transcript

### Introduction to on-device AI

**David Chou**

[00:05](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=5s) Hey.

[00:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=6s) Good Good afternoon, everyone.

[00:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=7s) My name is David Chou.

[00:09](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=9s) I'm a product lead on Android AICore.

[00:11](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=11s) Super excited to talk to you about our on-device AI capabilities.

[00:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=15s) Today, we'll talk through a few core areas.

[00:18](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=18s) The first is just thinking about why you should use on-device AI, especially given the acceleration in AI progress in the cloud.

[00:25](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=25s) Second is thinking about some of the improvements that we've been making over the last year to make these on-device AI models extremely potent.

[00:34](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=34s) And then the last thing that we're going to talk through is an end-to-end example that shows how you can leverage all these benefits using on-device AI.

[00:41](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=41s) So, first of all, why use on-device AI?

[00:45](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=45s) There are really 4 main reasons.

[00:47](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=47s) The first is speed, second is cost, third is availability, and fourth is privacy.

[00:53](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=53s) So, when it comes to speed, you can actually see that on-device AI inference is now getting to the point where it can be faster than running in the cloud simply because you don't have to pass information back and forth between the user's device and in the cloud.

[01:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=67s) So, on a lot of the use cases that we've been testing, we're actually seeing a 2x speedup when comparing on-device AI versus the cloud.

[01:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=75s) The second is thinking about cost.

[01:17](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=77s) And so, let's imagine that you were to run something a million inferences a day.

[01:21](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=81s) You could actually save hundreds of thousands of dollars if you're running on-device AI, specifically from the user's device versus running it in the cloud.

[01:31](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=91s) And the third is offline availability.

[01:33](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=93s) So, we sent out a survey earlier this year, and 62% of the developers that we were talking to were saying offline availability was a top three reason to use on-device AI.

[01:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=104s) And so whether your user is 35,000 feet in the air or whether they're in the Tokyo station sub-basement and don't have access to the internet, they can still leverage all these capabilities.

[01:55](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=115s) And finally, privacy was one of the biggest reasons that we saw when we surveyed developers.

[02:00](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=120s) So, 72% ranked it as a top three reason.

[02:03](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=123s) 40% Ranked it as number one.

[02:05](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=125s) So when it comes to your messages, your voice notes, photos, or even financial or medical information, users want to keep this private, and so running inference on-device is going to be really valuable to them.

[02:18](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=138s) So given all these benefits, right, we built ML Kit so that you can maintain and leverage all these capabilities.

[02:25](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=145s) We're trying to make this extremely simple.

[02:27](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=147s) So, if you have any complaints, please feel free to reach out to me.

[02:30](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=150s) I'll leave my contact info at the end of this talk.

[02:32](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=152s) We'd love to figure out how we can enable your use cases.

[02:35](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=155s) And so to begin with, we've enabled a bunch of capabilities for like speech and creative use cases.

[02:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=164s) So, like proofreading, rewriting, and also enabling flexible multimodal prompting.

[02:49](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=169s) And so, our goal is to try to make this as usable as possible.

[02:55](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=175s) And this is why a lot of our users have already started to adopt this, a lot of our developers.

[03:00](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=180s) We already have 9 third-party apps today that have 10 million downloads that are using these GenAI features in production.

[03:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=187s) And we're seeing a lot more scale up across both our first-party apps, as well as third-party apps, and 110% QoQ growth in app launches for AI production features.

[03:20](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=200s) And so, across messaging and creative and finance, we encourage you to just try these APIs that we have available.

[03:26](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=206s) We think the possibilities are endless, and you likely have better imagination than we do in many of these use cases, and so we'd love to partner together and figure out how we can work together.

[03:36](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=216s) So now like you understand the high-level benefits.

[03:40](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=220s) So you might be asking questions about like, okay, what is the reach actually like?

[03:43](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=223s) How is the performance on device?

[03:45](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=225s) When we talk about speed, how good is it actually?

[03:48](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=228s) Or like the quality of the model itself, right?

[03:51](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=231s) So we'll go through these one-by-one.

[03:53](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=233s) When it comes to reach, we're already available on 175 million AICore devices today.

[03:58](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=238s) We expect this to expand significantly over the coming year, bringing on hundreds of millions of other devices.

[04:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=246s) Part of what makes this possible is that we've built a very simple harness.

[04:09](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=249s) And so, ML Kit API is all you have to integrate into.

[04:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=252s) There's a lot of complexity that's built into AICore to manage the input and output safety signals, to manage downloading the model along with the weights, thinking about how to set it up in the runtime.

[04:23](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=263s) And so, the goal is to make ML Kit as easy to use so that you can just worry about how to deliver a great experience, and we can worry about the rest.

### Optimizing Gemini Nano performance and benchmarks

**David Chou**

[04:32](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=272s) We're also happy to announce that we've significantly reduced the RAM required to run Gemini Nano, which is the primary model that powers a lot of these GenAI APIs.

[04:40](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=280s) And so, we've reduced this from 3.6 GB RAM usage to 1.1.

[04:46](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=286s) So, there are really 4 core improvements that we've made in order to make this possible.

[04:51](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=291s) The first is using 2-bit quantization instead of 4-bit.

[04:55](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=295s) And so, half the bits just means half the bytes that you have to pipe through the model.

[04:59](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=299s) The second is using a shared KV cache.

[05:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=301s) So, we've significantly reduced this by allowing the first 20 layers of the model to share that in the 35 MB package, meaning that you can still access the 16K token context size within this very, very small package size.

[05:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=315s) The third is enabling lazy vision and audio encoders.

[05:18](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=318s) And so, for a lot of the text use cases that we're seeing, you actually don't need to load these to begin with.

[05:23](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=323s) And so that'll save you another half a gigabyte of RAM.

[05:26](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=326s) And then, finally, in order to improve the battery life, we've also enabled NPU clock scaling.

[05:32](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=332s) All this means is that the NPU can run as fast as it needs to save power while making sure it still delivers high quality on your use cases.

[05:42](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=342s) There are a few other improvements that we've made when it comes to speedup.

[05:45](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=345s) And so the first is enabling prefix caching.

[05:48](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=348s) You can see that in this example here, the system prompt is reused for multiple turns, right?

[05:52](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=352s) And so processing this becomes very expensive.

[05:55](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=355s) But with prefix caching, we're essentially saying you don't have to reprocess that system prompt again and again, right?

[06:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=361s) You can save that in the prefix cache, meaning that for this example, you're saving 60% of the tokens that you have to pass in, which also means a 60% speedup on whatever device you're on.

[06:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=372s) The second improvement that we've made is with speculative decoding.

[06:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=375s) And so in this example, right, you can see that Gemini Nano is generating one token per pass.

[06:20](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=380s) And this is because LLMs are autoregressive, which means that they have to keep looking at the previous tokens that were generated to figure out the next tokens.

[06:28](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=388s) So the way that we handled this was by building a drafter model that generates a bunch of tokens at a much faster speed.

[06:33](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=393s) This drafter model is substantially smaller than Gemini Nano, and so it can generate a bunch of tokens at once, pass that to Gemini to see which ones actually align with Gemini's Nano's Gemini Nano's guesses.

[06:46](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=406s) And so in this example, you can see, in the first pass, it generates six tokens, and the next one generates five, and then, by turn three, it's actually done.

[06:53](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=413s) And so what we've been seeing is that for a lot of the internal use cases we're looking at, this is leading to an on average 2x speedup.

[07:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=421s) You might get faster improvements depending on your hardware.

[07:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=426s) Okay.

[07:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=426s) So this is sort of why, you know, going back to the speed, you can actually see this dramatic speedup, right, between on device and cloud even on a 5G network.

[07:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=435s) You can see that simply just uploading that data to the cloud is going to take even just like one second, right, for 1,000 for 1,300 input tokens, and so even though the prefill and the decode in a cloud journey might be faster, the end-to-end journey is still substantially faster when you're running on device.

[07:36](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=456s) And then, when it comes to quality, happy to announce that this 2B model is actually performing against last year's 27B models.

[07:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=464s) While it's not as performance as you know, these frontier models that we have today, what we're showing here is that there's substantial progress when it comes to AI on these on-device models, right?

[07:56](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=476s) And so you can see that for MMLU-Pro, which is a standard knowledge benchmark, the 2B model is almost as good as the 27B parameter model from last year.

[08:05](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=485s) And then when it comes to graduate science, reasoning, and agentic tool use benchmarks, this 2B model is actually better, even though it's less than a 10th of the size of last year's model.

[08:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=495s) And so we expect these trends to continue.

[08:17](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=497s) We're continuing to work closely with the GDM team to make sure that we can keep making these improvements to the model so that you can get the best value out of these.

### Overview of new Gemini API capabilities

**David Chou**

[08:27](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=507s) Okay.

[08:28](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=508s) So, I've just sort of thrown a lot of benefits and new improvements at you, and maybe you're asking, "What can we actually do with these models?"

[08:36](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=516s) And so there are all these capability improvements that we've been making in the API as well.

[08:41](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=521s) So with system instructions, you can set a persona.

[08:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=524s) With multi-image input, you can upload a bunch of images at once.

[08:47](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=527s) We've also enabled thinking mode earlier this year, so that you can spend more time with reasoning problems as well as tool calling.

[08:54](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=534s) And finally, with structured output, we've also made it so that you can annotate a Kotlin class and get back an object that matches your specific schema.

[09:04](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=544s) Beyond this, we also try to ask developers for new improvements that they were like to they would like to make.

[09:09](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=549s) And so 91% of the developers we talked to when surveyed said that they would launch an experience if one of these capabilities was available.

[09:16](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=556s) And so, earlier this week, we actually launched these four capabilities: speech translation, embeddings, text-to-speech, and function calling.

[09:25](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=565s) With speech translation, you can transcribe audio from over 20 languages.

[09:29](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=569s) With embeddings, you're able to search your app's content by semantic meaning, rather than just using keyword search.

[09:37](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=577s) With text-to-speech, you can now provide human-like voices for your experiences, so you no longer have to have robotic voices.

[09:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=584s) And finally, with function calling, you can actually let your app take actions on behalf of your users.

[09:50](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=590s) So we've already been partnering with a few developers to see how they could build in-app agents in their experiences, and we hope to launch later next year.

### End-to-end scam detection walkthrough

**David Chou**

[10:00](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=600s) So given all these capabilities, let's actually walk through an end-to-end example where we're building with all these capabilities.

[10:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=607s) So, let's imagine there's a American photographer and they're traveling in Japan.

[10:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=612s) And, as you can see from the photo, they clearly don't have enough cameras or lenses.

[10:17](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=617s) And so, they order a new lens and send it to their hotel.

[10:22](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=622s) But there's a catch: As they're going about their day, they get a voicemail and they're told they need to pay a 6800 yen clearance fee, in order to get their package by the end of the day.

[10:34](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=634s) They've also received a text message that has a photo of the held parcel.

[10:38](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=638s) So, in this instance, the user is interested in having some kind of scam detection protection, right?

[10:45](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=645s) They want to understand how, they can identify voicemails as legitimate or fraud.

[10:50](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=650s) And so, we want to keep all the information private, we want to make sure that the information is grounded, but also that it's explainable.

[10:57](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=657s) So we're going to make sure that they can understand the Japanese text, and also, be able to use whatever language capabilities, are available in case they have accessibility needs.

[11:09](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=669s) So, they're going to be four steps here, right?

[11:11](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=671s) The first step is just processing that voicemail in the text, so it's going to translate that voicemail.

[11:17](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=677s) We're going to use the Speech Translation API for that.

[11:20](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=680s) And then they're going to be able to ask a question if they want, so we're going to use the Speech Recognition API.

[11:26](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=686s) And then after that, we want to define functions that are able to retrieve that information that is necessary for the user to get what they need.

[11:33](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=693s) So, we're going to use Prompt API and function calling.

[11:35](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=695s) Now, this will run in the background without anything visible to the user.

[11:40](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=700s) And then third, we'll actually figure out how to store and retrieve that information by using the embedding API and AppSearch.

[11:46](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=706s) This will also run in the background.

[11:48](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=708s) And then, finally, we'll make a verdict based on the information that we're collecting, using Prompt API, structured output, and speech synthesis.

[11:57](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=717s) Okay.

[11:58](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=718s) So, let's begin.

[11:59](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=719s) We're going to use speech translation and speech recognition.

[12:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=721s) So, the first step is actually just taking that voicemail and the text, turning that into useful information, right?

[12:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=726s) And so, the user has received this message, "Hey, you need to pay this 6,800-yen fee.

[12:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=732s) Duties, maybe not been prepaid, and so you need to pay it."

[12:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=735s) And so, they want to be able to have that translation, right?

[12:19](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=739s) And so, one of the things that, we'd like to point out here is both the translation as well as the user's question about the information all goes to Gemini Nano 4 using the same audio encoder, because the audio encoder's been trained on over 100 languages.

[12:31](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=751s) And so you can get that translation, as well as the ASR for the text of the user's question.

[12:38](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=758s) And so, for translation, all we have to do is set up a source and a destination language.

```kotlin
val translator = SpeechTranslation.getClient(
    SpeechTranslatorOptions.Builder(
        Locale.JAPANESE, Locale.US
    ).build()
)
```

[12:43](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=763s) In this case, we're going to use Japanese as well as American English.

[12:47](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=767s) We're going to set up a Kotlin flow that passes the bytes directly, into the speech translator.

```kotlin
val translator = SpeechTranslation.getClient(
    SpeechTranslatorOptions.Builder(
        Locale.JAPANESE, Locale.US
    ).build()
)

val audioFlow = flow {
    emit(TranslationAudio(Locale.JAPANESE, courierVoicemailBytes))
}

val request = SpeechTranslatorRequest.Builder(audioFlow).build()
translator.startTranslation(request).collect { response ->
    if (response is SpeechTranslatorResponse.Text) {
        translatedCourierClaim = response.translatedText
    }
}
```

[12:52](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=772s) And then, we can collect, the response using the translated text.

[12:56](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=776s) Okay.

[12:56](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=776s) Great.

[12:57](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=777s) So, now we understand what the voicemail is actually saying, right?

[13:00](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=780s) There's a camera lens parcel.

[13:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=781s) It's held at Customs.

[13:02](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=782s) They have to pay that 6,800 yen fee.

[13:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=786s) And now the user wants to be able to ask a question about it.

[13:08](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=788s) Okay.

[13:08](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=788s) So, is this customs hold actually real?

[13:13](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=793s) In this case, we're going to use the ASR API.

```kotlin
val recognizer = SpeechRecognition.getClient(
    speechRecognizerOptions {
        locale = Locale.US
        preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
    }
)
```

[13:16](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=796s) What we're doing is still setting American English as locale.

[13:19](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=799s) And then, we can set it to advanced mode, which leverages the audio encoder.

[13:24](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=804s) And then, we can pass that information in, as well, collect the final response, and then we're going to turn that, into information that we're going to pass to Gemini Nano.

```kotlin
val recognizer = SpeechRecognition.getClient(
    speechRecognizerOptions {
        locale = Locale.US
        preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
    }
)

val request = speechRecognizerRequest {
    audioSource = AudioSource.fromMic()
    recognitionOptions { enableWordTiming = true } // optional
}

recognizer.startRecognition(request).collect { response ->
    when (response) {
        is SpeechRecognizerResponse.PartialTextResponse ->
            liveCaption = response.text
        is SpeechRecognizerResponse.FinalTextResponse ->
            planAndCallVaultTool(translatedCourierClaim, response.text)
    }
}
```

[13:32](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=812s) Okay.

[13:33](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=813s) So, now we've finished the first part of intake.

[13:36](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=816s) As you can see, it didn't take that many lines of code.

### Triaging inputs using Prompt API and function calling

**David Chou**

[13:40](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=820s) So the second part is triage, which is thinking about how to leverage Prompt API and function calling to figure out what kind of information is needed.

[13:48](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=828s) Okay.

[13:49](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=829s) So, to build the architecture of this, we're going to set up some system instructions first.

[13:55](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=835s) And then we're also going to have the voicemail, the text, and photo, as well as the user's question, be passed as input into Gemini Nano.

[14:03](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=843s) And then we're going to define functions that actually retrieve that information that we need.

```kotlin
retrieveOrderAndDeliveryPhotos(
  searchQuery = "camera lens customs",
  claimedFeeUsd = 45,
  comparePackagePhotos = true
)
```

[14:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=847s) So, to start, we're going to set up the system instructions.

```kotlin
// Keep SystemInstruction static across Turn 1 & Turn 2:
val securitySystemInstruction = SystemInstruction(
    """
    You are an on-device parcel & phishing security auditor.
    1. Cross-check packaging, carrier labels and tracking across all images.
    2. Verify customs fee claims against local DDP (Delivered Duty Paid) order records.
    3. Flag any payment demand when the order record shows duties prepaid.
    """.trimIndent()
)
```

[14:10](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=850s) Right here you can see a very simple example of what it would look like, to set up a security auditor, what it says is we're going to cross-check packaging and labels, look at custom fees, as well as flag any payment demands that are necessary.

[14:24](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=864s) You might find that, prompt optimization will be really helpful here in getting you significantly better quality.

[14:33](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=873s) And then once we have that system instruction set, we can take all the content that we had from the prior step, the translation of both the voicemail and the text, as well as the user's question and pass that all in into this `TextPart` field, along with the system instructions in order to get that information.

```kotlin
// Pull in the Speech Translation (AST) & Speech Recognition (ASR) outputs:
val turn1Prompt = """
    Translated voicemail: $translatedCourierClaim
    Incoming text message: $callerText
    User's spoken question: $userQuestion
    """.trimIndent()

val turn1Request = generateContentRequest(
    securitySystemInstruction,
    TextPart(turn1Prompt)
) { /* enableThinking & setTools */ }
```

[14:51](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=891s) And then we want to define a function that we can use.

[14:54](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=894s) So, for this particular example, we're excited to announce we've actually integrated with app function.

[14:58](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=898s) So, if any of you have familiarity with that, you'll probably roughly understand how this works already.

[15:04](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=904s) So here we set up an `AppFunctionService` entry point, and then we define an app function.

```kotlin
// Registered in AndroidManifest.xml
@AppFunctionServiceEntryPoint("ScamInvestigationService", "scam_tools")
abstract class ScamInvestigationTools : AppFunctionService() {

}
```

[15:09](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=909s) You can see that it's exactly the same as you would if you are setting up app functions for a system agent.

[15:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=915s) You set in the parameters of the `searchQuery`, the claim fee, and also whether or not we should be comparing package photos just as some example parameters, right?

```kotlin
// Registered in AndroidManifest.xml
@AppFunctionServiceEntryPoint("ScamInvestigationService", "scam_tools")
abstract class ScamInvestigationTools : AppFunctionService() {
    /**
     * Searches local orders and order images in AppSearch.
     * @param searchQuery Semantic query to match the order
     * @param claimedFeeUsd Customs fee claimed in USD
     * @param comparePackagePhotos Whether to fetch order images to compare
     */
    @AppFunctionDeclaration(isDescribedByKDoc = true)
    suspend fun retrieveOrderAndDeliveryPhotos(
        searchQuery: String,
        claimedFeeUsd: Int,
        comparePackagePhotos: Boolean
    ): VaultMatch = withContext(Dispatchers.IO) { executeVaultSearch(searchQuery) }
}
```

[15:25](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=925s) You might define different ones.

[15:27](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=927s) And then, finally, for that function we have an `executeVaultSearch` function that we're going to define later.

[15:35](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=935s) One of the things I do want to call out here is that we have thinking mode available.

```kotlin
val vaultTools = AppFunctionTools.getClient().getAppFunctions()
val turn1Request = generateContentRequest(securitySystemInstruction, TextPart(turn1Prompt)) {

    enableThinking = true
    setTools(vaultTools)

}
val call = generativeModel.generateContent(turn1Request)
    .candidates.first().functionCalls.first()
```

[15:38](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=938s) I strongly recommend you use it when running tool calling.

[15:43](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=943s) The reason for that is because we've been doing a bunch of testing that shows that there's a 13-point improvement in accuracy out of 100 when running tool calling on Gemini Nano 4.

[15:53](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=953s) And so, even on some cases it's actually substantially higher, and so this might give you much better results.

[16:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=961s) And if you want to learn more about function calling, please attend my colleagues Jolanda and Victoria's session, which will be happening later.

[16:08](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=968s) This is on agentic capabilities in your app.

[16:10](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=970s) I think Victoria might be sitting somewhere in the crowd, so yeah, feel free to say hi to her as well.

### Retrieving order data with embeddings and AppSearch

**David Chou**

[16:16](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=976s) Okay, cool.

[16:17](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=977s) So, now we've finished the first two steps.

[16:19](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=979s) Let's actually talk about how retrieval works.

[16:21](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=981s) So, we're going to use embeddings and AppSearch.

[16:25](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=985s) So, maybe taking a few steps back, right, in the user's journey, at the point where they ordered the lens, what we'd want to do is actually index that information so it's accessible to the user as well as the developer, right?

[16:39](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=999s) So, in this we have information details about the order package number.

[16:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1004s) It has a tracking number as well.

[16:46](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1006s) It has some details about where it's sent.

[16:48](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1008s) And you can see that here it's already mentioning that duties were prepaid.

[16:52](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1012s) And so, what we're going to do is generate a multimodal embedding in order to index that information along with any other information that might be important for us.

[17:03](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1023s) So, here's what it looks like.

```kotlin
val embedModel = EmbeddingGeneration.getEmbeddingModels().first()
val docVector = embedModel.generateEmbedding(
    EmbeddingRequest(
        Content(listOf(
            ImagePart(orderEmailLensBmp),   // the lens photo in the email
            TextPart(orderEmailBody)        // the order details
        ))
    )
).embedding.vector
```

[17:05](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1025s) We take the order email.

[17:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1026s) Again, very simple, we just pass one image and one text, but you might have more content that you want to index here.

[17:13](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1033s) And then, we're going to generate the embedding vector.

[17:16](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1036s) So, after we do that, we're going to use AppSearch to create a document and then set whatever properties we think are necessary to generate that specific document.

```kotlin
val embedModel = EmbeddingGeneration.getEmbeddingModels().first()
val docVector = embedModel.generateEmbedding(
    EmbeddingRequest(
        Content(listOf(
            ImagePart(orderEmailLensBmp),   // the lens photo in the email
            TextPart(orderEmailBody)        // the order details
        ))
    )
).embedding.vector

val doc = GenericDocument.Builder<GenericDocument.Builder<*>>(
    "orders", "order_jp_904", "OrderRecord"
)
    .setPropertyString("summary", "Shipped DDP, duties prepaid ($0 owed). In transit.")
    .setPropertyString("photoUris", "order_email_lens.png")
    .setPropertyEmbedding(
        "embedding",
        EmbeddingVector(docVector, embedModel.getModelSignature())
    ).build()
```

[17:27](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1047s) And then, one thing to note here is that the reason we want to do multimodal embeddings, right, is because there might not be any shared words here, but because the images and the text are about similar content, this helps us to actually get the semantic similarity.

[17:42](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1062s) So, we can use cosine similarity to find items that don't have any shared words in common.

[17:48](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1068s) Okay, so this is the `executeVaultSearch` function that we had talked about earlier.

```kotlin
suspend fun executeVaultSearch(searchQuery: String): VaultMatch {
    val qVec = embedModel.generateEmbedding(
        EmbeddingRequest(
            content = Content(listOf(TextPart(searchQuery)))
        )
    ).embedding.vector

}
```

[17:52](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1072s) This is what's being called by the app function that we had defined.

[17:56](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1076s) So, in this example, we're passing in the search query that has been generated by Gemini Nano.

[18:03](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1083s) And then, we're going to use a ranking strategy, cosine similarity.

```kotlin
suspend fun executeVaultSearch(searchQuery: String): VaultMatch {
    val qVec = embedModel.generateEmbedding(
        EmbeddingRequest(
            content = Content(listOf(TextPart(searchQuery)))
        )
    ).embedding.vector

    val spec = SearchSpec.Builder()
        .addEmbeddingParameters(EmbeddingVector(qVec, embedModel.getModelSignature()))
        .setDefaultEmbeddingSearchMetricType(SearchSpec.EMBEDDING_SEARCH_METRIC_TYPE_COSINE)
        .setRankingStrategy("sum(this.matchedSemanticScores(getEmbeddingParameter(0)))")
        .setListFilterQueryLanguageEnabled(true).build()

}
```

[18:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1087s) And then, `this.matchedSemanticScores` just helps us to get that information.

[18:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1092s) And then, finally, we're going to specify a threshold of 0.68 to determine what the minimum cosine similarity is in order for us to fetch information.

```kotlin
suspend fun executeVaultSearch(searchQuery: String): VaultMatch {
    val qVec = embedModel.generateEmbedding(
        EmbeddingRequest(
            content = Content(listOf(TextPart(searchQuery)))
        )
    ).embedding.vector

    val spec = SearchSpec.Builder()
        .addEmbeddingParameters(EmbeddingVector(qVec, embedModel.getModelSignature()))
        .setDefaultEmbeddingSearchMetricType(SearchSpec.EMBEDDING_SEARCH_METRIC_TYPE_COSINE)
        .setRankingStrategy("sum(this.matchedSemanticScores(getEmbeddingParameter(0)))")
        .setListFilterQueryLanguageEnabled(true).build()

val results = appSearchSession.search(
        "semanticSearch(getEmbeddingParameter(0), 0.68)", spec
    )
    // Cleanly close the cursor.
    return results.use { it.nextPageAsync.await().first().genericDocument.toVaultMatch() }
}
```

[18:20](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1100s) So, this is something that you might want to play with as you run through your own use cases just to see what actually works for you.

[18:28](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1108s) And then, one thing that we want to call out here, too, is when you think about the security of this model, Gemini Nano is actually not the one that's running any of the code, right?

[18:37](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1117s) So, Gemini Nano is just asking a question, but your app defines the code, and then it's actually the one that's running the code itself, retrieving that information, and then determining what to pass Gemini Nano.

[18:49](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1129s) So, that gives you more control over what you want to expose to Gemini Nano and also helps to ensure more privacy and security for your users.

### Comparing order details and defining audit verdicts

**David Chou**

[18:59](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1139s) Okay, and now this is the final step.

[19:00](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1140s) So, we have all the information that we need.

[19:03](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1143s) We're going to use Prompt API, structured output, and speech synthesis to make the comparisons and then actually return that information back to the user in an understandable way.

[19:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1155s) Okay, so fundamentally, right, what we want to know is whether this parcel is actually the same as the information from the user, which is the real order.

[19:24](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1164s) And so, in order to do that, we are also going to generate embedding, right, on that information from the voicemail as well as the text, turn that into the same embedding vector that we did earlier.

```kotlin
val callerVector = embedModel.generateEmbedding(
    EmbeddingRequest(
        Content(listOf(

            ImagePart(callerParcelPhotoBmp),   // brown box w/ shipping label
            TextPart(callerText)               // "Clearance fee ¥6,800 due today."

        ))
    )
).embedding.vector
val matchScore = cosineSimilarity(callerVector, vaultMatch.embedding)
```

[19:36](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1176s) And then, we're going to use `matchScore`, by using cosine similarity, we can make a comparison between that match that we had retrieved from the vault as well as the information from that caller.

[19:50](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1190s) And so, what you can see here is, even if the images might not match up, right, the text is still really helpful.

[19:55](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1195s) So, this is where multimodal embeddings are actually quite valuable.

[19:59](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1199s) You can put in disparate parts of information and then get to a shared semantic meaning, right?

[20:04](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1204s) So, the multimodal embedding makes it so that you can map it all onto the same vector space.

[20:10](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1210s) Again, we're going to use thinking mode here to improve our thinking.

```kotlin
val forensicRequest = generateContentRequest(
    securitySystemInstruction,
    content {
        image(callerParcelPhotoBmp)
        image(orderEmailLensBmp)
        text(forensicAuditPrompt)
    }
) {

    // Enable Gemini Nano 4 reasoning before it answers
    enableThinking = true

}
```

[20:15](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1215s) And then we're also going to define a output class, right?

```kotlin
@Generable(description = "Details of a visual contradiction found during an audit")
data class VisualContradiction(
  @Guide(
    description = "The type of contradiction",
    enumValues = ["PACKAGING_MISMATCH", "STATUS_CONFLICT", "DUTY_FRAUD"],
  )
  val contradictionType: String,

  @Guide(description = "Detailed explanation of the visual contradiction")
  val explanation: String,
)
```

[20:19](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1219s) So all we're doing here is setting up a `VisualContradiction` and setting up a few different enums of the different kinds of possible contradictions we could have, right?

[20:28](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1228s) Whether it's `DUTY_FRAUD`, or a `PACKAGING_MISMATCH`, or a `STATUS_CONFLICT`.

[20:32](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1232s) And then we're going to have an explanation here.

[20:35](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1235s) And then we're going to pass all that information into an audit verdict, right?

```kotlin
@Generable(description = "Verdict and details of a phishing audit")
data class PhishingAuditVerdict(
  @Guide(
    description = "The audit verdict",
    enumValues = ["CONFIRMED_PHISHING_SCAM", "LEGITIMATE_HOLD"],
  )
  val verdict: String,

  @Guide(description = "The fraud risk score from 0 to 100", minimum = 0.0, maximum = 100.0)
  val fraudRiskScore: Int,

  @Guide(description = "Estimated amount of money saved in USD", minimum = 0.0)
  val moneySavedUsd: Int,

  @Guide(description = "List of visual contradictions identified during audit", minItems = 0)
  val contradictions: List<VisualContradiction>,

  // @Guide(description = "Summary of the warning to be spoken to the user")
  // `@Guide` annotations are optional.
  val spokenWarningSummary: String,
)
```

[20:39](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1239s) So you can see here that you know, it's very simple.

[20:43](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1243s) All we're saying is this a scam or not?

[20:46](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1246s) We maybe have a `fraudRiskScore` where you can set both a minimum and a maximum value from 0 to 100.

[20:52](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1252s) You can even let the user know how much money you've saved them if you want to keep a tracker of that over time.

[20:58](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1258s) And then you can also list whatever visual contradictions we just defined.

[21:02](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1262s) One of the other things to note here is that the `@Guide` annotation is actually optional.

[21:07](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1267s) You don't need to necessarily set it and provide that sort of description.

[21:11](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1271s) The reason it's helpful is because you might not be able to explain what's going on just by the variable definition.

[21:18](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1278s) And so having that description can be helpful for Gemini Nano in figuring out what specific functions to call and how to set it up such that it's inputting the right information, right, in the right fields.

[21:31](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1291s) And so yeah, after you set that up, it's very simple.

[21:35](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1295s) You just run three lines of code here.

```kotlin
val typedRequest = generateTypedContentRequest(forensicRequest, PhishingAuditVerdict::class)
val response = generativeModel.generateContent(typedRequest)
val verdict: PhishingAuditVerdict? = response.candidates.first().response
```

[21:37](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1297s) `generateTypedContentRequest`, you generate that content, and then you can retrieve that response.

[21:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1304s) And then here's something that shows what it would look like, right?

```json
{
  "verdict": "CONFIRMED_PHISHING_SCAM",
  "fraudRiskScore": 98,
  "moneySavedUsd": 45,
  "contradictions": [
    { "contradictionType": "DUTY_FRAUD", "explanation": "Fee demanded, but duties were prepaid." },
    { "contradictionType": "PACKAGING_MISMATCH", "explanation": "Carton has no carrier label or tracking." },
  ],
  "spokenWarningSummary": "Scam blocked. Do not pay the 45 dollar fee. Your lens shipped duties prepaid."
}
```

[21:47](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1307s) And so we confirm that this is a scam, we get a very high fraud risk score, it explains these contradictions that are evident in the data that you've retrieved comparing that to the caller information.

[21:58](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1318s) We've also saved this user a certain amount of money, so we can keep a tracker there.

[22:03](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1323s) Yeah.

[22:04](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1324s) And so that sort of encapsulates all the information that we need.

[22:08](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1328s) And then the final step would be to show this to the user in a way that is accessible.

[22:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1332s) So if they're on the go for example, or maybe their vision impaired, right, you might want to actually read that information aloud rather than having them have to read that text by hand.

[22:22](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1342s) And so we're going to implement the text-to-speech capability to show what that would look like.

```kotlin
val synthesizer = SpeechSynthesis.getClient(SpeechSynthesizerOptions.builder(Locale.US).build())
try {
    val voice = synthesizer.getSupportedVoices(Locale.US).firstOrNull()
    if (voice != null) {
        val ttsRequest = SpeechSynthesizerRequest(
            input = SpeechSynthesizerRequest.Input.Text(verdict.spokenWarningSummary, voice),
            audioAttributes = AudioAttributes.Builder()
                .setUsage(AudioAttributes.USAGE_ASSISTANT)
                .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
                .build(),
        )

        /* more code here */

    }
} finally {
    synthesizer.close() // safe here: the flow completes after DONE, STOP, or Error
}
```

[22:28](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1348s) So again, super simple, we specify a locale and a language as well, and then we pass in that verdict that we had just generated using `@Generable` and `@Guide`.

[22:38](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1358s) And then after we have that, right, it's almost instantaneous.

```kotlin
val synthesizer = SpeechSynthesis.getClient(SpeechSynthesizerOptions.builder(Locale.US).build())
try {
    val voice = synthesizer.getSupportedVoices(Locale.US).firstOrNull()
    if (voice != null) {
        val ttsRequest = SpeechSynthesizerRequest(
            input = SpeechSynthesizerRequest.Input.Text(verdict.spokenWarningSummary, voice),
            audioAttributes = AudioAttributes.Builder()
                .setUsage(AudioAttributes.USAGE_ASSISTANT)
                .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
                .build(),
        )
        synthesizer.synthesize(ttsRequest).collect { response ->
            when (response) {
                is SpeechSynthesizerResponse.StatusUpdate ->
                                            /* playback finished */
                    if (response.status == SpeechSynthesizerResponse.Status.DONE) { }

                /* response.errorCode is a TextToSpeech.ERROR_* code */
 is SpeechSynthesizerResponse.Error -> { }
                else -> {}
            }
        }
    }
} finally {
    synthesizer.close() // safe here: the flow completes after DONE, STOP, or Error
}
```

[22:42](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1362s) It'll take less than 120 ms on most devices in order to generate that response.

[22:47](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1367s) We'll clear whatever memory is necessary.

[22:49](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1369s) And then if needed, we can also set a fallback, right, using platform APIs.

```kotlin
val synthesizer = SpeechSynthesis.getClient(SpeechSynthesizerOptions.builder(Locale.US).build())
try {
    val voice = synthesizer.getSupportedVoices(Locale.US).firstOrNull()
    if (voice != null) {
        val ttsRequest = SpeechSynthesizerRequest(
            input = SpeechSynthesizerRequest.Input.Text(verdict.spokenWarningSummary, voice),
            audioAttributes = AudioAttributes.Builder()
                .setUsage(AudioAttributes.USAGE_ASSISTANT)
                .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
                .build(),
        )
        synthesizer.synthesize(ttsRequest).collect { response ->
            when (response) {
                is SpeechSynthesizerResponse.StatusUpdate ->
                                            /* playback finished */
                    if (response.status == SpeechSynthesizerResponse.Status.DONE) { }

                /* response.errorCode is a TextToSpeech.ERROR_* code */
 is SpeechSynthesizerResponse.Error -> {
     /* handle fallback to platform API */
 }
                else -> {}
            }
        }
    }
// more code here ...
```

[22:53](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1373s) This would still have the robotic voice, but it at least gives the user something to interact with in case they can't read.

### Recap and closing remarks

**David Chou**

[23:00](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1380s) Okay.

[23:01](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1381s) So that was a lot, but to recap what we did, right, we leveraged all these different APIs across the intake, triage, retrieve, and audit processes.

[23:10](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1390s) You know, using speech recognition and translation APIs, prompt API and function calling, embeddings, and AppSearch, as well as structured output and speech synthesis.

[23:19](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1399s) And so this is just an example of how you could leverage all of these capabilities in a coherent way and run this completely on device.

[23:25](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1405s) And then going back to what we talked about in the beginning, right, there are substantial benefits to running this, right?

[23:31](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1411s) If you were to actually run this on your device today, you would see that this is substantially faster on device in most hardware.

[23:38](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1418s) It doesn't cost you anything, right, and you're potentially saving hundreds of thousands of dollars if you're running this all the time.

[23:44](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1424s) And then this is available, right, regardless of whether the user is out in mountains or in a basement and doesn't have access to the internet.

[23:51](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1431s) And it's fully private because you don't have to send any information to the cloud.

[23:56](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1436s) And so given all these capabilities, please feel free to look at the docs, take a look at the QR code, or go to the link.

[24:02](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1442s) These are all the capabilities that we've launched over the last six months roughly.

[24:06](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1446s) Thank you all so much for taking a time to attend this talk.

[24:09](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1449s) Really appreciate you spending time with us.

[24:12](https://www.youtube.com/watch?v=p-SZ1rWvErU&t=1452s) So yeah, thank you so much.