# Balancing ultra-low latency and cloud economics in virtual voice agents - Speakers: Devin Reimer (CEO) - Track: Android XR - Video: https://www.youtube.com/watch?v=FQt2V0VafaE - Page: https://developer.android.com/events/dev-summit/talks/balancing-ultra-low-latency-and-cloud-economics-in-virtual-voice-agents ## About the talk Bringing conversational AI into immersive experiences opens up huge possibilities, yet there’s a constant friction between low response latency and sustainable cloud costs. In this session, Devin Reimer (Founder & CEO of AstroBeam) breaks down how the team behind Stellar Cafe achieved industry-leading voice interaction response times (1.15s–1.4s) while keeping the business model commercially sustainable. Explore the hybrid architecture behind these low latency pipelines, focusing on intelligent orchestration between on-device models and optimized cloud inference to cut expensive API calls without degrading dialogue. You’ll walk away with a practical blueprint covering Time-to-First-Token optimization across Speech-to-Text and streaming Text-to-Speech pipelines, parallel safety guardrails that bypass latency overhead, and unit economics per active user session. ## Summary - End-to-end latency around 1.5 seconds lets conversational voice agents feel comfortable to users without slowness complaints. Consistency matters even more than peak speed so conversations never feel broken. - Parallelizing speech pipelines cuts response times and stays 3 to 5 times cheaper than speech-to-speech models. Running safety checks concurrently alongside initial text-to-speech chunks adds zero latency. - Agent intelligence demands low transcription error rates and expressive speech. Crucially, models must also remain easy to build with so developers avoid brittle engineering overhead. - Cloud economics must be modeled up front by comparing per-user server costs against expected revenue. Stellar Cafe achieved viable margins at about $1.20 per hour per user with flat pricing. - Open-weight models avoid the frequent sunsets and rising fees typical of closed models. Self-hosting them provides predictable costs and protects production schedules from forced migrations. ## Resources - [Devin Reimer on Bluesky](https://devinreimer.bsky.social) - [AstroBeam website](https://astrobeam.com) - [Devin Reimer on LinkedIn](https://linkedin.com/in/devinreimer) ## Topics voice-agents, speech-to-text, text-to-speech, large-language-models, low-latency, cloud-economics, game-development ## Transcript ### Introduction to voice agents and core challenges **Devin Reimer** [00:05](https://www.youtube.com/watch?v=FQt2V0VafaE&t=5s) Welcome to Balancing Ultra-Low Latency and Cloud Economics in virtual voice agents. [00:13](https://www.youtube.com/watch?v=FQt2V0VafaE&t=13s) So, I'm Devin Reimer. [00:14](https://www.youtube.com/watch?v=FQt2V0VafaE&t=14s) I'm the founder and CEO of AstroBeam. [00:16](https://www.youtube.com/watch?v=FQt2V0VafaE&t=16s) Previously, I was the CEO and co-founder of Owlchemy Labs, probably most known for Job Simulator. [00:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=23s) So, at AstroBeam, we are focused on building real-time AI in games. [00:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=29s) Specifically, we're really focused around voice as an input. [00:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=34s) We've released two titles so far, Stellar Cafe, which is VR and PC. [00:39](https://www.youtube.com/watch?v=FQt2V0VafaE&t=39s) It's a game where you talk to robots in a space station cafe and help solve their problems. [00:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=44s) It's 100% voice driven. [00:45](https://www.youtube.com/watch?v=FQt2V0VafaE&t=45s) You play the entire thing including menus and navigation, only with your voice. [00:51](https://www.youtube.com/watch?v=FQt2V0VafaE&t=51s) Our next game, which we just announced today, Robo Detective, is built for Android XR and specifically XREAL's Project Aura. [00:59](https://www.youtube.com/watch?v=FQt2V0VafaE&t=59s) In this game, you play a detective that works with a robot partner to help interview robots to solve robo crimes, and once again, 100% voice-driven. [01:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=70s) So, voice agents are really tough, and in games, they're actually tougher. [01:16](https://www.youtube.com/watch?v=FQt2V0VafaE&t=76s) The reason is that games are very low-cost things, and a voice agent is now a new cost to add to that pipeline. [01:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=85s) Additionally, we're not competing against other voice agents, we're competing against the quality of every game ever made, right? [01:33](https://www.youtube.com/watch?v=FQt2V0VafaE&t=93s) So, we have to come in at a very high quality. [01:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=97s) That's not to say that there isn't specific voice agent use cases where there are unique challenges, or where the stakes are much higher, but games are definitely very technically challenging in this space. [01:48](https://www.youtube.com/watch?v=FQt2V0VafaE&t=108s) But, also because they're games, this leaves a lot of room for experimentation of how to solve some of these hard problems. [01:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=116s) So, I believe that if voice agents can work within games, they can work almost anywhere. [02:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=121s) And so, the applica-- the things that I'm talking about in this talk apply not just to games, but any voice-first application that you'd want to build. [02:11](https://www.youtube.com/watch?v=FQt2V0VafaE&t=131s) So, when you think about voice agents, I think there's kind of three core challenges. [02:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=135s) So, we have latency, intelligence, and cost. [02:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=140s) So, one thing I do want to make clear, though, is that when I'm talking about voice agents, I'm talking about voice agents as a conversation, not as a prompt. [02:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=149s) I'm not talking about a microphone wrapper that sends something to an LLM and then waits for a response. [02:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=154s) I'm talking about a real-world, real-time back and forth, and that is actually what becomes quite difficult to build. [02:43](https://www.youtube.com/watch?v=FQt2V0VafaE&t=163s) So, let's start with latency. [02:46](https://www.youtube.com/watch?v=FQt2V0VafaE&t=166s) How quick does it need to be? [02:52](https://www.youtube.com/watch?v=FQt2V0VafaE&t=172s) Turns out it needs to be fast, very fast, but depending on what you've read, it might not be as fast as you think, or it might be much faster than you think. [03:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=181s) So, let's talk about some numbers that we have kind of discovered in a lot of our testing. [03:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=186s) So, when I'm talking about end-to-end latency, what I'm talking about is time from last user utterance, the last sound the user made, to the time that they hear the audio coming back to them. [03:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=197s) What we've kind of discovered in our testing is about 1.5 seconds is the point in which when we are getting play test feedback from users, they start not mentioning speed or slowness or any of that. [03:31](https://www.youtube.com/watch?v=FQt2V0VafaE&t=211s) That seems to be the point where people start feeling really comfortable with it. [03:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=215s) As soon as we get much slower than that, then people often bring it up, "This is great, but", and so that seems to be a bit of a sweet spot. [03:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=224s) Then as you get even faster, we discovered at about 1.2 seconds, some more more super natural like back and forth starts happening. [03:53](https://www.youtube.com/watch?v=FQt2V0VafaE&t=233s) So, like I say something, in this case robot, will say something really quick and you kind of go back and forth. [03:58](https://www.youtube.com/watch?v=FQt2V0VafaE&t=238s) There seems to be a bit of a sweet spot there that's unlocked. [04:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=241s) That said, 1.2 seconds is quite difficult to actually do within a stack. [04:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=246s) And then, another interesting finding is we did a lot of testing, and we ended up getting some scenarios to get sub-one second. [04:14](https://www.youtube.com/watch?v=FQt2V0VafaE&t=254s) And so, you might be like, "Oh, that's going to be awesome!" [04:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=257s) And sometimes it was, and sometimes it felt terrible, very uncanny. [04:22](https://www.youtube.com/watch?v=FQt2V0VafaE&t=262s) So, you might end up hearing people talk about 800 milliseconds response time being the right number, or 500. [04:28](https://www.youtube.com/watch?v=FQt2V0VafaE&t=268s) I even read a bunch of articles about 300 cuz that's really where human perception lands. [04:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=274s) The reality is, it's super context dependent. [04:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=277s) And so, for example, if I ask you what your name was, I have an expectation on how long it's going to take you to respond. [04:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=284s) If I ask you what the meaning of life is, I'm expecting a very different turnaround time. [04:50](https://www.youtube.com/watch?v=FQt2V0VafaE&t=290s) When we actually did testing and got people to rate if this response felt fast or slow, when you actually looked at the numbers, it actually was all over the place. [05:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=300s) Responses that took a long time, people said were fast and vice versa because it turned out it was super context dependent. [05:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=306s) Also where they were in the conversation. [05:08](https://www.youtube.com/watch?v=FQt2V0VafaE&t=308s) Generally people expect the very first response to be faster than subsequent responses because there's thinking time. [05:14](https://www.youtube.com/watch?v=FQt2V0VafaE&t=314s) But then sometimes you'll end up having kind of really short back and forth, so your expectation goes up. [05:19](https://www.youtube.com/watch?v=FQt2V0VafaE&t=319s) So, it's not this like perfect science where there's a perfect number, but we discovered about 1.5, if you're in that spot, you're you're you're getting pretty good. [05:27](https://www.youtube.com/watch?v=FQt2V0VafaE&t=327s) So, once you end up hitting this one point one 1.5 seconds, now you really start caring about consistency, right? [05:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=335s) Because if you end up having this happen 90% of the time, but 10% of the time it takes 3 seconds, your experience will feel broken, not bad, but broken. [05:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=344s) And so, we end up Once you end up getting fast enough, you just care about consistency, so fast all the time, not most of the time. ### Voice pipeline architecture and optimization **Devin Reimer** [05:53](https://www.youtube.com/watch?v=FQt2V0VafaE&t=353s) So, here's a standard voice pipeline architecture. [05:57](https://www.youtube.com/watch?v=FQt2V0VafaE&t=357s) So, we're taking our Speech-to-text in, sending it to an LLM, and then sending the result from the LLM back to the Text-to-Speech. [06:03](https://www.youtube.com/watch?v=FQt2V0VafaE&t=363s) So, what I mean by this is that, Speech-to-text, we're sending audio up to the server and then converting that to text. [06:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=370s) On the LLM side, we're taking that text, that response dialogue, or converting it into a response dialogue, and then what actions are going to take place from that character. [06:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=380s) We tightly couple these because we actually think that that's very important. [06:24](https://www.youtube.com/watch?v=FQt2V0VafaE&t=384s) And then on the Text-to-Speech side, we take that text, we send it to our Text-to-Speech, and stream that down to the user, also timed with the actions that the agent wants to perform. [06:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=394s) One of the problems with this is that running this sequentially can be quite slow because you end up having to step through each one of these things. [06:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=404s) So, a common question that I end up getting is, well, if that's slow and you have to do three steps, why not just use a speech-to-speech model instead? [06:52](https://www.youtube.com/watch?v=FQt2V0VafaE&t=412s) Cuz that would be faster, right? [06:54](https://www.youtube.com/watch?v=FQt2V0VafaE&t=414s) And, yeah, like, most of the time that would be faster because you're running one thing. [06:59](https://www.youtube.com/watch?v=FQt2V0VafaE&t=419s) But it comes with some big tradeoffs. [07:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=421s) So, while it's fast, there's some big deal breakers for us, and the biggest one is cost. [07:08](https://www.youtube.com/watch?v=FQt2V0VafaE&t=428s) So, generally we found about 3 to 5 times more expensive to run a Speech-to-Speech model than it is this parallelized approach. [07:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=437s) And so, for us, the economics just simply don't work. [07:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=440s) On the safety side of things, it actually is quite a bit more tricky because it's harder to validate the response the agent is going to have until they've said it. [07:30](https://www.youtube.com/watch?v=FQt2V0VafaE&t=450s) But meanwhile, if you have a pipelined approach, you can take the response the LLM is saying and validate and safety check right there before you even kind of got close to sending things down to a user. [07:41](https://www.youtube.com/watch?v=FQt2V0VafaE&t=461s) On the voice styling, you're kind of stuck with the presets within that Speech-to-Speech model. [07:47](https://www.youtube.com/watch?v=FQt2V0VafaE&t=467s) You can't be like, "Oh, do you know what? [07:48](https://www.youtube.com/watch?v=FQt2V0VafaE&t=468s) For this case, I really want to use this type of voice." [07:51](https://www.youtube.com/watch?v=FQt2V0VafaE&t=471s) You're kind of stuck with that particular one. [07:53](https://www.youtube.com/watch?v=FQt2V0VafaE&t=473s) In the pipelined approach, you can kind of like, "Oh, I'm going to change it up. [07:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=476s) I'm going to pick this model because this model works really good for this particular use case." [08:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=480s) And the same thing goes for intelligence, right? [08:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=482s) If you have a really good Speech-to-Speech model producing good voice, but the intelligence isn't quite what you want, well, you can't really just switch that out, right? [08:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=490s) You're kind of locked in. [08:11](https://www.youtube.com/watch?v=FQt2V0VafaE&t=491s) In the pipelined approach, you're like, "Oh, for this use case, I want to use a particular model." [08:16](https://www.youtube.com/watch?v=FQt2V0VafaE&t=496s) One area where Speech-to-Speech definitely does win is understanding user tone. [08:21](https://www.youtube.com/watch?v=FQt2V0VafaE&t=501s) So, things like sarcasm and things like that. [08:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=503s) Right now, speech-to-text transcriptions generally don't have a lot of that information, but I think that's going to change over time. [08:30](https://www.youtube.com/watch?v=FQt2V0VafaE&t=510s) That is definitely one thing in the win bucket for Speech-to-Speech. [08:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=515s) So, Speech-to-Speech definitely has an advantage when it comes to speed, but it definitely has some challenges when it comes to configurability and the high cost, particularly for us, was just a deal breaker. [08:45](https://www.youtube.com/watch?v=FQt2V0VafaE&t=525s) So, is there another way? [08:48](https://www.youtube.com/watch?v=FQt2V0VafaE&t=528s) Yes. [08:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=529s) What we should be doing is doing our best to parallelize the pipeline. [08:54](https://www.youtube.com/watch?v=FQt2V0VafaE&t=534s) And so, some of the things that we do are we end up running the LLM before the speech-to-text has full confidence. [09:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=542s) And by confidence, I mean confidence in the words that are being generated in the transcript, and then confidence of the end-of-turn. [09:09](https://www.youtube.com/watch?v=FQt2V0VafaE&t=549s) So, as soon as we think there's a chance that this is the moment the user stopped talking and this is what they said, we run ahead. [09:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=555s) And so, the worst-case scenario for that is that we're wrong, right? [09:19](https://www.youtube.com/watch?v=FQt2V0VafaE&t=559s) The user keeps speaking. [09:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=560s) But, in even in that scenario, all we do is we cancel the LLM and then run it again. [09:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=565s) And because the context will probably be cached and we haven't generated that many tokens, it doesn't actually have a big cost to going about doing that, and the benefit of so is that the user ends up hearing things faster than they would if you waited for full confidence. [09:42](https://www.youtube.com/watch?v=FQt2V0VafaE&t=582s) We also don't wait for the LLM to finish generating its whole response before we start sending it to the Text-to-Speech. [09:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=589s) What we wait for is kind of the first speakable chunk. [09:52](https://www.youtube.com/watch?v=FQt2V0VafaE&t=592s) For most scenarios, it's a sentence. [09:54](https://www.youtube.com/watch?v=FQt2V0VafaE&t=594s) Sometimes it can be some other configurations, but as soon as we have that first sentence chunk, we send it to our Text-to-Speech. [10:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=600s) And then, if we have a good Text-to-Speech that'll stream, as soon as it's ready to stream, then that can come down to the user. [10:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=606s) We also are really careful on the number of tokens that need to be generated. [10:11](https://www.youtube.com/watch?v=FQt2V0VafaE&t=611s) So we don't use things like JSON wrappers and things like that that might add additional tokens. [10:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=617s) We try to figure out what the most simplified structure is to getting the fastest response possible. [10:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=623s) We also use things like assistant prefill if we know what the first chunk of tokens is going to end up being from the agent ahead of time. [10:30](https://www.youtube.com/watch?v=FQt2V0VafaE&t=630s) And so for the text-to-speech, it ended up being these two chunks, right? [10:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=634s) So that first speakable chunk comes in and then we then run everything else after. [10:40](https://www.youtube.com/watch?v=FQt2V0VafaE&t=640s) But because hearing audio playout takes longer than generation, almost universally that first chunk will be finished playing to the user by the time the second chunk is done. [10:51](https://www.youtube.com/watch?v=FQt2V0VafaE&t=651s) So that actually is invisible to the user. [10:54](https://www.youtube.com/watch?v=FQt2V0VafaE&t=654s) It comes just as fast as it possibly can there. [10:58](https://www.youtube.com/watch?v=FQt2V0VafaE&t=658s) So when you're thinking about kind of picking different services, here's some of our rules of thumb. [11:03](https://www.youtube.com/watch?v=FQt2V0VafaE&t=663s) When it comes to picking a good kind of audio and agent framework, what you care about is lowest latency in and out. [11:09](https://www.youtube.com/watch?v=FQt2V0VafaE&t=669s) So how fast can I send things up and pull things down? [11:12](https://www.youtube.com/watch?v=FQt2V0VafaE&t=672s) On the speech-to-text side of things, we care about time to confidence. [11:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=677s) So this is confidence in transcript and confidence in end of turn. [11:21](https://www.youtube.com/watch?v=FQt2V0VafaE&t=681s) Those paired together, right? [11:22](https://www.youtube.com/watch?v=FQt2V0VafaE&t=682s) If you don't have confidence fast in one, it's just going to be as slow as that, right? [11:26](https://www.youtube.com/watch?v=FQt2V0VafaE&t=686s) So it's only as fast as the slowest one within that. [11:30](https://www.youtube.com/watch?v=FQt2V0VafaE&t=690s) On the LLM side of things, we care about time to first sentence, that first speakable chunk. [11:36](https://www.youtube.com/watch?v=FQt2V0VafaE&t=696s) And so that's two parts, right? [11:38](https://www.youtube.com/watch?v=FQt2V0VafaE&t=698s) It's time to first token combined with tokens per second. [11:41](https://www.youtube.com/watch?v=FQt2V0VafaE&t=701s) So if all of a sudden you find a really good model that gives you really good time to first token, but is slow in tokens per second, that could be worse than the alternative. [11:50](https://www.youtube.com/watch?v=FQt2V0VafaE&t=710s) So figure out what the right combination is for that. [11:53](https://www.youtube.com/watch?v=FQt2V0VafaE&t=713s) And then on the text-to-speech side of things, you care about time to first audio. [11:58](https://www.youtube.com/watch?v=FQt2V0VafaE&t=718s) And so it's really important to measure what it is, because sometimes audio can come back with leading silence and the user isn't going to hear that. [12:05](https://www.youtube.com/watch?v=FQt2V0VafaE&t=725s) So you actually have to measure to the point the user is hearing things. [12:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=730s) And then, as I mentioned before, above all else you care about stability, not peak speeds, not best-case scenario. [12:16](https://www.youtube.com/watch?v=FQt2V0VafaE&t=736s) What is the consistent latency of this whole stack? ### Astrobeam's voice agent stack **Devin Reimer** [12:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=740s) So, currently for our audio agent frameworks, we're using LiveKit. [12:26](https://www.youtube.com/watch?v=FQt2V0VafaE&t=746s) For speech-to-text, we're using Deepgram. [12:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=749s) For LLMs, we're using Gemma 4 31B, and we host that on LiveKit. [12:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=755s) On the text-to-speech side of things, we're using Inworld AI text-to-speech 2. [12:40](https://www.youtube.com/watch?v=FQt2V0VafaE&t=760s) And then on top of that we've built our whole AstroBeam AI agent harness that not only ties this all together, but adds all the pieces that you need to have a really good voice agent, particularly one that's highly interactive like ours for games. [12:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=776s) So, one thing I haven't talked about is what about safety models? [13:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=780s) So, traditionally, safety models are part of the LLM pass, but that adds latency and adds an additional step in there. [13:08](https://www.youtube.com/watch?v=FQt2V0VafaE&t=788s) So, what if instead you run your own? [13:12](https://www.youtube.com/watch?v=FQt2V0VafaE&t=792s) So, this is actually more what our stack looks like. [13:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=795s) And so what we're doing here is when we get that first chunk of text back, that first speakable chunk that we're going to send over to the text-to-speech, at the exact same time we send it over to our safety model. [13:27](https://www.youtube.com/watch?v=FQt2V0VafaE&t=807s) And because the safety model is smaller and will run faster than the speech-to-text side of things, it is almost universally done ahead of time. [13:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=815s) And so what this ends up meaning is that adding a safety model adds zero latency to our stack, and even better that's a sizeable enough chunk that we can actually run a much bigger safety model than most people even think about running. [13:50](https://www.youtube.com/watch?v=FQt2V0VafaE&t=830s) And that's super important for us, particularly because we're building out games. [13:54](https://www.youtube.com/watch?v=FQt2V0VafaE&t=834s) There's so much context of that world that's super important. [13:59](https://www.youtube.com/watch?v=FQt2V0VafaE&t=839s) So, for example, let's say you're building a medieval game, and you have an NPC in this medieval game that he shouts like, "Charge! [14:05](https://www.youtube.com/watch?v=FQt2V0VafaE&t=845s) Off with their heads!", right? [14:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=846s) A traditional safety model might be like, "Whoa, that's not cool. [14:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=850s) You shouldn't be saying that." [14:11](https://www.youtube.com/watch?v=FQt2V0VafaE&t=851s) But within the context of this game, that makes perfect sense. [14:14](https://www.youtube.com/watch?v=FQt2V0VafaE&t=854s) And the way that we can do that is we use a bit more of a robust safety model with a lot more context of what's happening in the world to be able to make that work. [14:22](https://www.youtube.com/watch?v=FQt2V0VafaE&t=862s) You might notice here that we actually run the safety model twice, and so we end up running it on that first chunk and then we run it again on the whole thing when we have that final chunk all together. [14:32](https://www.youtube.com/watch?v=FQt2V0VafaE&t=872s) Once again, we have to run it twice, but it adds no latency to the stack. [14:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=877s) So, now we're fast enough, but unfortunately this doesn't matter if we're not smart enough. ### Defining and evaluating agent intelligence **Devin Reimer** [14:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=884s) So, how smart does it need to be? [14:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=889s) It needs to be smart enough for your users, but almost more importantly smart enough for you to develop with. [14:55](https://www.youtube.com/watch?v=FQt2V0VafaE&t=895s) And this is often a thing that people don't think about when they're kind of going about selecting kind of what they need for intelligence here. [15:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=902s) We've been working on voice agents for about three years now, and for example, in the early days we were using some early LLaMA models, and while we were able to get them to work-ish for users, the development side was very expensive because everything was very brittle. [15:18](https://www.youtube.com/watch?v=FQt2V0VafaE&t=918s) It wouldn't take much for the models to generate undesirable responses or things that just didn't make sense, and so we spent so much engineering effort to make that work. [15:28](https://www.youtube.com/watch?v=FQt2V0VafaE&t=928s) And so that's something we prioritize, because if we can't move fast in building these things, we can't get things out to the users. [15:36](https://www.youtube.com/watch?v=FQt2V0VafaE&t=936s) So, very important to kind of think about both of those. [15:40](https://www.youtube.com/watch?v=FQt2V0VafaE&t=940s) So, often when people think about intelligence, they think about the LLM, right? [15:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=944s) But that's just one part of it. [15:46](https://www.youtube.com/watch?v=FQt2V0VafaE&t=946s) We have to think about every piece when it comes to intelligence. [15:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=949s) So let's talk about speech-to-text intelligence. [15:51](https://www.youtube.com/watch?v=FQt2V0VafaE&t=951s) So what we care about here is a low word error rate. [15:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=956s) And additionally, it's not just that the error rate is low, it's that when it happens, we're generating something that's very close to what the actual thing is, because if it is really close, the LLM can often, with context, do a really good job of understanding what it is. [16:12](https://www.youtube.com/watch?v=FQt2V0VafaE&t=972s) If your error rate is low, but the missteps are very high, the LLM will struggle a lot more in trying to figure out what the actual intent was. [16:21](https://www.youtube.com/watch?v=FQt2V0VafaE&t=981s) Another thing to think about is what your target platform is. [16:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=985s) So, for example, if I am building for XR, and I've got microphones that are really close to user's mouth, I'm going to have really good, high quality microphone audio data, and that might work really well for certain models. [16:38](https://www.youtube.com/watch?v=FQt2V0VafaE&t=998s) But if all of a sudden, I'm building an app for a mobile phone that people are going to use out on a busy street, that has a lot different impact, and different models will behave differently and be able to be intelligent about what's being said in different ways. [16:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1009s) So you have to think about that, make sure you're picking for the right platform you're building for. [16:55](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1015s) So, then moving over to the text-to-speech side of the intelligence equation. [17:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1020s) Does it sound natural, right? [17:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1022s) Our goal is communicating to the user what it is the user able to understand that? [17:08](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1028s) But that's only part of it, right? [17:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1030s) Expressiveness of that agent is also super, super important because there's information in being expressive, and also in our play tests, we've discovered that the more expressive our agents can be, the more intelligent users felt they were, which makes a lot of sense. [17:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1049s) And then, is it fast enough? [17:31](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1051s) And I know in the previous section, I was talking about latency, but it actually also matters here for perceived intelligence. [17:36](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1056s) Users perceive speed of response directly correlated to how it actually is when it comes to intelligence. [17:45](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1065s) If I have a quick response, it probably sounds more intelligent. [17:48](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1068s) And so, these things can really matter when it comes to how the user is perceiving the experience, and also how immersed they can be in the experience. [17:57](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1077s) So then, on to the LLM side of the intelligence. [18:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1080s) This is, by far, the hardest to balance cuz there's so many different requirements here, right? [18:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1086s) So we have kind of speech understanding, context, and grounding. [18:11](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1091s) Within games, this is super tricky, because you have to ground on an entire world and universe, right? [18:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1097s) This isn't a text box where it can make up things that are happening. [18:21](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1101s) If If your agent within a game says, "There's an apple on the table," and there's no no apple on the table, that is going to break the user's experience. [18:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1109s) We also need things like strong rule following. [18:31](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1111s) Games are very rules based. [18:33](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1113s) Can the LLM kind of handle that? [18:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1115s) Can we have a low hallucination rate? [18:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1117s) Can we have consistent characters when you're interacting with that same agent over and over again? [18:41](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1121s) Does it feel like the same agent? [18:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1124s) And then, one that might trip you up, because it's not a problem at the start of development, is, can your model handle the prompt size growing? [18:50](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1130s) Because inevitably, as you continue to develop, you're going to be putting more stuff in the prompt. [18:55](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1135s) But then, also, as users keep continue to play, there's going to be more chat history in that prompt. [19:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1140s) So, will that prompt handle that growing? [19:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1142s) Because sometimes what happens is, for certain models, as your prompt grows, the quality of response goes down, and so you have to kind of figure out what your actual needs are for the amount that has to go into that prompt every time. [19:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1155s) But this is all just the tip of the iceberg, and it also really depends on what you're building. [19:19](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1159s) So you have to think really hard about what you're building, and what you're trying to achieve. [19:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1163s) So now we're fast enough, and we're smart enough, none of this matters if it costs too much. ### Voice agent economics and cost modeling **Devin Reimer** [19:31](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1171s) So how cheap does this need to be? [19:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1175s) Unfortunately, it's a little bit more complicated than that. [19:39](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1179s) So we have to think about the economics of the application or game that we're building, right? [19:43](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1183s) So, a few questions you need to ask. [19:46](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1186s) How long will the user be interacting with your voice agents? [19:50](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1190s) How much are you making per user? [19:52](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1192s) And how much does that voice stack cost per hour to run? [19:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1196s) We put them together to try to figure it out, see if things make sense. [20:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1201s) So, here's the economics of our first game, Stellar Cafe. [20:05](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1205s) So the playtime in Stellar Cafe is somewhere between 1 and 1/2 and 3 and 1/2 hours, so let's just average it to about 2 hours. [20:13](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1213s) Our cost of running voice agents is about $1.20 per hour per user, and users are interacting with it 100% of the time. [20:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1220s) It's a 100% voice-driven game. [20:24](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1224s) We charge a flat $10 fee for the game, which is traditional. [20:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1229s) We don't do subscriptions because they're very difficult in games, and I'm a firm believer as we have to get it to work at a flat cost, otherwise, this just isn't going to work, and isn't going to scale. [20:39](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1239s) Previously, we were charging $15, but we've been working on getting these server costs down, so now we've actually got it down to $10. [20:47](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1247s) And this $1.20, this is where we're at. [20:50](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1250s) This isn't some universal how much it costs. [20:52](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1252s) I've seen these things cost at a wide variety of things, and much more expensive at times. [20:58](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1258s) And so, you also have to kind of figure out kind of where you are kind of on the cost side of things. [21:03](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1263s) We could go cheaper, but we're trying to hold our quality bar as high as we possibly can. [21:09](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1269s) So, kind of working this all through, this means at that 2 hour play time, it's about $2.40 our voice stack per user. [21:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1277s) And so, because we charge $10, and the store takes a fee, that means we make about $4.59 per user, right? [21:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1285s) And so, the economics of that works, right? [21:26](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1286s) The voice is our hook, that's what we want to be spending money on, this works. [21:30](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1290s) So, this is a really tiny game. [21:32](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1292s) Let's talk about something a bit bigger. [21:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1295s) So, let's talk about a hypothetical game, a narrative game, that's somewhere between say 5 and 9 hours, so let's say on average 7 hours. [21:42](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1302s) In reality, most people-- a lot of people don't finish games, they play a little bit of a game, and kind of abandon it, but I want to pick something a little bit higher to make sure the economics kind of still work. [21:51](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1311s) So, we'll keep that $1.20 an hour, but what change we're going to make is that the voice component of it is going to be used 60% of the time, because a bigger game is going to have a lot more mechanics than just voice. [22:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1322s) So, it's still going to be a very voice-driven game, but it's going to have other kind of features within it. [22:08](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1328s) So, if we charge $20, kind of working all this through again, it costs us about-- would cost us about $5 per user, which means at the end of the day, we make about $9 per user. [22:18](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1338s) So, once again, with this, the economics work. [22:21](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1341s) This doesn't mean the economics are going to work for everything. [22:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1345s) So, for example, something not to do, if you were like, "Oh, I want to build a free-to-play game, and probably this game users are going to play somewhere between 1 hour and 100+ hours." [22:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1354s) So, let's average it to 10 hours. [22:36](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1356s) It's 100% voice-driven, but let's say we have a hit on our hands, and the game is generating about $3 per user, right, which would be pretty great as a free-to-play game. [22:45](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1365s) You would lose $10 per user. [22:47](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1367s) So, it doesn't yet work at all of these things, and you have to kind of think about what you're currently building, and does it make sense? [22:55](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1375s) Don't wait till the end to try to figure out the economics. ### Local models versus cloud deployment **Devin Reimer** [23:00](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1380s) So, then the next question that gets asked is, well, what about local models, right? [23:04](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1384s) Because if you have this big challenge of cost, wouldn't local models eliminate it? [23:09](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1389s) And yes, it would. [23:12](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1392s) But there's some tradeoffs here that we have to kind of think about. [23:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1395s) So, on the cost side of things, yeah, we would eliminate the server bill, which would be a huge benefit, open up a lot more opportunities. [23:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1403s) But there's a lot of things that right now the cloud can solve that local struggles with. [23:28](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1408s) So, for example, compute. [23:30](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1410s) Often people are like, "Look, I found this really good, say, speech-to-text model, and it runs locally on this device, right?" [23:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1417s) Great, but what we have to think about is it's not running one model, it's running multiple models, right? [23:43](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1423s) In our case, four, potentially at the same time, and so, it's not enough for something to run once, it has to run multiple times at the same time. [23:53](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1433s) And then, if you get that done, did you leave anything left for your game or application to run? [23:58](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1438s) If you've maxed out the GPU, you're not really going to render much of a video game. [24:03](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1443s) So, let's say you've gone and conquered those challenges. [24:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1446s) So, the next one is speed, right? [24:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1450s) Often you can get them to run, but they will be really slow, and it'll feel more like prompting than an actual real-time conversation. [24:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1457s) So, in theory, getting it local does have a future win, though, because you don't have to do that round trip from the internet. [24:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1463s) It is a challenge right now getting models that will run at the right speed to make this work. [24:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1469s) So, next let's say you kind of conquered the compute, and you've conquered the speed, now we have to think about quality, right? [24:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1475s) So, generally because you're running it on a local device, you're using a smaller LLM, say, a more primitive text-to-speech, this can actually represent some challenges when it comes to what content you're building. [24:48](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1488s) And as I mentioned previously, when you're thinking about the LLM, it's not just is it good enough for your users, is it good enough for you to develop with? [24:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1496s) And so, if that model becomes quite challenging to build something with, that's definitely something to think about. [25:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1502s) So, say you've conquered compute, you've conquered speed, you've conquered quality, next up is portability. [25:09](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1509s) If you won and went through this whole process, and you got this thing running on, say, an RTX 5070, that might be the only place it can run. [25:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1517s) So, if an opportunity comes up to bring it to a new platform, you might just be stuck. [25:22](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1522s) On the cloud side of things, all you need is the internet. [25:24](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1524s) And so, that is a kind of a big unlock here. [25:27](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1527s) Allows to go to multiple platforms. [25:29](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1529s) So, next, let's talk about a hybrid approach, right? [25:33](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1533s) So, let's run some of the stuff in the cloud, and some of the stuff locally. [25:38](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1538s) Kind of, could that work? [25:40](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1540s) Well, it's not as big of a win as you think. [25:43](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1543s) So, if you end up getting a model that's small enough to run on most local platforms, if you ran that same model in the cloud, it probably cost you pennies. [25:52](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1552s) And so, it generally isn't until you've kind of solved the whole stack that you get your biggest wins. [25:59](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1559s) Then the next, thing that kind of gets brought up is, what if instead you run it locally if you it's possible, right? [26:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1566s) They have the right hardware, and in the cloud if you don't, right? [26:10](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1570s) You could run a more powerful thing in the cloud. [26:13](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1573s) This has new challenges, right? [26:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1575s) If you're end up running a different model, this is could impact development quite a bit. [26:19](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1579s) And you end up having to spend a lot of time working on the min spec anyway. [26:24](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1584s) And you could have models that behaved quite differently. [26:27](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1587s) So, unless you're using the exact same models, right, you could add up end up adding a lot of dev time and cost. [26:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1595s) So, it's an important thing to kind of think about kind of what you're trying to achieve, because let's say you did figure this out and get a good local if it needs, cloud if it doesn't, it's still the economics have to work for the cloud model. [26:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1609s) Cuz if, all of a sudden, all your users are on the cloud side, right, you're going to lose if you didn't think about that, from an economic standpoint. [26:58](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1618s) So, when it comes to local models, it's definitely, a big economics win, but it's still early. [27:04](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1624s) For price-sensitive applications like games, I believe it'll likely win eventually. [27:09](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1629s) Today, it's kind of tricky outside of more specific use cases, particularly things around command-based stuff. [27:15](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1635s) So right now, today, we want the kind of the best experience possible. [27:18](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1638s) That is why we're using the cloud. [27:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1640s) But this is a very fast-moving space, and things are changing quite rapidly. [27:26](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1646s) So, the next part is let's think about the models we're using, right? [27:31](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1651s) So, the next question is open weights versus closed weights. [27:35](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1655s) What should we be doing here? [27:39](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1659s) I want to focus on LLMs when we're talking about this because we found this is the area that is most affected by the open versus closed question. [27:48](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1668s) So, there's definitely some pros and cons here. [27:51](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1671s) So, when it comes to closed, weight models, these are generally your frontier intelligence, right? [27:57](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1677s) So, if you need the most intelligence, this is often where you have to go. [28:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1681s) Meanwhile, on the open side, they're often a generation behind. [28:04](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1684s) So, this can be a real trade-off. [28:07](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1687s) On the speed side of things, I mentioned kind of caring a lot about time to first sentence. [28:12](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1692s) What we kind of discovered is over the lifetime of closed models, the speed often goes down. [28:19](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1699s) And there can be many reasons for this. [28:20](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1700s) Maybe more people using the model, maybe hardware has shifted to a newer model. [28:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1705s) But it's one thing that we've noticed. [28:27](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1707s) And so, you have to be careful because it can cause a problem where if you ship something and it's great, it could, over time, become unusable. [28:36](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1716s) Next is spikes. [28:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1717s) You care a lot about that kind of consistency, and so, often, kind of the closed models are built around kind of batching in asynchronous tasks, not kind of P99 latencies, where, on the open side, you can go pick a provider that cares about that very specific thing that you care about, to give you kind of the confidence and the speed and the latency that you end up needing. [29:01](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1741s) The next is lifetime of the model itself. [29:04](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1744s) So, with closed models, they often sunset after about a year. [29:08](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1748s) So, what this means for you is if you end up developing something for a year, and you expect it to be in production for 3 years, you could go through more than 3 model changes. [29:18](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1758s) And just because a model is better, doesn't mean it's going to work better for you. [29:23](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1763s) We've gone through a lot of model changes in the last 3 years, and often they take weeks, sometimes months of work to retool everything, which, on the open side, you can continue to run that model as long as someone's willing to host it, or you're willing to host it yourself. [29:39](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1779s) Lastly, is cost. [29:41](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1781s) Often, over time, these closed models, as they release new versions, are more expensive than the previous one. [29:47](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1787s) And so, this is something else to be careful of when it comes to economics, is, once again, your economics could have worked at launch and then not worked as you end up migrating models. [29:55](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1795s) On the open side, you have flexibility to be able to move it around, and odds are it's not going to drastically increase because it is the exact same model. [30:06](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1806s) So, we're using Gemma 4 31B and this was a huge unlock for us because, for us, this was kind of the best combination of speed, intelligence, and cost. [30:16](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1816s) It allowed us also to move really fast. [30:17](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1817s) It's kind of much easier to develop with than some things we've used before. [30:22](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1822s) This doesn't mean that we just use this. [30:24](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1824s) We use a lot of combinations of different open and closed models. [30:28](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1828s) So, for example, things like memory generation and safety and customization. [30:34](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1834s) It's just really important to pick the best model for your particular needs in that task. [30:38](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1838s) But Gemma 4 31B is our workhorse when it comes to voice agents. [30:44](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1844s) So, when it comes to this kind of differences between the two, open models definitely give you greater control and allow you to keep models as long as they can be hosted at a more predictable cost. [30:57](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1857s) And you don't have an impact in your schedule by a model being sunset. ### Balancing latency, intelligence, and cost **Devin Reimer** [31:02](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1862s) So, to kind of wrap this up, it's about balancing all these three things, right? [31:07](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1867s) Your latency, we're trying to balance kind of the streaming aspect, the parallel aspect, the speed, and the consistency of that. [31:14](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1874s) On the intelligence side of things, smart enough for the users, smart enough to build with. [31:18](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1878s) On the cost, we need to know how much it's going to cost per user and how much we can make per user so we can make sure the economics work. [31:25](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1885s) Don't build first and then figure out the economics second. [31:28](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1888s) Think about that right from the very start. [31:32](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1892s) So, it's still very early, but the economics here are starting to work. [31:37](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1897s) And so, cheaper will definitely be greater, more intelligent models will definitely be greater. [31:42](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1902s) But right now, today, we can start working on the really hard challenges we need to work on to make voice agents a thing everywhere. [31:49](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1909s) And so, I am super excited to bring these power of voice the power of voice agents to everyone. [31:56](https://www.youtube.com/watch?v=FQt2V0VafaE&t=1916s) Thank you.