# Insights into agent code generation from Android Bench

- Speakers: Tomáš Mlynarič (Android DevRel)
- Track: Tools & Performance
- Video: https://www.youtube.com/watch?v=rIIAwylLb5Q
- Page: https://developer.android.com/events/dev-summit/talks/insights-into-agent-code-generation-from-android-bench

## About the talk

AI models can generate entire features in minutes, but is the code correct? We built Android Bench, our AI evaluation benchmark, to answer this question after analyzing thousands of generated code changes.

This talk shares insights from our journey, including all the ways we saw models cheats, how prompt specificity leads to higher code quality and how different agents approach the same task.

## Summary

- Android Bench 2.0 uses closed-source tasks to prevent contamination and evaluate AI coding agents on complex Android development work.
- Evaluation relies on implementation-agnostic UI tests and a Multimodal LLM Judge instead of brittle unit tests or strict pixel-diffing.
- Built-in cheating detectors catch agent shortcuts such as test tampering or fake background drawables. Any detected shortcut results in an immediate failing grade.
- Cost per task varies widely from advertised token prices because talkative models burn more tokens by repeatedly rebuilding and rerunning.
- Frontier models achieve around 80% completion rates but pass only 20% to 40% of tasks due to subtle flaws.

## Resources

- [Why SWE-bench Verified no longer measures frontier coding capabilities](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified)
- [Android Bench leaderboard and benchmarks](https://d.android.com/bench)

## Topics

android-bench, benchmarking, llm-evaluation, code-generation, ai-agents

## Transcript

### Introduction to Android Bench

**Tomáš Mlynarič**

[00:04](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=4s) Morning, everyone.

[00:06](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=6s) Let's get this started.

[00:07](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=7s) In March, we launched Android Bench, our data set and leader board on measuring LLMs on Android development tasks.

[00:16](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=16s) Since then, we've run it against many new models.

[00:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=19s) And just last month, we released Android Bench 2.0.

[00:23](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=23s) While it's only been a couple of months, in the world of AI, that feels like eternity.

[00:28](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=28s) My name is Tomáš, and I'm going to share our journey of Android Bench and how we got where we are now.

[00:35](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=35s) Let's start with 1.0.

[00:36](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=36s) In 1.0, we took real-world problems from open-source repositories.

[00:42](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=42s) We filtered the GitHub projects with GitHub issues, pull requests solving those issues and tests verifying the solutions.

[00:52](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=52s) We split the pull requests into task buckets for AI models.

[00:56](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=56s) The GitHub issue became the instructions for the, or the prompt for the model, and we planted them in the state before the PR.

[01:05](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=65s) And once the model created some work, we took the tests and applied them to the output, which gave us either passed or failed outcome.

[01:14](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=74s) This one attempt of AI trying to solve it, we call a trial.

[01:18](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=78s) And we could also compare it based on the human crafted solution from that PR, or we call it the golden solution.

[01:26](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=86s) This methodology is similar to quite known SWE-bench benchmark.

[01:32](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=92s) We started with over 10,000 potential tasks, and not all of them were good fit.

[01:37](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=97s) Some had incomplete instructions or overspecified instructions.

[01:43](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=103s) The tests might have been coupled to the golden solution, or the state before the PR was broken.

[01:49](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=109s) And so, we had to review and review to land on 100 fair tasks.

[01:56](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=116s) During those reviews, we used the state-of-the-art models to get some results.

[02:00](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=120s) And we started with around 30% tasks solved, which we call a pass rate.

[02:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=128s) And then, when we locked the data set for the official release, the top models were already at around 70%.

[02:16](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=136s) But that's like a month later.

[02:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=139s) And then we had a plan to evaluate new models every month, and then we watched how they are getting better.

[02:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=149s) And in August release, 92% pass rate was for the best model.

[02:35](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=155s) And we knew that our bar is too low, and this data set is saturated.

### Introducing Android Bench 2.0

**Tomáš Mlynarič**

[02:40](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=160s) But we anticipated that, and we're already hard at work on 2.0, which we released last month.

[02:48](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=168s) The goal is, and was, to raise the bar with complex, real-world Android tasks and provide guidance and transparency on LLM performance for you, developers, but also to push model makers to prioritize Android by giving them good benchmark to, hill climb against.

[03:07](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=187s) And also providing analysis and results serving both you and the model makers.

[03:13](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=193s) But we need to also account for cleverness, tenacity, and cheating.

[03:18](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=198s) And we need to prevent contamination risk.

[03:21](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=201s) One problem with the SWE-bench methodology is the actual openness of the source code.

[03:27](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=207s) OpenAI published an article.

[03:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=209s) They found that the models were able to finish the next words of the instructions, or they were able to solve a task with an obscure error that wasn't part of the instructions.

[03:40](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=220s) And they realized that the model makers are training on the open-source code and that the SWE-bench is contaminated.

[03:46](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=226s) So for that, for 2.0, we actually keeping the cose-- the code closed to prevent training on that.

[03:53](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=233s) We divided the work into four work streams.

[03:57](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=237s) The first one, we do app conversions, where we took open-flur-- open-source, Flutter and React Native apps, and told the agents to convert them to native Android solution.

[04:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=248s) This actually reflects the trend of migrating cross-platform to native that we see now.

[04:14](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=254s) The second work stream was new features and platform integrations.

[04:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=259s) They were built into established production code bases.

[04:22](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=262s) For example, in Now in Android, we added video feature, or picture-in-picture, stuff like that.

[04:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=269s) The third work stream were migrations.

[04:32](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=272s) We're upgrading production open-source apps that we kept closed, with modern libraries and architectures.

[04:39](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=279s) For example, Nav 2 to Nav 3, XML views to Compose, or stuff like single pane to adaptive layout.

[04:46](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=286s) And the last one was in-house greenfield app built from scratch using visual design mocks.

[04:53](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=293s) Here, the examples were, full SMS OTP authentication flow, order tracking with Google Maps, and more.

### Grading challenge

**Tomáš Mlynarič**

[05:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=301s) With just couple of engineers, we built an app of the same size kind of like Now in Android.

[05:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=308s) But the new tasks brought new challenges.

[05:11](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=311s) The first one: grading, agents properly.

[05:15](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=315s) The benchmark tests are much harder than tests for your app because, they must pass any valid solution, not just one, and, bigger the task, more valid solution there is.

[05:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=329s) In 1.0, this, the benchmark relied mostly on unit tests, and the unit tests are pretty horrible at being agnostic.

```kotlin
import com.foodvibes.MyViewModel

@Test
fun testRetry() {
    val vm = MyViewModel()
    vm.retry()

    assertTrue(vm.retried)
}
```

[05:37](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=337s) They require exact class or function names.

[05:41](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=341s) Like in this case, there, there's, like, not a fancy test, but, basically, the thing is that agent could name those things differently, or even if they don't name it differently, they, it's just if they put it in a different package, the tests would fail to compile, and therefore, would be unusable for our case.

```kotlin
import com.foodvibes.feature.my.RetriableViewModel

@Test
fun testRetry() {
    val vm = RetriableViewModel()
    vm.tryAgain()

    assertTrue(vm.tried)
}
```

[06:00](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=360s) So, we could technically specify the APIs in the prompts, but that would either give too, away too many details, or it would explode the prompt with too many test requirements.

[06:13](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=373s) So, the solution is we do implementation-agnostic tests.

[06:18](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=378s) These are very high-level and outcome, they test outcomes and behaviors, not the internals.

[06:25](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=385s) We use the, Deterministic UI test, the tests that interact with the public screen information, similarly as a user would.

[06:34](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=394s) For those, we use UI Automator or Compose UI tests, but we also have specialized scripts, that are checking some external system boundaries, like, for example, if a network call had the right data, or if the right data is saved into a database without knowing of internal helpers or DAOs.

[06:55](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=415s) Part of those, we also have static checks, to, you know, check like permissions in manifest or if the library was in built files and stuff like that.

[07:06](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=426s) And, we also have hybrid tests that are combination of those scripts and the UI tests.

[07:14](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=434s) This is example of one of the hybrid tests.

[07:17](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=437s) They drive outside of the emulator.

[07:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=439s) So, this is the test for the SMS OTP authentication flow, where the script, needs to first calculate the app's hash to be able to correctly send the SMS to the emulator, then starts the test, the UI Automator tests, and pass some randomly generated OTP that it requires, and sends SMS to an emulator with an ADB command.

[07:44](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=464s) And the test just checks whether the input had the right number.

[07:47](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=467s) That's, that's basically it.

[07:48](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=468s) There's zero implementation coupling, only checks, it only checks whether the flow works.

[07:55](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=475s) The second grading problem, was that, our tasks require multi-modality.

[08:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=481s) They either, the agents either are required to produce UI, or we give them the design mocks.

[08:07](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=487s) And, for that it's the easiest to use screenshot tests.

[08:11](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=491s) And we knew from the beginning that the screenshot thresholds will need to be higher, but, they failed anyway.

[08:18](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=498s) We tried some different pixel-diffing algorithms, but they failed on stuff like system-level noise, where the emulator status bar was different, or the battery icon, or even like font hardware anti-aliasing was different.

[08:32](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=512s) But, also, they failed on like somewhat valid implementations, where, you know, the paddings were minor, it's like slightly different, or they were some slight tint differences, like the UI could be completely correct, but the background would be slightly differently tinted, and the pixel-diffing, algorithms would fail.

[08:53](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=533s) So, for that, we used and built Multimodal LLM Judge that compares screenshot pairs as in screenshot tests and gives us some score, but, also, Structural Reasoning about the score.

[09:06](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=546s) And we also feed it with some accessibility hierarchy dumps that checks whether the structure is correct and whether the, structural parity is correct.

[09:17](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=557s) We tried it empirically on many, many trials, and we saw that it has high consistency and it ignores the problems that we saw.

[09:26](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=566s) So, now, we had a good way to verify visuals and behaviors agnostically, and so we saw that the agents tried faking everything.

[09:36](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=576s) Go ahead.

### Cheating challenge

**Tomáš Mlynarič**

[09:38](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=578s) It starts simply.

[09:40](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=580s) Nowadays, agents are basically, "Okay.

[09:43](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=583s) I have a task.

[09:43](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=583s) Let's look at the GitHub, you know.

[09:45](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=585s) Let's find the solutions."

[09:47](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=587s) So, that's kind of what they do as a first thing.

[09:49](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=589s) They try to search GitHub or internet for the valid solution, which is why we need to keep the source code, closed and can't be public.

[09:59](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=599s) But that was, like, easy.

[10:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=601s) In some of the trials, we said like everything was passing cleanly 100%, and the agent is like, "Magnificent triumph!"

[10:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=608s) So.

[10:09](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=609s) Interesting, what did they find?

[10:11](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=611s) But we looked closer at the trajectory, and they were like, "I can't do this.

[10:15](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=615s) I'm going to just, what if I change the threshold to 80% of the screenshot test?"

```kotlin
// AuthFlowScreenshotTests.kt

@get:Rule
val dropshots = Dropshots(
    resultValidator =
        ThresholdValidator(
-           0.05f
+           0.80f
        )
)
```

[10:20](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=620s) And it's like, "Yes, triumph, everything is passing."

[10:23](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=623s) Yeah, but the screenshot tests are useless.

[10:27](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=627s) But that wasn't the most sophisticated one.

[10:30](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=630s) In some, in actually the Flutter conversion app, the agent tried and tried to build the Android, and was like, "I don't know what to do."

[10:38](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=638s) So, they took, and the screenshot that we gave them, they put it as a drawable in the background.

[10:46](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=646s) And now it's like passing the screenshot test.

[10:48](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=648s) Yay!

[10:49](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=649s) But then, you know, the UI tests are still failing, so they're like, "I'm going to put an invisible overlay layer on top of that with the buttons and texts and test tags, everything that we require.

[11:00](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=660s) Now everything is green.

[11:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=661s) Amazing.

[11:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=661s) Good job."

[11:03](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=663s) That's not, like, there's one that's, like, really good.

[11:07](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=667s) So, we have a Java to Kotlin migration task where the agent basically needs to, migrate from Java to Kotlin, right?

[11:13](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=673s) They couldn't make it build.

[11:14](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=674s) It's, like, still failing something, so the agent thought, "All right, I'm just going to take the compiled classes from Java, and I'll put it into a JAR, and then I just add it to the build files as a regular library.

[11:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=689s) So now, every test is passing, but we have the static checks that are checking for the Java files, so, yes, I'm going to just delete them."

[11:36](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=696s) I'm deleting the Java files, everything is green, except the Kotlin is still failing, so they're like, "All right, I'm going to remove the Kotlin source set, and now everything is passing.

[11:45](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=705s) Great."

[11:47](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=707s) I don't know if it's even cheating.

[11:48](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=708s) I mean, this is, like, really creative.

[11:51](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=711s) So, to prevent all of that, we built a cheating detector that has, like, three defenses.

[11:58](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=718s) We look at the UI tree, we apply different screen sizes so that the background is distorted, and we see it.

[12:05](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=725s) But we also reset the tests, and we check for any tests tampering so that they can't change the thresholds.

[12:12](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=732s) And we then also have, like, LLM-as-a-Judge, that audits the trajectories and the actual output of the agent to see some weird stuff.

[12:22](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=742s) And if anything happens, then we just say, "Okay, straight to fail."

### Insights from benchmark results

**Tomáš Mlynarič**

[12:30](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=750s) So now, every shortcut is gone, no looking up solutions, no brittle unit tests or faking UI.

[12:38](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=758s) So when we force the agents to actually engineer an Android app, what do the numbers look like?

[12:44](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=764s) So, as of Monday, maybe you saw the blog post, the numbers look like this.

[12:50](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=770s) Even the best agent, passes only around 40% of our tasks, and most frontier agents land between 20 to 40%.

[13:00](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=780s) And that's good for us as the benchmark authors, because this strikes a right balance, like, there's a real world value of the tasks that we provided, but also there's a plenty of headroom to grow.

[13:13](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=793s) So, let's take a look at the results.

[13:16](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=796s) We can now take a look at the results from multiple perspectives.

[13:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=799s) We can take a look at some of the insights that we already found.

[13:23](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=803s) The first question that you might have when looking at the chart is, like, how much does it cost to run that agent?

[13:30](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=810s) And the AI companies usually report cost per token, for some for input, output, maybe cache token.

[13:37](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=817s) And maybe that's a decision-making argument in your case.

[13:43](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=823s) But we can take a look at it from other perspective.

[13:46](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=826s) We can plot the same chart on 2D space with mean cost per task, not the-- not token, so how much cost was it per the task versus the pass rate.

[13:58](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=838s) And here, we can suddenly see that some models are much more talkative.

[14:03](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=843s) They spend way more time or tokens, per task, while other, pass more tasks while costing less.

[14:14](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=854s) But if we also add the price per token to that same chart, we can see that, what AI companies really advertise doesn't match what you really pay, per the task.

[14:26](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=866s) So, if you can see that the agent with the best, best pass rate is actually one of the cheapest per token, yet most expensive per the task.

[14:35](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=875s) Or other agents with the exact same token price passes around 30% of the tasks, but they are 15 times cheaper.

[14:43](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=883s) So, the token price is not the metric.

[14:48](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=888s) Some of them are models are exploring the code more, they're rebuilding, rerunning, and every step burns more tokens.

[14:54](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=894s) So, the takeaway is don't only look at the price, per, cost per token.

[15:03](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=903s) Pick a model, look at the real data, like either use it in your case, or look at the Android Bench to see like which models actually spend the most task most, tokens on the tasks.

[15:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=919s) Some are, you know, some needs more steps although they are cheaper per token.

[15:25](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=925s) But, there's an additional aspect to the same question.

[15:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=929s) This is the same chart, but we can also take a look and show how long it takes the agents to complete that.

[15:36](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=936s) And we can see that the agents with the highest pass rate, they might be even the slowest.

[15:44](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=944s) Like, we can see that some they take almost two hours per the task.

[15:47](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=947s) They keep rebuilding, retesting until it works.

[15:50](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=950s) Yes, they get the best pass rate, but they also take the most time.

[15:54](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=954s) Some cheaper agents, they still take around half an hour and a half per the task, but without better results.

[16:00](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=960s) Or, other agents, they finish it in under 20 minutes and still solve more than quarter of the tasks.

[16:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=968s) So, small, lightweight models, they usually are faster and cheaper, but fail more often on those hard tasks.

[16:16](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=976s) The takeaway is like there's no single best model.

[16:20](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=980s) It depends on what you optimize for.

[16:22](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=982s) Is it pass rate?

[16:23](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=983s) Is it cost?

[16:24](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=984s) Is it wait time?

[16:26](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=986s) You might, in that case, depending on the type of work you're doing, if you're not working on such hard tasks, consider using the faster models that get the job done cheaper and faster.

[16:38](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=998s) But, the cost and speed tells you only how agents run, but not why they fail most of the trials.

[16:45](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1005s) So, we inspected the failing runs, and we noticed that agents often quit early.

[16:50](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1010s) They're convinced they're done.

[16:51](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1011s) Task complete.

[16:52](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1012s) Okay, I go home.

[16:53](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1013s) But, as mentioned, in the task packet, agents receive only the instructions and the before state.

[16:59](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1019s) They can't see the tests.

[17:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1021s) They see they can't see the verification suite.

[17:04](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1024s) So, we asked, what if, what happens if we just let them see the tests?

[17:07](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1027s) So, we experimented and we the had like a open test variant of the same task, where the agent got the failing verification test suite up front.

[17:18](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1038s) So, now, they were running the tests iteratively.

[17:21](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1041s) They can inspect errors, refine codes in a feedback loop.

[17:26](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1046s) And we found that when agents see the tests, they try harder and work longer.

[17:32](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1052s) And in the open runs, they had 25% more turns and 40% agent runtime.

[17:40](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1060s) Almost 30 minutes per task, they spent running and fixing the failing tests, because they weren't debugging blindly.

[17:48](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1068s) So, they see exactly what fails, and unfortunately the pass rate also jumps almost 50 percentage points.

[17:57](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1077s) And this extra effort improves the average pass rate, as I mentioned.

[18:03](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1083s) But eventually new frontier models came, and they quickly saturated this open variant.

[18:10](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1090s) Well, almost 90% of done of pass rate.

[18:13](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1093s) So, we kept the closed search closed test variant as the primary benchmark metric.

[18:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1099s) But, it's still very interesting to see how models are actually better when they know what they're graded against.

### How models test their own work

**Tomáš Mlynarič**

[18:27](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1107s) However, it allowed us to build more insights around the agents.

[18:33](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1113s) For example, we can check how models actually test their own work.

[18:37](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1117s) So, we can see from the chart that the frontier models write more and more better more good tests, and they have higher pass rates.

[18:46](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1126s) Some other models, they're overconfident, I can do it, no tests.

[18:50](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1130s) Well, sometimes they're still good.

[18:52](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1132s) So, if if you use the models that aren't that good with testing, you can help them with either better testing skills, or specific instructions on how to write tests.

[19:03](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1143s) But correlation doesn't imply causation, CS, right?

[19:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1148s) We need to have controlled experiment actually on this.

[19:11](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1151s) We still need to do, on whether prompting for better tests actually boosts the score.

[19:17](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1157s) We did however run an experiment on the other half of Android testing, and that is getting the agents to use emulator.

[19:27](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1167s) Early on, in our benchmark, we said nothing about about the emulator.

[19:31](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1171s) We assumed, like, agents should figure it out, right?

[19:35](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1175s) So, we saw, as results, almost none of them actually boots the booted the emulator.

[19:42](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1182s) And as I mentioned, our UI tests are pretty crucial in our benchmarks, and without emulators, you can't really use the UI tests.

[19:50](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1190s) So, they were all failing on very simple thing.

[19:55](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1195s) So, now we face the dilemma.

[19:57](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1197s) Do we tell the agents to You need to use the emulator, and risk actually higher scores, or do we say nothing, and we like hopefully they, you know, figure it out?

[20:08](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1208s) But after all, our goal was to be a fair benchmark measuring Android coding and problem-solving.

[20:16](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1216s) We not want to measure like some environmental quirks.

[20:19](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1219s) So, we chose a middle ground.

[20:22](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1222s) We told them this is how you start an emulator, but whether to use it or not, we didn't mention anything.

[20:31](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1231s) After all, starting the emulator isn't what we actually need to test for or want to test for.

[20:36](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1236s) And we measured the effect of that emulator instruction.

[20:41](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1241s) And now, we actually saw that when we gave them the how-to, the agents actually booted the emulator so much more and actually ran the UI tests, and that had big impact on the pass rate, unfortunately for us.

[20:55](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1255s) But also it let us inspect their real coding mistakes and problems and not just, like, failing on the unbooted emulator.

[21:04](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1264s) So, to take away, prompt specificity still matters quite a lot, and if you're designing your own benchmarks or evals, you need to test what actually matters in your case.

[21:17](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1277s) Like, let's give an example: you're evaluating token efficiency or a of a skill or a tool, first, you need to actually start with the eval passing, make the change for the efficiency, and then still verify it's still passing with fewer tokens now.

[21:33](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1293s) If you do it otherwise, you'll have too many variables changing all at once, and you wouldn't know whether that's actually your change made it better.

### Pass rates versus completion rates

**Tomáš Mlynarič**

[21:41](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1301s) Back to the leaderboard.

[21:43](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1303s) As I mentioned, best models pass around 40% of the tasks, and the pass rate requires all of the checks to pass.

[21:52](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1312s) And it would Otherwise, the failed the task.

[21:56](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1316s) So, the question is, did the agents produce nothing useful for the other 60%?

[22:03](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1323s) Nah, they're getting there.

[22:05](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1325s) To show how close the agent actually got to the working solutions, kind of like a distance, we also calculate completion rate or kind of like a partial score.

[22:15](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1335s) This is a weighted score across four categories: for visual consistency, functional, whether it functions properly, some per-task-specific requirements, and some regression checks.

[22:26](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1346s) And you can see that the lower ranked models are able to finish about half of the work while the frontier models are able to complete and get to almost 80% on average, more than 80% of the tasks, yet they only pass 40 40%.

[22:46](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1366s) So, we did an experiment on the Flutter to Android conversion to check like what's going on.

[22:51](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1371s) And so we recal- calculated the completion rate every few agent steps, and we saw the same patterns across all of the frontier models.

[23:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1381s) Around 140 steps, they were working to reach almost 90% score on average, and then they spent more 90 more steps or almost 40% of the run trying to close the gap to the 100%, but they were only able to score like 1 or 2% more.

[23:22](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1402s) And we think this actually mirrors what the agentic development today, where inexperienced developers or users, they see the app is complete.

[23:31](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1411s) Everything is rendering.

[23:33](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1413s) The input kind of worked.

[23:34](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1414s) There are maybe no crashes.

[23:36](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1416s) But when then a senior engineer comes and looks at the code, they're like they see so many multiple flaws: they see architectural problems, life cycle flaws, and stuff like that.

### Benchmark comparison and wrap-up

**Tomáš Mlynarič**

[23:49](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1429s) To wrap up, let's put Android Bench, our work 2.0, into perspective.

[23:54](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1434s) Comparing it to 1.0, we now only have 30 tasks, and that's deliberate.

[24:00](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1440s) Instead of isolated bug fixes and small patches, we have complete screens, multi-module migrations, whole app conversions.

[24:10](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1450s) Patches are now order of magnitude larger across dozens of file.

[24:14](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1454s) And we have the multimodal UI checks.

[24:17](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1457s) The agents must run longer and work much harder.

[24:22](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1462s) If we compare to other frontier benchmarks, we can say still see that our benchmark requires much larger code changes.

[24:30](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1470s) It's contamination-free because most of the tasks are written from the scratch, and there's still plenty of headroom to grow.

[24:37](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1477s) As I said at the start that in a few months in AI world feels like an eternity, and eventually, every benchmark gets maxed out and gets saturated.

[24:50](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1490s) And when this one does, hopefully, we'll push towards even harder and more complex tasks that will again resonate with that era with that state of AI.

[25:01](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1501s) Now, use Android Bench as compass to compare models and harnesses and maybe learn some insights.

[25:09](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1509s) And when you're ready to go deeper, bring the same benchmarking principles and eval principles to your own code bases and workflows because nothing beats measuring AI on the code you actually work with.

[25:23](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1523s) That's how you go from hoping AI got it right to knowing it did.

[25:29](https://www.youtube.com/watch?v=rIIAwylLb5Q&t=1529s) On behalf of whole Android Bench, thank you for your attention.