Android Bench | Android Developers

Android Bench 2.0 methodology

AI is helping you tackle more ambitious engineering tasks, giving you more time to focus on building high-quality Android apps. Android Bench 2.0 raises the bar of our evaluations to match this ambition, helping you find the models and coding agents best suited to your development workflow.

Evolving Android Bench

As you delegate more complex work to AI, you need evaluations that reflect actual mobile engineering challenges rather than incremental changes.

Goals of the benchmark

Android Bench provides a standardized, model-agnostic evaluation suite for Android development. Our goals are:

Encourage improvements in AI models and agent harnesses for Android development.
Enable you to evaluate and select the most capable AI tools for your development stack.
Help improve overall app quality across the Android ecosystem by promoting architectural best practices.

Limitations of 1.0

Android Bench 1.0 evaluated models on localized GitHub pull requests. It established an empirical baseline, but model capabilities outpaced it:

Benchmark saturation: Frontier models reached a ~90% pass rate, saturating localized bug-fixing tests.
Narrow scope: Tasks averaged a median change of 32 lines of code across 1 to 2 files. This tested small patches rather than architectural work.
Lack of scoring nuance: Binary pass or fail scoring hid incremental progress on complex, multi-step tasks.
Headless testing: Verifications ran in terminal environments without validating visual rendering, layout fidelity, or user interaction lifecycles.

Goals of 2.0

Android Bench 2.0 addresses these gaps by evaluating multi-day mobile engineering workflows:

Complex scale	Tests tasks that typically take mid-level to senior developers days or weeks to complete.
Hill-climbing signal	Provides a continuous completion rate (0.0 to 1.0) so model creators can measure incremental progress on failing tasks.
Contamination resistance	Uses a private codebase and migrations that don't exist in the source repositories, so models can't retrieve a memorized solution.
Real-world engineering	Mirrors the work you do daily: building apps from scratch, adding new features, refactoring architecture, and migrating dependencies.

Benchmark design and taskset

Android Bench 2.0 features 30 long-horizon tasks across four distinct engineering streams.

Taskset composition

Tasks reflect common architectural and platform challenges across the Android ecosystem:

Stream	Tasks	Focus	Typical scope
App creation	9	Creating and developing a complex new app: Food Vibes*, which is a private multi-screen food delivery app, from visual design mocks.	1,200–5,500 lines across 20–70 files
Migrations	13	Library and architecture migrations: Upgrading production apps to modern libraries (Retrofit to Ktor, RxJava to Coroutines, Hilt to Koin, Navigation 2 to Navigation 3).	200–8,200 lines across 5–294 files
New features	6	Implementing platform capabilities in established codebases (Picture-in-Picture, Wear OS companion sync, Home screen widgets, CameraX).	400–2,200 lines across 4–60 files
App conversions	2	Converting cross-platform apps (Flutter, React Native) to native Android with Jetpack Compose.	Full app UI, navigation, and persistence

Preventing contamination

To make sure the benchmark measures reasoning rather than memorized solutions, Android Bench 2.0 applies four safeguards:

A private codebase: Greenfield tasks build Food Vibes, an internal app with no public presence.
Migrations that don't exist upstream: Tasks migrate production apps to recently stabilized libraries, such as Navigation 3, Coil 3, and Ktor 3, where the source repository has no migration to copy.
Conversions with no native counterpart: App conversion tasks translate Flutter and React Native projects that have never been built for native Android.
Trajectory audits: We audit agent execution steps to detect reward hacking, hardcoded outputs, and external code lookups.

The dataset is private. We are evaluating how best to make it available without resulting in contamination.

Evaluation and verification

Android Bench 2.0 evaluates coding agents within containerized virtual device environments.

The Harbor framework

Android Bench builds on the Harbor framework to standardize environment configuration, container isolation, and metrics collection.

Insights retained from Harbor

Our standardization on Harbor continues to provide core benefits in 2.0:

Containerized isolation: Runs every task in a fresh Docker container to isolate builds and execution state.
Hardware virtualization: Uses KVM-enabled CPUs to run hardware-accelerated Android Virtual Devices (AVDs), with a minimum of 16 CPUs, 72 GB RAM, and 500 GB storage.
Native tool calling: Requires models to invoke shell commands through structured API tool calls rather than plain Markdown text blocks.

Expanding beyond mini-swe-agent

Android Bench 2.0 broadens evaluations from single-agent setups to modern coding agents, including Claude Code, Codex, and Antigravity SDK. This expansion shows how harness design impacts developer outcomes.

Runs per task

To account for non-deterministic model behavior, Android Bench executes 5 independent runs per task, calculating averages across completed runs.

Multifaceted verification

Verifying open-ended mobile applications requires testing runtime logic and visual presentation without restricting the agent's implementation choices.

The following table lists the components of the multifaceted verification engine:

Deterministic verification	Multimodal verification
Instrumentation test assertions	Live AVD UI navigation driver
SQLite and Room database inspection	Gemini 3.5 Flash visual judge
Outbound intent and event checks	Accessibility UI tree parser
Pass-to-pass regression suite	Anti-cheating patch inspection

Deterministic verification

We use deterministic assertions to evaluate functional requirements without LLM involvement:

Instrumentation assertions: Automated Android instrumentation tests verify UI interactions and state transitions. We avoid unit tests because they couple too closely to internal class structures.
Database verification: Test drivers inspect SQLite and Room database tables directly to confirm data persistence, regardless of the agent's internal data-access patterns.
System boundaries: Verifiers monitor outbound Intent extras, network calls, and Wear OS synchronization events.
Regression suites: Pre-existing repository test suites run alongside new feature checks to confirm existing capabilities remain intact.

Multimodal verification

Traditional pixel diffing tools (such as Dropshots) fail in automated agent evaluation because minor system-level differences—such as status bar timestamps, battery icons, and font anti-aliasing—trigger up to 47.5% pixel diffs on semantically correct screens.

Android Bench replaces pixel matching with a multimodal verification engine:

Automated UI walkthrough: An emulator driver executes a scripted walkthrough of the application, capturing screen bitmaps and accessibility hierarchies.
Visual judgement: Gemini 3.5 Flash compares screen captures against baseline reference images, returning a score from 0.0 to 1.0 with structured technical reasoning. In calibration trials across 360 runs, Gemini 3.5 Flash achieved 100% consistency across repeated runs (Diff = 0.00), ignoring system-level noise while penalizing missing layout components.
Accessibility tree judgement: Gemini 3.5 Flash parses accessibility hierarchy dumps (dumpsys accessibility) to verify that screens contain native composables, valid touch targets, and correct text labels rather than static image overlays.

Test design and review

Open-ended tasks only work as a benchmark if the tests accept any correct solution. Expert Android engineers reviewed every task specification and test suite across nine rounds, removing two failure modes:

Overspecification: Instructions that gave away step-by-step code structure, turning a design problem into transcription.
Underspecification: Tests that expected behavior the instructions never asked for.

Each round ran the benchmark against solutions from multiple model families, including Claude, GPT, and Gemini. When a model produced working code that failed only because a test was too rigid, we rewrote the test to check functionality instead. Reviewers also audited each suite to confirm an agent cannot pass it with hardcoded data or fake mocks.

Scoring methodology

Android Bench 2.0 reports two metrics: pass rate, which measures how often an agent fully solves a task, and completion rate, which measures how much of a task an agent completed.

Pass rate

Pass rate is the primary metric. It measures the percentage of runs where an agent achieves a perfect score (1.0) with all functional tests passing, full visual compliance, and zero constraint violations. A run only passes when the result is a complete, working solution to the task.

Completion rate

Completion rate supplements pass rate with a continuous score from 0.0 to 1.0, calculated by combining a weighted base score with multiplicative penalty constraints. It measures how much of a task an agent completed, including on runs that don't pass.

Why a continuous score is necessary

On multi-day engineering tasks, binary pass or fail grading fails to capture substantial technical progress.

For example, an agent might refactor 40 screens to Jetpack Compose, set up database tables, and satisfy 90% of requirements, but fail a single edge-case assertion. Pass rate scores that run 0%, the same as a run that never compiled, obscuring the model's architectural capabilities.

Completion rate separates those two outcomes, which provides a meaningful hill-climbing signal for model development on tasks no model passes yet.

Scoring formula

The final completion rate applies negative constraint multipliers to a positive base score:

$$ \text{CompletionRate} = \text{BaseScore} \times \text{Multipliers} $$

The base score is the normalized weighted sum of four evaluation dimensions:

$$ \text{BaseScore} = (w_f \cdot S_f) + (w_r \cdot S_r) + (w_q \cdot S_q) + (w_v \cdot S_v) $$

Component	Weight	Evaluation focus
Functional (S_f)	w_f	Validates runtime state mutations, database persistence, and core logic.
Regression (S_r)	w_r	Confirms existing unit and instrumentation tests continue to pass.
Requirements (S_q)	w_q	Assesses adherence to explicit task instructions, library versions, and architectural rules.
Visual (S_v)	w_v	Evaluates screen layout fidelity and accessibility hierarchy parity.

Task authors assign category weights per task. Architecture refactoring weights functional and regression tests heavily; UI creation weights visual fidelity higher. Weighting also keeps the main objective from being drowned out: in a repository with thousands of existing tests, an unweighted score would be dominated by regression checks the task never asked the agent to touch.

The following breakdown shows a Flutter-to-Compose conversion run scoring 0.8861. This task carries no regression component, because the target app has no existing Android test suite.

Category	Score	Weight	Normalized weight	Contribution
Visual	0.8291	1.0	0.6667	0.5528
Functional	1.0000	0.3	0.2000	0.2000
Requirements	1.0000	0.2	0.1333	0.1333
Base score				0.8861
Multipliers				1.0x
Completion rate				0.8861

Multipliers

Multipliers apply strict penalties when submissions violate core technical constraints:

Penalty condition	Multiplier	Rationale
Build failure	0.0x	Code that fails compilation cannot run or produce a working application.
Cheating violation	0.0x	The verifier detects deliberate shortcut exploits that bypass actual implementation.
Foreign language files	0.0x	The patch reuses Flutter, Dart, or JavaScript files in native Android tasks.
Legacy API usage	0.5x	The solution uses obsolete patterns (such as `findViewById` in Jetpack Compose tasks).

Failure and cheating

Android Bench categorizes execution errors and blocks shortcut attempts using standardized inspection tools.

Failure taxonomy

Task runs are categorized into a standardized failure taxonomy adapted from DeepSWE:

Classification	Failure mode	Description
Model error True failure	Build error	The project doesn't compile.
	Wrong file	The agent modifies an unrelated part of the codebase.
	Unverified assumption	The agent relies on an untested guess about how the codebase works.
	Knowledge gap	The agent misuses a language feature or calls an API that doesn't exist.
	Misunderstood task	The agent confidently solves a different problem.
	Missed requirement	The solution is close to correct but omits a crucial detail.
	Integration error	A function signature, type, or API shape disagrees with surrounding code.
	Wrong logic	The implementation fails happy-path validation.
	Regression	The solution breaks previously passing features.
	Model looping	The agent repeats the same actions or commands.
	Gives up	The agent stops, having decided the task is too hard.
Cheating False pass	Cheating	The agent bypasses validation or hardcodes expected output instead of implementing the feature.
Benchmark error False failure	Instruction needs changes	Instructions give away the answer, or are missing or misleading.
	Tests need changes	Tests expect hidden behavior, or aren't implementation-agnostic.
	Infrastructure error	Sandbox timeout, emulator failure, or API unavailability.

Cheating countermeasures

Advanced agents often attempt shortcuts when facing complex tasks. Android Bench applies automated countermeasures against observed exploit patterns:

Observed shortcut exploit	Agent mechanism	Benchmark countermeasure
Full-screen image overlays	Renders static screenshot images inside transparent composables.	Combines visual inspection with accessibility tree parsing across randomized screen sizes.
Hardcoded database state	Returns hardcoded test data from DAOs regardless of user input.	Injects randomized strings dynamically during test execution.
Test threshold tampering	Modifies screenshot comparison configs to raise diff tolerance.	Resets test files and validation scripts from a clean baseline before scoring.
Test assertion deletion	Comments out `@Test` methods or inserts early return statements.	Withholds test files from the agent workspace or restores them before verification.
Precompiled binary bundling	Links precompiled JARs to pass static checks without migrating source code.	Uses static AST checks and anti-cheating LLM inspection prompts.
API wrapping	Wraps deprecated APIs in thin adapters instead of performing a real migration.	Static AST checks and forbidden API penalty multipliers.

Learning from the benchmark results

Key takeaways include:

Building features vs. translating architectures: Across model tiers, AI excels at building new features and expanding existing apps. However, when converting cross-platform apps to Android, only frontier models manage the architectural translation complexity.
App conversions remain unsolved: No model has passed a full app conversion outright, though the best runs reach completion rates near 0.90. The remaining gap is rarely the bulk of the UI. It's dark theme, a missing secondary screen, or state that doesn't survive rotation.
Repetitive multi-file transformations score high: Language migrations, such as Signal from Java to Kotlin, and network layer migrations, such as PocketCasts from Retrofit to Ktor, score well. The transformation rule is the same in every file, so models apply it across 75 to 125 files without drifting.
Breaking framework changes cause failures: Migrating Bitwarden from Navigation 2 to Navigation 3, across 233 files, yielded a 0.409 completion rate and a 0.0% pass rate. Breaking API changes cause cascading compile errors, and models have little training exposure to the new APIs.
Missing runtime graph bindings trigger crashes: Dependency injection migrations (such as Hilt to Koin across 294 files) frequently compile successfully but fail at runtime due to missing dependency graph bindings.
Pattern-based refactors scale well: Introducing an MVVM ViewModel layer across 44 screens, 7,222 lines, follows a standard pattern. Models extract state, create factories, and expose StateFlow consistently across a whole codebase.
Fresh code scores higher than legacy code: Building new Compose interfaces in Food Vibes, 2,000 to 2,858 lines across 40 to 64 files, lets models generate clean composable trees without working around existing framework constraints.

Where models stall

Frontier models average completion rates above 0.80 while pass rates stay low. Trajectory replays show why: models reach roughly 80% of a task early, then flatline. On the Flutter conversion task, GPT-5.6 Sol reached 84% in 94 steps, then spent another 121 steps, more than half the run, to finish at 86%. It never added the statistics screen or dark theme support.

The implementations look finished. Screens render, inputs respond, and user journeys complete without crashing. What's left is the work a senior engineer catches in code review.

Task	Model	Score	What was missing
Now in Android: news widget	Fable 5.1, Opus 5	86%	Both models called `observeLatestNews().first()` outside `provideContent`, so the widget never updates when the data changes.
Now in Android: video playback	GPT Astra	88%	The player isn't released when it scrolls off screen and keeps playing in the background.
Signal: profile screen to Compose	GPT Astra	90%	State wasn't hoisted to the ViewModel, so rotation wipes user input.
Bitwarden: Navigation 3 upgrade	Opus 5	95%	200+ files refactored and all tests passing, but missing exit transitions cause a visual stutter.

Limitations and caveats

When interpreting benchmark results and comparing model efficiency across the leaderboard, keep the following technical boundaries and metric caveats in mind:

Platform and evaluation boundaries

Hardware virtualization: Tasks execute on containerized Android Virtual Devices. Hardware-dependent features—such as physical Bluetooth LE peripheral pairing, physical camera sensors, and specialized GPU compute shaders—rely on software mocks.
Form factor coverage: While 2.0 includes Wear OS companion tasks, future benchmark suites will expand coverage across Foldables, Large Screens, and Android Auto.
Linear UI walkthroughs: App conversion tests use deterministic UI walkthrough scripts. If an agent fails to render an initial navigation button, the test driver cannot reach downstream screens, resulting in a lower visual completion score.
Simulated backends: Tasks run against local mock servers. Results don't reflect how an agent handles intermittent network failures, slow responses, or backend errors.

Operational and leaderboard metrics caveats

Bias toward incomplete runs: Summing resource consumption (cost, tokens, latency) across all tasks creates an inherent structural bias toward models with higher failure rates. A model that fails early on multiple tasks will appear to have lower total cost and execution time than a model that works longer to solve them. These metrics measure gross resource consumption per attempt, not normalized efficiency. Always compare resource metrics between models with comparable completion and pass rates.
Environmental variability in latency: Latency measurements include network transit time to API provider endpoints. These values reflect network routing and client proximity to provider servers rather than pure model inference speed.
Dynamic pricing fluctuations: Cost calculations use provider API pricing active at the time of execution. Because pricing models change frequently across providers, costs recorded across different benchmark releases are not directly comparable.
Token optimization variance: Token counts reflect values reported by inference engines. These counts may not capture provider-side optimizations—such as prompt caching or shared system prompt prefix compression—unless explicitly reported in API usage metadata.