Hero benchmarks

Hero benchmarks are a set of benchmarks that cover high-level app user journeys, such as app startup or scrolling in the open-source Pokedex app.

• Startup hero benchmarks: Tracking the time from opening the app to content being displayed.
• Scroll hero benchmarks: Measuring scrolling performance of a lazy grid with images.

Test setup

• App selection: We conducted these tests using the open-source Pokedex app, comparing its View-based and Compose-based versions. The app was selected to represent real-world scenarios, and was not developed by the Compose team.
• Hardware setup: Benchmarks were run on a Pixel 6 running Android 17 (API 37) with locked CPUs and GPUs. Prior to Compose 1.12, benchmarks were conducted on a Pixel 3a. As of Compose 1.12, Pixel 6 is used as the standard performance baseline for hero benchmarks.
• Build setup: The benchmarks use the app built in release mode with R8 enabled. The app is fully pre-compiled to reduce instability from just-in-time (JIT) compilation.
• Result selection: To ensure worst-case analysis, extreme outliers and non-representative noise were excluded from the benchmark results. A baseline was then established by selecting the median values of the upper-bound performance data

Startup hero benchmarks

To measure startup, we used PokedexStartupBenchmark. The PokedexStartupBenchmark measures the time it takes for the Pokedex app to be visible to the user.

App launch can take place in one of three states: cold start, warm start, or hot start. Each state affects how long it takes for an app to become visible to the user. In a cold start, the app starts from scratch. In this benchmark, we measure the app launch performance in cold start state. We recommend that you always optimize the app based on an assumption of a cold start. Doing so can improve the performance of warm and hot starts as well.

The metrics we use to measure the app startup time are time to initial display and the time to full display.

Time to initial display

Time to initial display (TTID) measures the time it takes for an app to produce its first frame, including process initialization during a cold start, activity creation during a cold or warm start, and displaying the first frame.

In the PokedexStartupBenchmark, Compose 1.12 is on par with Views for TTID with a cold start.

Time to full display

Time to full display (TTFD) measures the time until the app becomes interactive. This means that all data has been loaded and has been drawn. For more information about time to full display, see App Startup Time.

In the PokedexStartupBenchmark, Compose 1.12 is approximately 2% slower than Views for TTFD with a cold start.

Scroll hero benchmarks

To measure the scrolling performance we used the PokedexScrollBenchmark. It measures the scroll performance of the Pokedex app screen for a lazy grid with images. In this benchmark, the entire screen is scrolled and flung multiple times and new items are loaded. Scroll performance is measured by jank rate.

Jank rate

Android devices render at up to 60 or 120 frames per second (fps). This means the system has a strict deadline (for example, 16.6ms for 60fps) to produce a frame. If the app takes too long to do its work, this can cause a visible stutter that we call jank.

In the PokedexScrollBenchmark, Compose matches Views performance with 0 percentage points of excess jank since Compose 1.9.0. On Pixel 6 (tested starting with Compose 1.11), both Compose and Views achieve a 0% jank rate.

Run hero benchmarks

To validate the performance results and run the benchmarks locally:

1. Follow the AndroidX guide to check out the source code.
2. Follow the instructions in the Hero Benchmarks source code README to build the benchmark and target APKs (pokedex-macrobenchmark and pokedex-macrobenchmark-target), install them on a test device, and run the PokedexStartupBenchmark and PokedexScrollBenchmark suites using adb or Android Studio.

You can run these benchmarks on any supported device, such as a specific device that you are optimizing your app for. However, to validate the official results, use the same hardware setup.