Automate measurement with the Macrobenchmark library

Using Jetpack Macrobenchmark is the recommended way to test how an app performs when Baseline Profiles are enabled, and then compare those results to a benchmark with Baseline Profiles disabled. This way, you can measure app startup time (time to initial and full display), or runtime rendering performance (to see if the frames produced can cause jank).

Macrobenchmarks let you control pre-measurement compilation via the CompilationMode API. To measure the results, you need to set the compilationMode parameter to the correct value as shown in the following snippet:

@RunWith(AndroidJUnit4ClassRunner::class)
class ColdStartupBenchmark {
    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun startupNoCompilation() = startup(CompilationMode.None())

    @Test
    fun startupBaselineProfile() = startup(CompilationMode.Partial())

    @Test
    fun startupFullCompilation() = startup(CompilationMode.Full())

    private fun startup(compilationMode: CompilationMode) = benchmarkRule.measureRepeated(
        packageName = "com.example.macrobenchmark.target",
        metrics = listOf(StartupTimingMetric()),
        compilationMode = compilationMode,
        iterations = 10,
        startupMode = StartupMode.COLD,
        setupBlock = {
            pressHome()
        }
    ) {
        // Waits for the first rendered frame, which represents time to initial display.
        startActivityAndWait()

        // Waits for content to be visible, which represents time to fully drawn.
        device.wait(Until.hasObject(By.res("my-content")), 5_000)
    }
}

You can see the results directly in Android Studio as shown in the following screenshot for Now in Android sample app ran on Google Pixel 7. From the results you can see that app startup was fastest when using Baseline Profiles (275.1ms) in contrast with no compilation (378.6ms). Note, that full AOT compilation can take even longer (393ms), because the system needs to load bigger file from disk.

Note, while the previous example shows app startup results captured with StartupTimingMetric, there are other important metrics worth considering, like FrameTimingMetric. You can get more information on all the types of metrics on Capture the metrics.