Introduction to the Jetpack Benchmark library

The Jetpack Benchmark library allows you to quickly benchmark your Kotlin-based or Java-based code from within Android Studio. The library handles warmup, measures your code performance and allocation counts, and outputs benchmarking results to both the Android Studio console and a JSON file with more detail.

Use cases include scrolling a RecyclerView, performing database queries, and measuring any slow part of your code you want to make faster.

You can use the library in a continuous integration (CI) environment, as described in Run benchmarks in Continuous Integration.

If you haven't yet adopted AndroidX in a project you want to benchmark, see Migrate an existing project using Android Studio.

Quickstart

This section provides a quick set of steps to try out benchmarking without requiring you to move code into modules. Because the steps involve disabling debugging for accurate performance results, you won't commit the changes to your source control system, but it can still be helpful when you want to run one-off measurements.

To quickly perform one-off benchmarking, do the following:

  1. Add the library to your module’s build.gradle file:

    project_root/module_dir/build.gradle

    Groovy

    buildscript {
...
dependencies {
    ...
    classpath "androidx.benchmark:benchmark-gradle-plugin:1.0.0"
}
}

    Kotlin

    dependencies {
    androidTestImplementation("androidx.benchmark:benchmark-junit4:1.0.0")
}

    Groovy

    dependencies {
    androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0"
}

  2. To disable debugging in the test manifest, update your <application> element to force-disable debugging temporarily as follows:

    project_root/module_dir/src/androidTest/AndroidManifest.xml 

    <!-- Important: disable debuggable for accurate performance results -->
<application
    android:debuggable="false"
    tools:ignore="HardcodedDebugMode"
    tools:replace="android:debuggable"/>

  3. To add your benchmark, add an instance of BenchmarkRule in a test file in the androidTest directory. For more information on writing benchmarks, see Write a benchmark.

    The following code snippet shows how to add a benchmark to a JUnit test:

    Kotlin

    @RunWith(AndroidJUnit4::class)
class MyBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun benchmarkSomeWork() = benchmarkRule.measureRepeated {
        doSomeWork()
    }
}

    Java

    @RunWith(AndroidJUnit4.class)
class MyBenchmark {
    @Rule
    public BenchmarkRule benchmarkRule = new BenchmarkRule();

    @Test
    public void benchmarkSomeWork() {
        final BenchmarkState state = benchmarkRule.getState();
        while (state.keepRunning()) {
            doSomeWork();
        }
    }
}

When to benchmark

It's recommended to profile your code before writing a benchmark. This helps you find expensive operations that are worth optimizing. It can also expose why the operations are slow by showing what is happening while they run. Examples could include running on a low-priority thread, sleeping due to disk access, or unexpectedly calling into a expensive function, like bitmap decoding.

Adding custom trace points via the TraceCompat API (or the -ktx wrapper) allows you to view them in the Android Studio CPU Profiler or Systrace:

Kotlin

fun proccessBitmap(bitmap: Bitmap): Bitmap {
    trace("processBitmap") {
        // perform work on bitmap...
    }
}

Java

public Bitmap processBitmap(Bitmap bitmaptest) {
    TraceCompat.beginSection("processBitmap");
    try {
        // perform work on bitmap...
    } finally {
        TraceCompat.endSection();
    }
}

What to benchmark

Benchmarks are most useful for CPU work that is run many times in your app. Good examples are RecyclerView scrolling, data conversions/processing, and pieces of code that get used repeatedly.

Other types of code are more difficult to measure with benchmarking. Because benchmarks run in a loop, any code that isn't run frequently, or performs differently when called multiple times, may not be a good fit for benchmarking.

Caching

Try to avoid measuring just the cache. For example, a custom view's layout benchmark might measure only the performance of the layout cache. To avoid this, you can pass different layout parameters in each loop. In other cases, such as when measuring file system performance, this may be difficult because the OS caches the file system while in a loop.

Infrequently-run code

Code that's run once during application startup is not very likely to get JIT compiled by Android Runtime (ART). Because of that, benchmarking this code as it runs in a loop isn't a realistic way to measure its performance.

For this sort of code, we recommend tracing or profiling the code in your app, instead. Note that this doesn't mean you can't benchmark code in your startup path, but rather that you should pick code that's run in a loop, and that is likely to get JIT compiled.

Full project setup

To set up benchmarking for regular benchmarking rather than one-off benchmarking, you isolate benchmarks into their own module. This ensures that their configuration, such as setting debuggable to false, is separate from regular tests.

Before adding your Benchmark module, place code and resources you want to benchmark into a library module if they aren't already in one.

Our samples give examples of how to set up a project in this way.

Set Android Studio properties

If you are using Android Studio 3.5, you must manually set Android Studio properties to enable benchmark module wizard support. This is not required for Android Studio 3.6 or higher.

To enable the Android Studio template for benchmarking, do the following:

  1. In Android Studio 3.5, click Help > Edit Custom Properties.

  2. Add the following line to the file that opens:

    npw.benchmark.template.module=true

  3. Save and close the file.

  4. Restart Android Studio.

Create a new module

The benchmarking module template automatically configures settings for benchmarking.

To use the module template to create a new module, do the following:

  1. Right-click your project or module and select New > Module.

  2. Select Benchmark Module and click Next.

    Figure 1. Benchmark module

  3. Enter a module name, choose the language, and click Finish.

    A module is created that is pre-configured for benchmarking, with a benchmark directory added and debuggable set to false.

Write a benchmark

Benchmarks are standard instrumentation tests. To create a benchmark, use the BenchmarkRule class provided by the library. To benchmark activities, use ActivityTestRule or ActivityScenarioRule. To benchmark UI code, use @UiThreadTest.

The following code shows a sample benchmark:

Kotlin

@RunWith(AndroidJUnit4::class)
class ViewBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun simpleViewInflate() {
        val context = ApplicationProvider.getApplicationContext<Context>()
        val inflater = LayoutInflater.from(context)
        val root = FrameLayout(context)

        benchmarkRule.keepRunning {
            inflater.inflate(R.layout.test_simple_view, root, false)
        }
    }
}

Java

@RunWith(AndroidJUnit4::class)
public class ViewBenchmark {
    @Rule
    public BenchmarkRule benchmarkRule = new BenchmarkRule();

    @Test
    public void simpleViewInflate() {
        Context context = ApplicationProvider.getApplicationContext<Context>();
        final BenchmarkState state = benchmarkRule.getState();
        LayoutInflater inflater = LayoutInflater.from(context);
        FrameLayout root = new FrameLayout(context);

        while (state.keepRunning()) {
            inflater.inflate(R.layout.test_simple_view, root, false);
        }
    }
}

You can disable timing for sections of code you don't want to measure, as shown in the following code sample:

Kotlin

@Test
fun bitmapProcessing() = benchmarkRule.measureRepeated {
    val input: Bitmap = runWithTimingDisabled { constructTestBitmap() }
    processBitmap(input)
}

Java

@Test
public void bitmapProcessing() {
    final BenchmarkState state = benchmarkRule.getState();
    while (state.keepRunning()) {
        state.pauseTiming();
        Bitmap input = constructTestBitmap();
        state.resumeTiming();

        processBitmap(input);
    }
}

For information on running the benchmark, see Run the benchmark.

Run the benchmark

In Android Studio, run your benchmark as you would any @Test. On Android Studio 3.4 and higher, you can see output sent to the console.

To run the benchmark, in the module, navigate to benchmark/src/androidTest and press Control+Shift+F10 (Command+Shift+R on Mac). Results of the benchmark appear in the console, as shown in Figure 2:

Benchmarking output in Android Studio

Figure 2. Benchmarking output in Android Studio

From the command line, run the regular connectedCheck:

./gradlew benchmark:connectedCheck

Collect the data

A full benchmark report with additional metrics and device information is available in JSON format.

The androidx.benchmark Gradle plugin enables JSON output by default. To manually enable JSON output in non-Gradle build environments, you pass an instrumentation argument, androidx.benchmark.output.enable, set to true.

The following is an example using the adb shell am instrument command:

adb shell am instrument -w -e "androidx.benchmark.output.enable" "true" com.android.foo/androidx.benchmark.junit4.AndroidBenchmarkRunner

By default, the JSON report is written to disk on device in the test APK's external shared downloads folder, which is typically located at:

/storage/emulated/0/Download/app_id-benchmarkData.json

You can configure the location where benchmark reports are saved on the device using the instrumentation argument additionalTestOutputDir.

adb shell am instrument -w -e "androidx.benchmark.output.enable" "true" -e "additionalTestOutputDir" "/path_to_a_directory_on_device_test_has_write_permissions_to/" com.android.foo/androidx.benchmark.junit4.AndroidBenchmarkRunner

JSON reports are copied from device to host. These are written on the host machine at:

project_root/module/build/outputs/connected_android_test_additional_output/debugAndroidTest/connected/device_id/app_id-benchmarkData.json

Android Gradle Plugin 3.6 and higher

When running benchmarks from the command line with Gradle, projects using Android Gradle Plugin 3.6 and higher can add the following flag to the project's gradle.properties file:

android.enableAdditionalTestOutput=true

This enables an experimental Android Gradle Plugin feature to pull benchmark reports from devices running API 16 and higher.

Android Gradle Plugin 3.5 and lower

For users of older versions of Android Gradle Plugin, the androidx.benchmark Gradle plugin handles copying JSON benchmark reports from device to host.

When using AGP 3.5 or lower and targeting API level 29 or higher, you need to add a flag to your Android manifest in the androidTest directory of your benchmarks to enable legacy external storage behavior.

<manifest ... >
  <!-- This attribute is "false" by default on apps targeting Android 10. -->
  <application android:requestLegacyExternalStorage="true" ... >
    ...
  </application>
</manifest>

For more information, see Temporarily opt-out of scoped storage.

Clock stability

Clocks on mobile devices dynamically change from high state (for performance) to low state (to save power, or when the device gets hot). These varying clocks can make your benchmark numbers vary widely, so the library provides ways to deal with this issue.

Lock clocks (requires root)

Locking clocks is the best way to get stable performance. It ensures that clocks never get high enough to heat up the device, or low if a benchmark isn't fully utilizing the CPU. While this is the best way to ensure stable performance, it isn't supported on most devices, due to requiring adb root.

To lock your clocks, add the supplied helper plugin to the top level project’s classpath in the main build.gradle file:

buildscript {
    ...
    dependencies {
        ...
        classpath("androidx.benchmark:benchmark-gradle-plugin:1.0.0")
    }
}

Apply the plugin in the build.gradle of your benchmark module:

Groovy

apply plugin: com.android.app
apply plugin: androidx.benchmark
...

Kotlin

plugins {
    id("com.android.app")
    id("androidx.benchmark")
    ...
}

This adds benchmarking Gradle tasks to your project, including ./gradlew lockClocks and ./gradlew unlockClocks. Use these tasks to lock and unlock a device’s CPU using adb.

If you have multiple devices visible to adb, use the environment variable ANDROID_SERIAL to specify which device the Gradle task should operate on:

ANDROID_SERIAL=device-id-from-adb-devices ./gradlew lockClocks

Sustained performance mode

Window.setSustainedPerformanceMode() is a feature supported by some devices that enables an app to opt for a lower max CPU frequency. When running on supported devices, the Benchmark library uses a combination of this API and launching its own activity to both prevent thermal throttling and stabilize results.

This functionality is enabled by default by the testInstrumentationRunner set by the Gradle plugin. If you want to use a custom runner, you can subclass the AndroidBenchmarkRunner and use it as your testInstrumentationRunner.

The runner launches an opaque, fullscreen activity to ensure that the benchmark runs in the foreground and without any other app drawing.

Automatic execution pausing

If neither clock-locking nor sustained performance are used, the library performs automatic thermal throttling detection. When enabled, the internal benchmark periodically runs to determine when the device temperature has gotten high enough to lower CPU performance. When lowered CPU performance is detected, the library pauses execution to let the device cool down, and retries the current benchmark.

Configuration Errors

The library detects the following conditions to ensure your project and environment are set up for release-accurate performance:

  • Debuggable is set to false.
  • A physical device, not an emulator, is being used.
  • Clocks are locked if the device is rooted.
  • Sufficient battery level on device.

If any of the above checks fail, the benchmark will throw an error to discourage inaccurate measurements.

To suppress these errors as warnings, and prevent them from throwing an error and halting the benchmark, pass the type of errors you want to suppress in a comma-separated list to the instrumentation argument androidx.benchmark.suppressErrors:

adb shell am instrument -w -e "androidx.benchmark.suppressErrors" "DEBUGGABLE" com.android.foo/androidx.benchmark.junit4.AndroidBenchmarkRunner

Which you may set in Gradle as follows:

Groovy

android {
    defaultConfig {
        testInstrumentationRunnerArgument 'androidx.benchmark.suppressErrors', 'DEBUGGABLE'
    }
}

Kotlin

android {
    defaultConfig {
        testInstrumentationRunnerArguments(mapOf(
            "androidx.benchmark.suppressErrors" to "DEBUGGABLE"
        ))
    }
}

Note that suppressing errors allows the benchmark to run in an incorrectly configured state, but the output of the benchmark will be intentionally corrupted by prepending test names with the error. That is, running a debuggable benchmark with the above suppression will prepend test names with DEBUGGABLE_.

Profiling

You can profile a benchmark to investigate why the measured code is running slowly.

Add the following to your benchmark module's build.gradle file:

Groovy

android {
    defaultConfig {
        // must be one of: none, sampled, or method
        testInstrumentationRunnerArgument 'androidx.benchmark.profiling.mode', 'sampled'
    }
}

Kotlin

android {
    defaultConfig {
        // must be one of: none, sampled, or method
        testInstrumentationRunnerArgument = mapOf(
            "androidx.benchmark.profiling.mode", "sampled"
        )
    }
}

When you run a benchmark, an output .trace file will be copied to host, in the directory alongside JSON results. Open this file with Android Studio using File > Open to inspect profiling results in the CPU Profiler.

Method tracing

With method tracing, the benchmark will warm up before capturing a method trace, recording every method called by your benchmark. Note that performance results will be significantly affected by the overhead of capturing each method entry/exit.

Method profiling

With method sampling, the benchmark will sample stack traces at a 100 microsecond interval, after warmup has completed. The benchmark will loop significantly longer than usual to ensure enough samples are captured for meaningful results, and measurement results will be somewhat affected.

To read more about how to use method tracing and sampling, see the CPU profiling configuration.

Benchmarking samples

Sample benchmark code is available in the following projects:

The sample projects include:

  • BenchmarkSample: this is a standalone sample, which shows how to use a benchmark module to measure code and UI.

  • PagingWithNetworkSample: Android Architecture Components sample, which shows how to benchmark RecyclerView performance.

  • WorkManagerSample: Android Architecture Components sample, which shows how to benchmark WorkManager workers.

Additional resources

To learn more about Benchmarking, consult the following additional resources.

Blogs

Provide feedback

To report issues or submit feature requests when using benchmarking, see the public issue tracker.