Benchmark app code

The Jetpack Benchmark library allows you to quickly benchmark your Kotlin-based or Java-based code from within Android Studio. The library handles warmup, measures your code performance, and outputs benchmarking results to the Android Studio console.

Use cases include scrolling a RecyclerView, inflating a non-trivial View hierarchy, and performing database queries.

If you haven't yet adopted AndroidX in a project you want to benchmark, see Migrate an existing project using Android Studio.

Quickstart

This section provides a quick set of steps to try out benchmarking without requiring you to move code into modules. Because the steps involve disabling debugging for accurate performance results, you won't commit the changes to your source control system, but it can still be helpful when you want to run one-off measurements.

To quickly perform one-off benchmarking, do the following:

Add the library to your module’s build.gradle file:

project_root/module_dir/build.gradle

dependencies {
    androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0-alpha04"
}

To disable debugging in the test manifest, update your <application> element to force-disable debugging temporarily as follows:

project_root/module_dir/src/androidTest/AndroidManifest.xml

<!-- Important: disable debuggable for accurate performance results -->
<application
    android:debuggable="false"
    tools:ignore="HardcodedDebugMode"
    tools:replace="android:debuggable"/>

To add your benchmark, add an instance of BenchmarkRule in a test file in the androidTest directory. For more information on writing benchmarks, see Write a benchmark.

The following code snippet shows how to add a benchmark to a JUnit test:

Kotlin

@RunWith(AndroidJUnit4::class)
class MyBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun benchmarkSomeWork() = benchmarkRule.measureRepeated {
        doSomeWork()
    }
}

Java

@RunWith(AndroidJUnit4.class)
class MyBenchmark {
    @Rule
    public BenchmarkRule benchmarkRule = new BenchmarkRule();

    @Test
    public void myBenchmark() {
        final BenchmarkState state = benchmarkRule.getState();
        while (state.keepRunning()) {
            doSomeWork();
        }
    }
}

What to benchmark

Benchmarks are most useful for CPU work that is run many times in your app. Good examples are RecyclerView scrolling, data conversions/processing, and pieces of code that get used repeatedly.

Other types of code are more difficult to measure with benchmarking. Because benchmarks run in a loop, any code that isn't run frequently, or performs differently when called multiple times, may not be a good fit for benchmarking.

Caching

Try to avoid measuring just the cache. For example, a custom view's layout benchmark might measure only the performance of the layout cache. To avoid this, you can pass different layout parameters in each loop. In other cases, such as when measuring file system performance, this may be difficult because the OS caches the file system while in a loop.

Infrequently-run code

Code that's run once during application startup is not very likely to get JIT compiled by Android Runtime (ART). Because of that, benchmarking this code as it runs in a loop isn't a realistic way to measure its performance.

For this sort of code, we recommend tracing or profiling the code in your app, instead. Note that this doesn't mean you can't benchmark code in your startup path, but rather that you should pick code that's run in a loop, and that is likely to get JIT compiled.

Full project setup

To set up benchmarking for regular benchmarking rather than one-off benchmarking, you isolate benchmarks into their own module. This ensures that their configuration, such as setting debuggable to false, is separate from regular tests.

To do this, you need to complete these tasks:

Place code and resources you want to benchmark into a library module if they aren't already in one.
Add a new library module to hold the benchmarks themselves.

Our samples give examples of how to set up a project in this way.

Set Android Studio properties

Jetpack Benchmark library is currently in Alpha, and requires manually setting Android Studio properties to enable benchmark module wizard support.

To enable the Android Studio template for benchmarking, do the following:

Download Android Studio 3.5 Beta 1 or newer.
In Android Studio, click Help > Edit Custom Properties.
Add the following line to the file that opens:

npw.benchmark.template.module=true
Save and close the file.
Restart Android Studio.

Create a new module

The benchmarking module template automatically configures settings for benchmarking.

To use the module template to create a new module, do the following:

Right-click your project or module and select New > Module.
Select Benchmark Module and click Next.

Figure 1. Benchmark module
Enter a module name, choose the language, and click Finish.

A module is created that is pre-configured for benchmarking, with a benchmark directory added and debuggable set to false.

Write a benchmark

Benchmarks are standard instrumentation tests. To create a benchmark, use the BenchmarkRule class provided by the library. To benchmark activities, use ActivityTestRule or ActivityScenarioRule. To benchmark UI code, use @UiThreadTest.

The following code shows a sample benchmark:

Kotlin

@RunWith(AndroidJUnit4::class)
class ViewBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun simpleViewInflate() {
        val context = ApplicationProvider.getApplicationContext()
        val inflater = LayoutInflater.from(context)
        val root = FrameLayout(context)

        benchmarkRule.keepRunning {
            inflater.inflate(R.layout.test_simple_view, root, false)
        }
    }
}

Java

@RunWith(AndroidJUnit4::class)
public class ViewBenchmark {
    @Rule
    public BenchmarkRule benchmarkRule = new BenchmarkRule();

    @Test
    public void simpleViewInflate() {
        Context context = ApplicationProvider.getApplicationContext();
        final BenchmarkState state = benchmarkRule.getState();
        LayoutInflater inflater = LayoutInflater.from(context);
        FrameLayout root = new FrameLayout(context);

        while (state.keepRunning()) {
            inflater.inflate(R.layout.test_simple_view, root, false);
        }
    }
}

You can disable timing for sections of code you don't want to measure, as shown in the following code sample:

Kotlin

@Test
fun bitmapProcessing() = benchmarkRule.measureRepeated {
    val input: Bitmap = runWithTimingDisabled { constructTestBitmap() }
    processBitmap(input)
}

Java

@Test
public void bitmapProcessing() {
    final BenchmarkState state = benchmarkRule.getState();
    while (state.keepRunning()) {
        state.pauseTiming();
        Bitmap input = constructTestBitmap();
        state.resumeTiming();

        processBitmap(input);
    }
}

For information on running the benchmark, see Run the benchmark.

Run the benchmark

In Android Studio, run your benchmark as you would any @Test. On Android Studio 3.4 and higher, you can see output sent to the console.

To run the benchmark, in the module, navigate to benchmark/src/androidTest and press Control+Shift+F10 (Command+Shift+R on Mac). Results of the benchmark appear in the console, as shown in Figure 2:

Figure 2. Benchmarking output in Android Studio

From the command line, run the regular connectedCheck:

./gradlew benchmark:connectedCheck

Collect the data

A full benchmark report with additional metrics and device information is available in JSON format.

The androidx.benchmark Gradle plugin enables JSON output by default. To manually enable JSON output in non-Gradle build environments, you pass an instrumentation argument, androidx.benchmark.output.enable, set to true.

The following is an example using the adb shell am instrument command:

adb shell am instrument -w -e "androidx.benchmark.output.enable" "true" com.android.foo/androidx.benchmark.junit.AndroidBenchmarkRunner

By default, the JSON report is written to disk on device in the test APK's external shared downloads folder, which is typically located at:

/storage/emulated/0/Download/app_id-benchmarkData.json

You can configure the location where benchmark reports are saved on the device using the instrumentation argument additionalTestOutputDir.

adb shell am instrument -w -e "androidx.benchmark.output.enable" "true" -e "additionalTestOutputDir" "/path_to_a_directory_on_device_test_has_write_permissions_to/" com.android.foo/androidx.benchmark.AndroidBenchmarkRunner

Android Gradle Plugin 3.6 and higher

When running benchmarks from the command line with Gradle, projects using Android Gradle Plugin 3.6 and higher can add the following flag to the project's gradle.properties file:

android.enableAdditionalTestOutput=true

This will enable an experimental Android Gradle Plugin feature to pull benchmark reports from devices running API 16 and later to the following directory on the host machine:

project_root/module/build/outputs/connected_android_test_additional_output/debugAndroidTest/connected/device_id/app_id-benchmarkData.json

Android Gradle Plugin 3.5 and lower

The androidx.benchmark Gradle plugin will copy the JSON report from device to host. These are written on the host machine at:

project_root/module/build/benchmark_reports/device_id/app_id-benchmarkData.json

To copy data when using AGP 3.5 or lower, you need to add a flag to your Android manifest in the androidTest directory of your benchmarks to enable legacy external storage behavior. See Opt out of filtered view for more information on opting out of scoped storage.

<manifest ... >
  <!-- This attribute is "false" by default on apps targeting Android Q. -->
  <application android:requestLegacyExternalStorage="true" ... >
    ...
  </application>
</manifest>

Clock stability

Clocks on mobile devices dynamically change from high state (for performance) to low state (to save power, or when the device gets hot). These varying clocks can make your benchmark numbers vary widely, so the library provides ways to deal with this issue.

Lock clocks (requires root)

Locking clocks is the best way to get stable performance. It ensures that clocks never get high enough to heat up the device, or low if a benchmark isn't fully utilizing the CPU. While this is the best way to ensure stable performance, it isn't supported on most devices, due to requiring adb root.

To lock your clocks, add the supplied helper plugin to the top level project’s classpath in the main build.gradle file:

buildscript {
    ...
    dependencies {
        ...
        classpath "androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha04"
    }
}

Apply the plugin in the build.gradle of the module you are benchmarking:

apply plugin: com.android.app
apply plugin: androidx.benchmark
...

This adds benchmarking Gradle tasks to your project, including ./gradlew lockClocks and ./gradlew unlockClocks. Use these tasks to lock and unlock a device’s CPU using adb.

If you have multiple devices visible to adb, use the environment variable ANDROID_SERIAL to specify which device the Gradle task should operate on:

ANDROID_SERIAL=device-id-from-adb-devices ./gradlew lockClocks

Sustained performance mode

Window.setSustainedPerformanceMode() is a feature supported by some devices that enables an app to opt for a lower max CPU frequency. When running on supported devices, the Benchmark library uses a combination of this API and launching its own activity to both prevent thermal throttling and stabilize results.

This functionality is enabled by default by the testInstrumentationRunner set by the Gradle plugin. If you want to use a custom runner, you can subclass the AndroidBenchmarkRunner and use it as your testInstrumentationRunner.

The runner launches an opaque, fullscreen activity to ensure that the benchmark runs in the foreground and without any other app drawing.

Automatic execution pausing

If neither clock-locking nor sustained performance are used, the library performs automatic thermal throttling detection. When enabled, the internal benchmark periodically runs to determine when the device temperature has gotten high enough to lower CPU performance. When lowered CPU performance is detected, the library pauses execution to let the device cool down, and retries the current benchmark.

Configuration Errors

The library detects the following conditions to ensure your project and environment are set up for release-accurate performance:

Debuggable is set to false.
A physical device, not an emulator, is being used.
Clocks are locked if the device is rooted.
Sufficient battery level on device.

If any of the above checks fail, the benchmark will throw an error to discourage inaccurate measurements.

To suppress these errors as warnings, and prevent them from throwing an error and halting the benchmark, pass the type of errors you want to suppress in a comma-separated list to the instrumentation argument androidx.benchmark.suppressErrors:

adb shell am instrument -w -e "androidx.benchmark.suppressErrors" "DEBUGGABLE" com.android.foo/androidx.benchmark.junit.AndroidBenchmarkRunner

Which you may set in Gradle as follows:

android {
    defaultConfig {
        testInstrumentationRunnerArgument 'androidx.benchmark.suppressErrors', 'DEBUGGABLE'
    }
}

Note that suppressing errors allows the benchmark to run in an incorrectly configured state, but the output of the benchmark will be intentionally corrupted by prepending test names with the error. That is, running a debuggable benchmark with the above suppression will prepend test names with DEBUGGABLE_.

Benchmarking samples

Sample benchmark code is available in the following projects:

The sample projects include:

BenchmarkSample: this is a standalone sample, which shows how to use a benchmark module to measure code and UI.
PagingWithNetworkSample: Android Architecture Components sample, which shows how to benchmark RecyclerView performance.
WorkManagerSample: Android Architecture Components sample, which shows how to benchmark WorkManager workers.

Provide feedback

To report issues or submit feature requests when using benchmarking, see the public issue tracker.