Writing a Microbenchmark

You can quickly start using the Microbenchmark library by adding changes to your application code. For the proper project setup, follow the Full setup section, which requires more complicated changes to your codebase.

Quickstart

This section provides a quick guide to try out benchmarking and run one-off measurements without requiring you to move code into modules. For accurate performance results, these steps involve disabling debugging in your application, so you should keep this in local working copy without committing the changes to your source control system.

To quickly perform one-off benchmarking, do the following:

1. Add the library to your module’s build.gradle file:

   project_root/module_dir/build.gradle

   Groovy

   dependencies {
       androidTestImplementation 'androidx.benchmark:benchmark-junit4:1.1.1'
   }
   

   Kotlin

   dependencies {
       androidTestImplementation("androidx.benchmark:benchmark-junit4:1.1.1")
   }
   

2. To disable debugging in the test manifest, update your <application> element to force-disable debugging temporarily:

   project_root/module_dir/src/androidTest/AndroidManifest.xml

   <!-- Important: disable debuggable for accurate performance results -->
   <application
       android:debuggable="false"
       tools:ignore="HardcodedDebugMode"
       tools:replace="android:debuggable"/>
   

3. To add your benchmark, add an instance of BenchmarkRule in a test file in the androidTest directory. For more information on writing benchmarks, see Create a Microbenchmark class.

   The following code snippet shows how to add a benchmark to a JUnit test:

   Kotlin

   @RunWith(AndroidJUnit4::class)
   class SampleBenchmark {
       @get:Rule
       val benchmarkRule = BenchmarkRule()
   
       @Test
       fun benchmarkSomeWork() {
           benchmarkRule.measureRepeated {
               doSomeWork()
           }
       }
   }
   SampleBenchmark.kt
   

   Java

   @RunWith(AndroidJUnit4.class)
   class SampleBenchmark {
       @Rule
       public BenchmarkRule benchmarkRule = new BenchmarkRule();
   
       @Test
       public void benchmarkSomeWork() {
           final BenchmarkState state = benchmarkRule.getState();
           while (state.keepRunning()) {
               doSomeWork();
           }
       }
   }
   

To see how to write a benchmark, skip to Create a Microbenchmark class section.

Full project setup

To set up regular benchmarking rather than one-off benchmarking, isolate benchmarks into their own module. This ensures that their configuration, such as setting debuggable to false, is separate from regular tests.

Because Microbenchmark runs your code directly, you need to place the code you want to benchmark into a separate Gradle module and set dependency on that module as shown in the following figure.

To add a new Gradle module, you can use the module wizard in Android Studio. The wizard creates a module that is pre-configured for benchmarking, with a benchmark directory added and debuggable set to false.

After the module is created, change its build.gradle file and add testImplementation to the module containing code to benchmark:

dependencies {
    // Note, that the module name may be different
    androidTestImplementation project(':benchmarkable')
}

dependencies {
    // Note, that the module name may be different
    androidTestImplementation(project(":benchmarkable"))
}

Create a Microbenchmark class

Benchmarks are standard instrumentation tests. To create a benchmark, use the BenchmarkRule class provided by the library. To benchmark activities, use ActivityTestRule or ActivityScenarioRule. To benchmark UI code, use @UiThreadTest.

The following code shows a sample benchmark:

@RunWith(AndroidJUnit4::class)
class SampleBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun benchmarkSomeWork() {
        benchmarkRule.measureRepeated {
            doSomeWork()
        }
    }
}
SampleBenchmark.kt
    

@RunWith(AndroidJUnit4.class)
class SampleBenchmark {
    @Rule
    public BenchmarkRule benchmarkRule = new BenchmarkRule();

    @Test
    public void benchmarkSomeWork() {
        final BenchmarkState state = benchmarkRule.getState();
        while (state.keepRunning()) {
            doSomeWork();
        }
    }
}
    

Disable timing for setup

You can disable timing for sections of code you don't want to measure with the runWithTimingDisabled{} block. These sections usually represent some code which you need to run on each iteration of the benchmark.

// using random with the same seed, so that it generates the same data every run
private val random = Random(0)

// create the array once and just copy it in benchmarks
private val unsorted = IntArray(10_000) { random.nextInt() }

@Test
fun benchmark_quickSort() {
    // ...
    benchmarkRule.measureRepeated {
        // copy the array with timing disabled to measure only the algorithm itself
        listToSort = runWithTimingDisabled { unsorted.copyOf() }

        // sort the array in place and measure how long it takes
        SortingAlgorithms.quickSort(listToSort)
    }

    // assert only once not to add overhead to the benchmarks
    assertTrue(listToSort.isSorted)
}
SortingBenchmarks.kt
    

private final int[] unsorted = new int[10000];

public SampleBenchmark() {
    // using random with the same seed, so that it generates the same data every run
    Random random = new Random(0);

    // create the array once and just copy it in benchmarks
    Arrays.setAll(unsorted, (index) -> random.nextInt());
}

@Test
public void benchmark_quickSort() {
    final BenchmarkState state = benchmarkRule.getState();

    int[] listToSort = new int[0];

    while (state.keepRunning()) {
        
        // copy the array with timing disabled to measure only the algorithm itself
        state.pauseTiming();
        listToSort = Arrays.copyOf(unsorted, 10000);
        state.resumeTiming();
        
        // sort the array in place and measure how long it takes
        SortingAlgorithms.quickSort(listToSort);
    }

    // assert only once not to add overhead to the benchmarks
    assertTrue(SortingAlgorithmsKt.isSorted(listToSort));
}
    

Try to minimize the amount of work done inside the measureRepeated block, as well as inside runWithTimingDisabled. The measureRepeated block is run multiple times and it can affect the overall time needed to run the benchmark. If you need to verify some results of a benchmark, you can assert the last result instead of doing it every iteration of the benchmark.

Run the benchmark

In Android Studio, run your benchmark as you would any @Test using the gutter action next to your test class or method, as shown in the following image.

Alternatively, from the command line, run the connectedCheck to run all of the tests from specified Gradle module:

./gradlew benchmark:connectedCheck

Or a single test:

./gradlew benchmark:connectedCheck -P android.testInstrumentationRunnerArguments.class=com.example.benchmark.SampleBenchmark#benchmarkSomeWork

Benchmark results

After a successful Microbenchmark run, metrics are displayed directly in Android Studio and a full benchmark report with additional metrics and device information is available in JSON format.

JSON reports and any profiling traces are also automatically copied from device to host. These are written on the host machine at:

project_root/module/build/outputs/connected_android_test_additional_output/debugAndroidTest/connected/device_id/

By default, the JSON report is written to disk on-device in the test APK's external shared media folder, which is typically located at /storage/emulated/0/Android/media/**app_id**/**app_id**-benchmarkData.json.

Configuration errors

The library detects the following conditions to ensure your project and environment are set up for release-accurate performance:

• Debuggable is set to false.
• A physical device is being used (emulators are not supported).
• Clocks are locked if the device is rooted.
• Sufficient battery level on device (at least 25%).

If any of the checks previous fail, the benchmark reports an error to discourage inaccurate measurements.

To suppress specific error types as warnings and prevent them from halting the benchmark, pass the error type in a comma-separated list to the instrumentation argument androidx.benchmark.suppressErrors.

This can be set either from your Gradle script as shown below:

android {
    defaultConfig {
       …
      testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE,LOW-BATTERY"
    }
}

android {
    defaultConfig {
       …
      testInstrumentationRunnerArguments["androidx.benchmark.suppressErrors"] = "DEBUGGABLE,LOW-BATTERY"
    }
}

Suppressing errors lets the benchmark run in an incorrectly configured state, and the output of the benchmark is intentionally renamed by prepending test names with the error. For example, running a debuggable benchmark with the suppression in the preceding snippet prepends test names with DEBUGGABLE_.