Benchmark

Accurately measure your code's performance within Android Studio.
Latest Update Current Stable Release Next Release Candidate Beta Release Alpha Release
June 10, 2020 1.0.0 - - 1.1.0-alpha01

Declaring dependencies

To add a dependency on Benchmark, you must add the Google Maven repository to your project. Read Google's Maven repository for more information.

Add the dependencies for the artifacts you need in the build.gradle file for your app or module:

dependencies {
  androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
}

android {
  ...
  defaultConfig {
    ...
    testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner"
  }
}

The Benchmark library also provides a Gradle plugin to use with your benchmark module. This plugin sets build configuration defaults for the module, sets up benchmark output copy to the host, and provides the ./gradlew lockClocks task.

To use the plugin, include the following classpath in your top level build.gradle file:

buildscript {
    repositories {
        google()
    }
    dependencies {
        classpath "androidx.benchmark:benchmark-gradle-plugin:1.0.0"
    }
}

Then apply the plugin to your benchmark module's build.gradle file 

apply plugin: "androidx.benchmark"

Feedback

Your feedback helps make Jetpack better. Let us know if you discover new issues or have ideas for improving this library. Please take a look at the existing issues in this library before you create a new one. You can add your vote to an existing issue by clicking the star button.

Create a new issue

See the Issue Tracker documentation for more information.

Version 1.1.0

Version 1.1.0-alpha01

June 10, 2020

androidx.benchmark:benchmark-common:1.1.0-alpha01, androidx.benchmark:benchmark-gradle-plugin:1.1.0-alpha01, and androidx.benchmark:benchmark-junit4:1.1.0-alpha01 are released. Version 1.1.0-alpha01 contains these commits.

New Features of 1.1

  • Allocation Metric - Benchmarks now run an additional phase after warmup and timing, capturing allocation counts. Allocations can cause performance problems on older versions of the platform (140ns in O became 8ns in M - measured on Nexus5X, with locked clocks). This metric is displayed in Android Studio console output, as well as in the
  • Profiling support - You can now capture profiling data for a benchmark run, to inspect why your code may be running slowly. Benchmark supports capturing either method tracing, or method sampling from ART. These files can be inspected with the Profiler inside Android Studio using File > Open.
  • The Benchmark Gradle plugin now provides defaults for simpler setup:
    • testBuildType is set to release by default, to avoid using dependencies with code coverage built-in. The release buildType is also configured as the default buildType, which allows Android Studio to automatically select the correct build variant when opening a project for the first time. (b/138808399)
    • signingConfig.debug is used as the default signing config (b/153583269)

** Bug Fixes **

  • Significantly reduced the warmup transition overhead, where the first measurement for each benchmark was artificially higher than others. This issue was more pronounced in very small benchmarks (1 microsecond or less). (b/142058671)
  • Fixed InstrumentationResultParser error printed for each benchmark when running from command line. (I64988, b/154248456)

Known Issues

  • Command line, gradle invocations of Benchmark do not print out results directly. You can work around this by either running through Studio, or parsing the JSON output file for results.
  • Benchmark reporting fails to pull the report from devices that have an app installed with an applicationId ending with either “android” or “download” (case insensitive). Users hitting this issue should upgrade the Android Gradle Plugin to 4.2-alpha01 or later.

Version 1.0.0

Benchmark Version 1.0.0

November 20, 2019

androidx.benchmark:benchmark-common:1.0.0, androidx.benchmark:benchmark-gradle-plugin:1.0.0, and androidx.benchmark:benchmark-junit4:1.0.0 are released with no changes from 1.0.0-rc01. Version 1.0.0 contains these commits.

Major features of 1.0.0

The Benchmark library allows you to write performance benchmarks of app code and get results quickly.

It prevents build and runtime configuration issues and stabilizes device performance to ensure that measurements are accurate and consistent. Run the benchmarks directly in Android Studio, or in Continuous Integration to observe code performance over time, and to prevent regressions.

Major features include:

  • Clock stabilization
  • Automatic thread prioritization
  • Support for UI performance testing, such as in the RecyclerView Sample
  • JIT-aware warmup and looping
  • JSON benchmark output for post-processing

Version 1.0.0-rc01

October 23, 2019

androidx.benchmark:benchmark-common:1.0.0-rc01, androidx.benchmark:benchmark-gradle-plugin:1.0.0-rc01, and androidx.benchmark:benchmark-junit4:1.0.0-rc01 are released. Version 1.0.0-rc01 contains these commits.

New features

  • Added systrace tracing to benchmarks

Bug fixes

  • Fixed metric instability issue where JIT wouldn't finish before warm up due to deprioritization (b/140773023)
  • Unified JSON output directory across Android Gradle Plugin 3.5 and 3.6

Version 1.0.0-beta01

October 9, 2019

androidx.benchmark:benchmark-common:1.0.0-beta01, androidx.benchmark:benchmark-gradle-plugin:1.0.0-beta01, and androidx.benchmark:benchmark-junit4:1.0.0-beta01 are released. Version 1.0.0-beta01 contains these commits.

New features

  • Run garbage collection before each warmup to reduce memory pressure from one benchmark to leak to the next (b/140895105)

Bug fixes

  • Added androidx.annotation:android-experimental-lint dependency, so that Java code will correctly produce lint errors when experimental API is not used, similar to what is provided by the Kotlin experimental annotation for Kotlin callers.
  • Now correctly detects usage of additionalTestOutputDir instrumentation argument for output in Android Gradle Plugin 3.6, to know when AGP will handle data copy.
  • Fix undetected clock frequency in JSON to correctly print -1 (b/141945670).

Version 1.0.0-alpha06

September 18, 2019

androidx.benchmark:benchmark-common:1.0.0-alpha06, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha06, and androidx.benchmark:benchmark-junit4:1.0.0-alpha06 are released. Version 1.0.0-alpha06 contains these commits.

New features

  • Added a check for incorrectly using the old package for the test runner, which now provides a more-helpful error message

API changes

  • The experimental annotation ExperimentalAnnotationReport is now correctly public. Usage of the experimental BenchmarkState#report API now requires this annotation

Version 1.0.0-alpha05

September 5, 2019

androidx.benchmark:benchmark-common:1.0.0-alpha05, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha05, and androidx.benchmark:benchmark-junit4:1.0.0-alpha05 are released. The commits included in this version can be found here.

API changes

  • BenchmarkState.reportData API is now marked experimental

Bug fixes

  • Fix for the clock-locking script, which would fail on devices that were either missing the cut or expr shell utilities.
  • Fixed an issue with ./gradlew lockClocks task that would hang on devices that were rooted with an older version of the su utility, which did not support the -c flag.

Version 1.0.0-alpha04

August 7, 2019

androidx.benchmark:benchmark-common:1.0.0-alpha04, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha04, and androidx.benchmark:benchmark-junit4:1.0.0-alpha04 are released. The commits included in this version can be found here.

New documentation has also been added for how to use the Benchmark library without Gradle, both for usage with different build systems (such as Bazel or Buck), and when running in CI. For more information, see Build benchmarks without Gradle and Run benchmarks in Continuous Integration.

New features

  • Gradle plugin
    • Now automatically disables test coverage, and sets the AndroidBenchmarkRunner by default (b/138374050)
    • Added support for new AGP-based data copy, when running benchmarks and when using AGP 3.6+
  • JSON format additions
    • Output total benchmark test run time (b/133147694)
    • @Parameterized benchmarks that use a name string (for example @Parameters(name = "size={0},depth={1}")) now output parameter names and values per benchmark in the JSON output (b/132578772)
  • Dry Run mode (b/138785848)
    • Added a "dry run" mode for running each benchmark loop only once, to check for errors/crashes without capturing measurements. This can be useful e.g. for, for example, quickly running benchmarks in presubmit to check that they're not broken.

API changes

  • Module structure has changed, splitting the library (b/138451391)
    • benchmark:benchmark-junit4 contains classes with JUnit dependency: AndroidBenchmarkRunner, and BenchmarkRule, both of which have moved into the androidx.benchmark.junit4 package
    • benchmark:benchmark-common contains the rest of the logic, including the BenchmarkState API
    • This split will allow the library to support benchmarking without JUnit4 APIs in the future
  • Configuration warnings are now treated as errors, and will crash the test (b/137653596)
    • This is done to further encourage accurate measurements, especially in CI
    • These errors can be reduced back to warnings with an instrumentation argument. For example: -e androidx.benchmark.suppressErrors "DEBUGGABLE,LOW_BATTERY"

Bug fixes

  • Errors when writing to external storage on Q devices provide more-descriptive messages, with suggestions of how to resolve the issue
  • Screens are automatically turned on during benchmark runs, instead of failing when the screen is off

External contributions

  • Thanks to Sergey Zakharov for contributing JSON output improvements and the fix for screen off issues!

Version 1.0.0-alpha03

July 2, 2019

androidx.benchmark:benchmark:1.0.0-alpha03 and androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha03 are released. The commits included in this version can be found here.

New features

  • Expose sleep duration due to thermal throttling per benchmark in the full JSON report

Bug fixes

  • The Gradle plugin should no longer be required to be applied after Android plugins and the Android block
  • Adds support for benchmark reports on Android 10 devices using scoped storage

Version 1.0.0-alpha02

June 6, 2019

androidx.benchmark:1.0.0-alpha02 and androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha02 are released. The commits included in this version can be found here.

Note that we are treating the JSON schema as an API. We plan to follow the same stability constraints as other APIs: stable (with very rare exceptions) once in beta, and fixed in final release, with only additions in minor releases and changes/removals in major releases.

API changes

  • Overhauled JSON schema. Further changes to the JSON schema are likely to be limited to additions:

    • Reorganized the result object structure to support additional metric groups in the future (b/132713021)
    • Added test run context information, such as device and build info and whether clocks are locked, to the top-level object (b/132711920)
    • Time metric names now have ‘ns’ in their name (b/132714527)
    • Additional stats added per reported metric (maximum, median, minimum), and removed simplified 'nanos' summary stat (b/132713851)

  • Removed XML output (b/132714414)

  • Thermal throttle detection removed from BenchmarkState.reportData API (b/132887006)

Bug fixes

  • Fixed ./gradlew lockClocks not sticking on some recent OS devices (b/133424037)
  • Throttling detection disabled for emulator (b/132880807)

Version 1.0.0-alpha01

May 7, 2019

androidx.benchmark:benchmark:1.0.0-alpha01 is released. The commits included in this version are available here.