Benchmark

The Benchmark library makes it easy to benchmark your Kotlin-based or Java-based code from within Android Studio. The library handles warmup, measures your code performance, and outputs benchmarking results to the Android Studio console.

For information on how to use the Benchmark library in your app or library, see Benchmark app code.

Declaring dependencies

To add a dependency on Benchmark, you must add the Google Maven repository to your project. Read Google's Maven repository for more information.

Add the dependencies for the artifacts you need in the build.gradle file for your app or module:

dependencies {
  androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0-alpha06"
}

android {
  ...
  defaultConfig {
    ...
    testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner"
  }
}

Version 1.0.0-alpha06

September 18, 2019

androidx.benchmark:benchmark-common:1.0.0-alpha06, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha06, and androidx.benchmark:benchmark-junit4:1.0.0-alpha06 are released. Version 1.0.0-alpha06 contains these commits.

New features

  • Added a check for incorrectly using the old package for the test runner, which now provides a more-helpful error message

API changes

  • The experimental annotation ExperimentalAnnotationReport is now correctly public. Usage of the experimental BenchmarkState#report API now requires this annotation

Version 1.0.0-alpha05

September 5, 2019

androidx.benchmark:benchmark-common:1.0.0-alpha05, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha05, and androidx.benchmark:benchmark-junit4:1.0.0-alpha05 are released. The commits included in this version can be found here.

API changes

  • BenchmarkState.reportData API is now marked experimental

Bug fixes

  • Fix for the clock-locking script, which would fail on devices that were either missing the cut or expr shell utilities.
  • Fixed an issue with ./gradlew lockClocks task that would hang on devices that were rooted with an older version of the su utility, which did not support the -c flag.

Version 1.0.0-alpha04

August 7, 2019

androidx.benchmark:benchmark-common:1.0.0-alpha04, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha04, and androidx.benchmark:benchmark-junit4:1.0.0-alpha04 are released. The commits included in this version can be found here.

New documentation has also been added for how to use the Benchmark library without Gradle, both for usage with different build systems (such as Bazel or Buck), and when running in CI. For more information, see Build benchmarks without Gradle and Run benchmarks in Continuous Integration.

New features

  • Gradle plugin
    • Now automatically disables test coverage, and sets the AndroidBenchmarkRunner by default (b/138374050)
    • Added support for new AGP-based data copy, when running benchmarks and when using AGP 3.6+
  • JSON format additions
    • Output total benchmark test run time (b/133147694)
    • @Parameterized benchmarks that use a name string (for example @Parameters(name = "size={0},depth={1}")) now output parameter names and values per benchmark in the JSON output (b/132578772)
  • Dry Run mode (b/138785848)
    • Added a "dry run" mode for running each benchmark loop only once, to check for errors/crashes without capturing measurements. This can be useful e.g. for, for example, quickly running benchmarks in presubmit to check that they're not broken.

API changes

  • Module structure has changed, splitting the library (b/138451391)
    • benchmark:benchmark-junit4 contains classes with JUnit dependency: AndroidBenchmarkRunner, and BenchmarkRule, both of which have moved into the androidx.benchmark.junit4 package
    • benchmark:benchmark-common contains the rest of the logic, including the BenchmarkState API
    • This split will allow the library to support benchmarking without JUnit4 APIs in the future
  • Configuration warnings are now treated as errors, and will crash the test (b/137653596)
    • This is done to further encourage accurate measurements, especially in CI
    • These errors can be reduced back to warnings with an instrumentation argument. For example: -e androidx.benchmark.suppressErrors "DEBUGGABLE,LOW_BATTERY"

Bug fixes

  • Errors when writing to external storage on Q devices provide more-descriptive messages, with suggestions of how to resolve the issue
  • Screens are automatically turned on during benchmark runs, instead of failing when the screen is off

External contributions

  • Thanks to Sergey Zakharov for contributing JSON output improvements and the fix for screen off issues!

Version 1.0.0-alpha03

July 2, 2019

androidx.benchmark:benchmark:1.0.0-alpha03 and androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha03 are released. The commits included in this version can be found here.

New features

  • Expose sleep duration due to thermal throttling per benchmark in the full JSON report

Bug fixes

  • The Gradle plugin should no longer be required to be applied after Android plugins and the Android block
  • Adds support for benchmark reports on Android Q devices using scoped storage

Version 1.0.0-alpha02

June 6, 2019

androidx.benchmark:1.0.0-alpha02 and androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha02 are released. The commits included in this version can be found here.

Note that we are treating the JSON schema as an API. We plan to follow the same stability constraints as other APIs: stable (with very rare exceptions) once in beta, and fixed in final release, with only additions in minor releases and changes/removals in major releases.

API changes

  • Overhauled JSON schema. Further changes to the JSON schema are likely to be limited to additions:

    • Reorganized the result object structure to support additional metric groups in the future (b/132713021)
    • Added test run context information, such as device and build info and whether clocks are locked, to the top-level object (b/132711920)
    • Time metric names now have ‘ns’ in their name (b/132714527)
    • Additional stats added per reported metric (maximum, median, minimum), and removed simplified 'nanos' summary stat (b/132713851)
  • Removed XML output (b/132714414)

  • Thermal throttle detection removed from BenchmarkState.reportData API (b/132887006)

Bug fixes

  • Fixed ./gradlew lockClocks not sticking on some recent OS devices (b/133424037)
  • Throttling detection disabled for emulator (b/132880807)

Version 1.0.0-alpha01

May 7, 2019

androidx.benchmark:benchmark:1.0.0-alpha01 is released. The commits included in this version are available here.