Benchmark
Latest Update | Current Stable Release | Next Release Candidate | Beta Release | Alpha Release |
---|---|---|---|---|
June 10, 2020 | 1.0.0 | - | - | 1.1.0-alpha01 |
Declaring dependencies
To add a dependency on Benchmark, you must add the Google Maven repository to your project. Read Google's Maven repository for more information.
Add the dependencies for the artifacts you need in the build.gradle
file for
your app or module:
dependencies { androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0" } android { ... defaultConfig { ... testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner" } }
The Benchmark library also provides a Gradle plugin to use with your benchmark module. This
plugin sets build configuration defaults for the module, sets up
benchmark output copy to the host, and provides
the
./gradlew lockClocks
task.
To use the plugin, include the following classpath
in your top level
build.gradle
file:
buildscript { repositories { google() } dependencies { classpath "androidx.benchmark:benchmark-gradle-plugin:1.0.0" } }
Then apply the plugin to your benchmark module's build.gradle
file
apply plugin: "androidx.benchmark"
Feedback
Your feedback helps make Jetpack better. Let us know if you discover new issues or have ideas for improving this library. Please take a look at the existing issues in this library before you create a new one. You can add your vote to an existing issue by clicking the star button.
See the Issue Tracker documentation for more information.
Version 1.1.0
Version 1.1.0-alpha01
June 10, 2020
androidx.benchmark:benchmark-common:1.1.0-alpha01
, androidx.benchmark:benchmark-gradle-plugin:1.1.0-alpha01
, and androidx.benchmark:benchmark-junit4:1.1.0-alpha01
are released. Version 1.1.0-alpha01 contains these commits.
New Features of 1.1
- Allocation Metric - Benchmarks now run an additional phase after warmup and timing, capturing allocation counts. Allocations can cause performance problems on older versions of the platform (140ns in O became 8ns in M - measured on Nexus5X, with locked clocks). This metric is displayed in Android Studio console output, as well as in the
- Profiling support - You can now capture profiling data for a benchmark run, to inspect why your code may be running slowly. Benchmark supports capturing either method tracing, or method sampling from ART. These files can be inspected with the Profiler inside Android Studio using File > Open.
- The Benchmark Gradle plugin now provides defaults for simpler setup:
testBuildType
is set to release by default, to avoid using dependencies with code coverage built-in. The release buildType is also configured as the default buildType, which allows Android Studio to automatically select the correct build variant when opening a project for the first time. (b/138808399)signingConfig.debug
is used as the default signing config (b/153583269)
** Bug Fixes **
- Significantly reduced the warmup transition overhead, where the first measurement for each benchmark was artificially higher than others. This issue was more pronounced in very small benchmarks (1 microsecond or less). (b/142058671)
- Fixed
InstrumentationResultParser
error printed for each benchmark when running from command line. (I64988, b/154248456)
Known Issues
- Command line, gradle invocations of Benchmark do not print out results directly. You can work around this by either running through Studio, or parsing the JSON output file for results.
- Benchmark reporting fails to pull the report from devices that have an app installed with an applicationId ending with either “android” or “download” (case insensitive). Users hitting this issue should upgrade the Android Gradle Plugin to 4.2-alpha01 or later.
Version 1.0.0
Benchmark Version 1.0.0
November 20, 2019
androidx.benchmark:benchmark-common:1.0.0
, androidx.benchmark:benchmark-gradle-plugin:1.0.0
, and androidx.benchmark:benchmark-junit4:1.0.0
are released with no changes from 1.0.0-rc01. Version 1.0.0 contains these commits.
Major features of 1.0.0
The Benchmark library allows you to write performance benchmarks of app code and get results quickly.
It prevents build and runtime configuration issues and stabilizes device performance to ensure that measurements are accurate and consistent. Run the benchmarks directly in Android Studio, or in Continuous Integration to observe code performance over time, and to prevent regressions.
Major features include:
- Clock stabilization
- Automatic thread prioritization
- Support for UI performance testing, such as in the RecyclerView Sample
- JIT-aware warmup and looping
- JSON benchmark output for post-processing
Version 1.0.0-rc01
October 23, 2019
androidx.benchmark:benchmark-common:1.0.0-rc01
, androidx.benchmark:benchmark-gradle-plugin:1.0.0-rc01
, and androidx.benchmark:benchmark-junit4:1.0.0-rc01
are released. Version 1.0.0-rc01 contains these commits.
New features
- Added systrace tracing to benchmarks
Bug fixes
- Fixed metric instability issue where JIT wouldn't finish before warm up due to deprioritization (b/140773023)
- Unified JSON output directory across Android Gradle Plugin 3.5 and 3.6
Version 1.0.0-beta01
October 9, 2019
androidx.benchmark:benchmark-common:1.0.0-beta01
, androidx.benchmark:benchmark-gradle-plugin:1.0.0-beta01
, and androidx.benchmark:benchmark-junit4:1.0.0-beta01
are released. Version 1.0.0-beta01 contains these commits.
New features
- Run garbage collection before each warmup to reduce memory pressure from one benchmark to leak to the next (b/140895105)
Bug fixes
- Added
androidx.annotation:android-experimental-lint
dependency, so that Java code will correctly produce lint errors when experimental API is not used, similar to what is provided by the Kotlin experimental annotation for Kotlin callers. - Now correctly detects usage of
additionalTestOutputDir
instrumentation argument for output in Android Gradle Plugin 3.6, to know when AGP will handle data copy. - Fix undetected clock frequency in JSON to correctly print
-1
(b/141945670).
Version 1.0.0-alpha06
September 18, 2019
androidx.benchmark:benchmark-common:1.0.0-alpha06
, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha06
, and androidx.benchmark:benchmark-junit4:1.0.0-alpha06
are released. Version 1.0.0-alpha06 contains these commits.
New features
- Added a check for incorrectly using the old package for the test runner, which now provides a more-helpful error message
API changes
- The experimental annotation
ExperimentalAnnotationReport
is now correctly public. Usage of the experimental BenchmarkState#report API now requires this annotation
Version 1.0.0-alpha05
September 5, 2019
androidx.benchmark:benchmark-common:1.0.0-alpha05
, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha05
, and androidx.benchmark:benchmark-junit4:1.0.0-alpha05
are released. The commits included in this version can be found here.
API changes
BenchmarkState.reportData
API is now marked experimental
Bug fixes
- Fix for the clock-locking script, which would fail on devices that were either missing the
cut
orexpr
shell utilities. - Fixed an issue with
./gradlew lockClocks
task that would hang on devices that were rooted with an older version of the su utility, which did not support the-c
flag.
Version 1.0.0-alpha04
August 7, 2019
androidx.benchmark:benchmark-common:1.0.0-alpha04
, androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha04
, and androidx.benchmark:benchmark-junit4:1.0.0-alpha04
are released. The commits included in this version can be found here.
New documentation has also been added for how to use the Benchmark library without Gradle, both for usage with different build systems (such as Bazel or Buck), and when running in CI. For more information, see Build benchmarks without Gradle and Run benchmarks in Continuous Integration.
New features
- Gradle plugin
- Now automatically disables test coverage, and sets the
AndroidBenchmarkRunner
by default (b/138374050) - Added support for new AGP-based data copy, when running benchmarks and when using AGP 3.6+
- Now automatically disables test coverage, and sets the
- JSON format additions
- Output total benchmark test run time (b/133147694)
@Parameterized
benchmarks that use a name string (for example@Parameters(name = "size={0},depth={1}")
) now output parameter names and values per benchmark in the JSON output (b/132578772)
- Dry Run mode (b/138785848)
- Added a "dry run" mode for running each benchmark loop only once, to check for errors/crashes without capturing measurements. This can be useful e.g. for, for example, quickly running benchmarks in presubmit to check that they're not broken.
API changes
- Module structure has changed, splitting the library (b/138451391)
benchmark:benchmark-junit4
contains classes with JUnit dependency:AndroidBenchmarkRunner
, andBenchmarkRule
, both of which have moved into theandroidx.benchmark.junit4
packagebenchmark:benchmark-common
contains the rest of the logic, including the BenchmarkState API- This split will allow the library to support benchmarking without JUnit4 APIs in the future
- Configuration warnings are now treated as errors, and will crash the test (b/137653596)
- This is done to further encourage accurate measurements, especially in CI
- These errors can be reduced back to warnings with an instrumentation argument. For example:
-e androidx.benchmark.suppressErrors "DEBUGGABLE,LOW_BATTERY"
Bug fixes
- Errors when writing to external storage on Q devices provide more-descriptive messages, with suggestions of how to resolve the issue
- Screens are automatically turned on during benchmark runs, instead of failing when the screen is off
External contributions
- Thanks to Sergey Zakharov for contributing JSON output improvements and the fix for screen off issues!
Version 1.0.0-alpha03
July 2, 2019
androidx.benchmark:benchmark:1.0.0-alpha03
and androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha03
are released. The commits included in this version can be found here.
New features
- Expose sleep duration due to thermal throttling per benchmark in the full JSON report
Bug fixes
- The Gradle plugin should no longer be required to be applied after Android plugins and the Android block
- Adds support for benchmark reports on Android 10 devices using scoped storage
Version 1.0.0-alpha02
June 6, 2019
androidx.benchmark:1.0.0-alpha02
and
androidx.benchmark:benchmark-gradle-plugin:1.0.0-alpha02
are released. The
commits included in this version can be found here.
Note that we are treating the JSON schema as an API. We plan to follow the same stability constraints as other APIs: stable (with very rare exceptions) once in beta, and fixed in final release, with only additions in minor releases and changes/removals in major releases.
API changes
Overhauled JSON schema. Further changes to the JSON schema are likely to be limited to additions:
- Reorganized the result object structure to support additional metric groups in the future (b/132713021)
- Added test run context information, such as device and build info and whether clocks are locked, to the top-level object (b/132711920)
- Time metric names now have ‘ns’ in their name (b/132714527)
- Additional stats added per reported metric (maximum, median, minimum), and removed simplified 'nanos' summary stat (b/132713851)
Removed XML output (b/132714414)
Thermal throttle detection removed from
BenchmarkState.reportData
API (b/132887006)
Bug fixes
- Fixed
./gradlew lockClocks
not sticking on some recent OS devices (b/133424037) - Throttling detection disabled for emulator (b/132880807)
Version 1.0.0-alpha01
May 7, 2019
androidx.benchmark:benchmark:1.0.0-alpha01
is released. The commits included
in this version are available
here.