Image analysis

The image analysis use case provides your app with a CPU-accessible image to perform image processing, computer vision, or machine learning inference on. The application implements an analyze() method that is run on each frame.

Implementation

Images are processed by passing an executor in which the image analysis is run and an ImageAnalysis.Analyzer parameter to the setAnalyzer() method. Only one analyzer can be registered at any time: registering a new analyzer replaces the existing one.

The code example in this topic shows how to do this, as well as how to bind the image analysis use case and a preview use case to the LifecycleOwner. To read about creating preview use cases, see Implement a preview.

Image analysis can work in two modes: blocking and non-blocking. Blocking mode is enabled by calling setBackpressureStrategy() with STRATEGY_BLOCK_PRODUCER. In this mode, the executor receives frames from the camera in sequential order; this means that, if the analyze() method takes longer than the latency of a single frame at the current framerate, the frames may no longer be current since new frames are blocked from entering the pipeline until the method returns.

Non-blocking mode is enabled by calling setBackpressureStrategy() with STRATEGY_KEEP_ONLY_LATEST. In this mode, the executor receives the last available frame from the camera at the time that the analyze() method is called. If the method takes longer than the latency of a single frame at the current framerate, some frames might be skipped so that the next time analyze() receives data, it gets the last frame available in the camera pipeline.

Before returning from analyze(), close the image reference by calling image.close() to avoid blocking the production of further images (causing the preview to stall) and to avoid potentially dropping images. The method must complete analysis or make a copy instead of passing the image reference beyond the analysis method.

Kotlin

val imageAnalysis = ImageAnalysis.Builder()
    .setTargetResolution(Size(1280, 720))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()

imageAnalysis.setAnalyzer(executor, ImageAnalysis.Analyzer { image ->
    val rotationDegrees = image.imageInfo.rotationDegrees
    // insert your code here.
})

cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, imageAnalysis, preview)

Java

ImageAnalysis imageAnalysis =
    new ImageAnalysis.Builder()
        .setTargetResolution(new Size(1280, 720))
        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
        .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
        .build();

imageAnalysis.setAnalyzer(executor, new ImageAnalysis.Analyzer() {
    @Override
    public void analyze(@NonNull ImageProxy image) {
        int rotationDegrees = image.getImageInfo().getRotationDegrees();
            // insert your code here.
        }
    });


cameraProvider.bindToLifecycle((LifecycleOwner) this, cameraSelector, imageAnalysis, preview);

By default, CameraX sets images in OUTPUT_IMAGE_FORMAT_YUV_420_888 format.

The application can check the color space of the ImageProxy passed in by CameraX with the ImageProxy::getFormat() function. If it returns ImageFormat.YUV_420_888, the image is in YUV color space. If it returns PixelFormat.RGBA_8888, the image is in RGBA color space.

The YUV output format is available for all camera-core versions. The RGBA output format is available for camera-core:1.1.0-alpha09 and later versions, for Image Analysis use case only.

When setting an RGBA output format, CameraX internally converts images from YUV to RGBA space and stores image bits in the ByteBuffer of the ImageProxy’s first plane (other planes are not used) with the following sequence:

ImageProxy::planes[0].buffer[0]: alpha
ImageProxy::planes[1].buffer[1]: red
ImageProxy::planes[2].buffer[2]: green
ImageProxy::planes[3].buffer[3]: blue
...

Color conversion is an add-on step to the YUV output format, and conversion time is part of total image analysis time. The application will have less per frame processing time when performing real-time analysis. You should use it when your application needs RGBA output.

Additional resources

To learn more about CameraX, consult the following additional resources.

Codelab

  • Getting Started with CameraX
  • Code sample

  • Official CameraX sample app