MlKitAnalyzer


public class MlKitAnalyzer implements ImageAnalysis.Analyzer


An implementation of ImageAnalysis.Analyzer with ML Kit libraries.

This class is a wrapper of one or many ML Kit Detectors. It forwards ImageAnalysis frames to all the Detectors sequentially. Once all the Detectors finish analyzing the frame, accept will be invoked with the aggregated analysis results.

This class handles the coordinate transformation between ML Kit output and the target coordinate system. Using the targetCoordinateSystem set in the constructor, it calculates the Matrix with the value provided by CameraX via updateTransform and forwards it to the ML Kit Detector. The coordinates returned by MLKit will be in the specified coordinate system.

This class is designed to work seamlessly with the CameraController class in camera-view. When used with ImageAnalysis in camera-core, the following scenarios may need special handling:

Code sample:
 cameraController.setImageAnalysisAnalyzer(executor,
      new MlKitAnalyzer(List.of(barcodeScanner), COORDINATE_SYSTEM_VIEW_REFERENCED,
      executor, result -> {
   // The value of result.getResult(barcodeScanner) can be used directly for drawing UI overlay.
 });

Summary

Nested types

public final class MlKitAnalyzer.Result

The aggregated MLKit result of a camera frame.

Public constructors

MlKitAnalyzer(
    @NonNull List<Detector<Object>> detectors,
    int targetCoordinateSystem,
    @NonNull Executor executor,
    @NonNull Consumer<MlKitAnalyzer.Result> consumer
)

Constructor of MlKitAnalyzer.

Public methods

final void
analyze(@NonNull ImageProxy imageProxy)

Analyzes the image with the ML Kit Detectors.

final @NonNull Size

Implement this method to set a default target resolution for the .

final int

Implement this method to return the target coordinate system.

final void

Implement this method to receive the for coordinate transformation.

Public constructors

MlKitAnalyzer

Added in 1.1.0
public MlKitAnalyzer(
    @NonNull List<Detector<Object>> detectors,
    int targetCoordinateSystem,
    @NonNull Executor executor,
    @NonNull Consumer<MlKitAnalyzer.Result> consumer
)

Constructor of MlKitAnalyzer.

The list of detectors will be invoked sequentially in order.

When the targetCoordinateSystem is COORDINATE_SYSTEM_ORIGINAL, the output coordinate system is defined by ML Kit, which is the buffer with rotation applied. For example, if getHeight is h and the rotation is 90°, (0, 0) in the result maps to the pixel (0, h) in the original buffer.

The constructor throws IllegalArgumentException if Detector#getDetectorType() is TYPE_SEGMENTATION and targetCoordinateSystem is COORDINATE_SYSTEM_ORIGINAL. Currently ML Kit does not support transformation with segmentation.

Parameters
@NonNull List<Detector<Object>> detectors

list of ML Kit Detector.

int targetCoordinateSystem

e.g. COORDINATE_SYSTEM_ORIGINAL the coordinates in ML Kit output will be based on this value.

@NonNull Executor executor

on which the consumer is invoked.

@NonNull Consumer<MlKitAnalyzer.Result> consumer

invoked when there is a new ML Kit result.

Public methods

analyze

Added in 1.1.0
public final void analyze(@NonNull ImageProxy imageProxy)

Analyzes the image with the ML Kit Detectors.

This method forwards the image and the transformation Matrix to the Detectors. The Matrix is calculated based on the target coordinate system set in the constructor.

Usually this method is invoked by ImageAnalysis when a new frame is available.

See also
analyze

getDefaultTargetResolution

Added in 1.2.0
public final @NonNull Size getDefaultTargetResolution()

Implement this method to set a default target resolution for the .

Implement this method if the requires a specific resolution to work. The return value will be used as the default target resolution for the . Return null} if no falling back is needed. By default, this method returns null}.

If the app does not set a target resolution for , then this value will be used as the target resolution. If the has set a target resolution, e.g. if ImageAnalysis.Builder#setTargetResolution(Size) is called, then the will use the app value over this value.

Note that this method is invoked by CameraX at the time of binding to lifecycle. In order for this value to be effective, the has to be set before is bound to a lifecycle. Otherwise, the value will be ignored.

getTargetCoordinateSystem

Added in 1.1.0
public final int getTargetCoordinateSystem()

Implement this method to return the target coordinate system.

The coordinates detected by analyzing camera frame usually needs to be transformed. For example, in order to highlight a detected face, the app needs to transform the bounding box from the 's coordinate system to the View's coordinate system. This method allows the implementer to set a target coordinate system.

The value will be used by CameraX to calculate the transformation and forward it to the via #updateTransform. By default, this method returns ImageAnalysis#COORDINATE_SYSTEM_ORIGINAL.

For now, camera-core only supports ImageAnalysis#COORDINATE_SYSTEM_ORIGINAL, please see libraries derived from camera-core, for example, camera-view.

updateTransform

Added in 1.1.0
public final void updateTransform(@Nullable Matrix matrix)

Implement this method to receive the for coordinate transformation.

The value represents the transformation from the camera sensor to the target coordinate system defined in #getTargetCoordinateSystem(). It should be used by the implementation to transform the coordinates detected in the camera frame. For example, the coordinates of the detected face.

If the value is null}, it means that no valid transformation is available. It could have different causes depending on the value of #getTargetCoordinateSystem():

  • If the target coordinate system is #COORDINATE_SYSTEM_ORIGINAL, it is always invalid because in that case, the coordinate system depends on how the analysis algorithm processes the .
  • It is also invalid if the target coordinate system is not available, for example if the analyzer targets the viewfinder and the view finder is not visible in UI.

This method is invoked whenever a new transformation is ready. For example, when the view finder is first a launched as well as when it's resized.