OverlayEffect


class OverlayEffect : CameraEffect, AutoCloseable


A CameraEffect for drawing overlay on top of the camera frames.

This class manages and processes camera frames with OpenGL. Upon arrival, frames are enqueued into an array of GL textures for deferred rendering. Calling drawFrameAsync dequeues frames and renders them to the output. Additionally, when the texture queue reaches its capacity, the oldest frame is automatically dequeued and rendered. The size of the texture queue can be defined in the constructor.

The queuing mechanism provides the flexibility to postpone frame rendering until analysis results are available. For instance, to highlight on a QR code in a preview, one can apply a QR code detection algorithm using ImageAnalysis. Once the frame's analysis result is ready, invoke drawFrameAsync and pass in the getTimestamp to release the frame and draw overlay. If the app doesn't render real-time analysis results, set the queue depth to 0 to avoid unnecessary buffer copies. For example, when laying over a static watermark.

Prior to rendering a frame, the OverlayEffect invokes the listener set in setOnDrawListener. This listener provides a Frame object, which contains both a Canvas object for drawing the overlay and the metadata like crop rect, rotation degrees, etc to calculate how the overlay should be positioned. Once the listener returns, OverlayEffect updates the SurfaceTexture behind the Canvas and blends it with the camera frame before rendering to the output.

This class is thread-safe. The methods can be invoked on any thread. The app provides a Handler object in the constructor which is used for listening for Surface updates, performing OpenGL operations and invoking app provided listeners.

Summary

Constants

const Int

The drawFrameAsync call failed because the caller cancelled the drawing.

const Int

The drawFrameAsync call failed because the frame with the exact timestamp was not found in the queue.

const Int

The drawFrameAsync call failed because the output surface is missing, or the output surface no longer matches the frame.

const Int

The drawFrameAsync call was successful.

Public constructors

OverlayEffect(
    targets: Int,
    queueDepth: Int,
    handler: Handler,
    errorListener: Consumer<Throwable!>
)

Creates an OverlayEffect.

Public functions

Unit

Clears the listener set in setOnDrawListener.

Unit

Closes the OverlayEffect.

ListenableFuture<Int!>
drawFrameAsync(timestampNs: Long)

Draws the queued frame with the given timestamp.

Handler

Gets the Handler set in the constructor.

Int

Gets the depth of the queue set in the constructor.

Unit
setOnDrawListener(onDrawListener: Function<Frame!, Boolean!>)

Sets the listener for drawing overlay.

Inherited Constants

From androidx.camera.core.CameraEffect
const Int

Bitmask option to indicate that CameraX should apply this effect to ImageCapture.

const Int

Bitmask option to indicate that CameraX should apply this effect to Preview.

const Int

Bitmask option to indicate that CameraX should apply this effect to VideoCapture.

Inherited functions

From androidx.camera.core.CameraEffect
Consumer<Throwable!>

Gets the Error listener associated with this effect.

Executor

Gets the Executor associated with this effect.

SurfaceProcessor?

Gets the SurfaceProcessor associated with this effect.

Int

Ges the target UseCases of this effect.

Constants

RESULT_CANCELLED_BY_CALLER

Added in 1.4.0
const val RESULT_CANCELLED_BY_CALLER = 4: Int

The drawFrameAsync call failed because the caller cancelled the drawing. This happens when the listener in setOnDrawListener returned false.

RESULT_FRAME_NOT_FOUND

Added in 1.4.0
const val RESULT_FRAME_NOT_FOUND = 2: Int

The drawFrameAsync call failed because the frame with the exact timestamp was not found in the queue. It could be one of the following reasons:

  • the timestamp provided was incorrect, or
  • the frame was removed because drawFrameAsync had been called with a newer timestamp, or
  • the frame was removed due to the queue being full.
If it's the last case, the caller may avoid this issue by increasing the queue depth.

RESULT_INVALID_SURFACE

Added in 1.4.0
const val RESULT_INVALID_SURFACE = 3: Int

The drawFrameAsync call failed because the output surface is missing, or the output surface no longer matches the frame. It can happen when the output Surface is replaced or disabled. For example, when the UseCase was unbound.

RESULT_SUCCESS

Added in 1.4.0
const val RESULT_SUCCESS = 1: Int

The drawFrameAsync call was successful. The frame with the exact timestamp was drawn to the output surface.

Public constructors

OverlayEffect

Added in 1.4.0
OverlayEffect(
    targets: Int,
    queueDepth: Int,
    handler: Handler,
    errorListener: Consumer<Throwable!>
)

Creates an OverlayEffect.

Parameters
targets: Int

The targets the effect applies to. For example, PREVIEW | VIDEO_CAPTURE. See addEffect for supported targets combinations.

queueDepth: Int

The depth of the queue. This value indicates how many frames can be queued before the oldest frame being automatically released. OverlayEffect allocates an array of OpenGL 2D textures that matches this size. The maximum value of the queueDepth depends on the size of the image and the device capabilities. Set a larger value if an ImageAnalysis processing takes a long time to produce a result to be used for overlay, so the frame is not auto-released before the result is ready. If the queue depth is 0, the input frames are rendered immediately without queuing.

handler: Handler

The Handler for listening for the input Surface updates and for performing OpenGL operations.

errorListener: Consumer<Throwable!>

invoked if the effect runs into unrecoverable errors. The Throwable will be the error thrown by this CameraEffect. For example, ProcessingException. This is invoked on the provided {@param Handler}.

Public functions

clearOnDrawListener

Added in 1.4.0
fun clearOnDrawListener(): Unit

Clears the listener set in setOnDrawListener.

close

Added in 1.4.0
fun close(): Unit

Closes the OverlayEffect.

Once closed, the OverlayEffect can no longer be used.

drawFrameAsync

Added in 1.4.0
fun drawFrameAsync(timestampNs: Long): ListenableFuture<Int!>

Draws the queued frame with the given timestamp.

Once invoked, OverlayEffect retrieves the queued frame with the given timestamp and draws it to the output Surface. If the frame is successfully drawn, ListenableFuture completes with RESULT_SUCCESS. Otherwise, it completes with one of the following results: RESULT_FRAME_NOT_FOUND, RESULT_INVALID_SURFACE or RESULT_CANCELLED_BY_CALLER. If this method is called after the OverlayEffect is released, the ListenableFuture completes with an IllegalStateException.

This method is thread safe. When calling from the getHandler thread, it's executed right away; otherwise, it posts the execution on the getHandler. It's recommended to call this method from the getHandler thread to avoid thread hopping.

getHandler

Added in 1.4.0
fun getHandler(): Handler

Gets the Handler set in the constructor.

getQueueDepth

Added in 1.4.0
fun getQueueDepth(): Int

Gets the depth of the queue set in the constructor.

setOnDrawListener

Added in 1.4.0
fun setOnDrawListener(onDrawListener: Function<Frame!, Boolean!>): Unit

Sets the listener for drawing overlay.

Each time before OverlayEffect draws a frame to the output, the listener receives a Frame object, which contains the necessary APIs for drawing overlay.

To draw an overlay, first call getOverlayCanvas ()} to get a Canvas object. The Canvas object is backed by a SurfaceTexture with the size of getSize. getSensorToBufferTransform represents the mapping from camera sensor coordinates to the frame's coordinates. To draw objects in the sensor coordinates, call setMatrix with the value of getSensorToBufferTransform.

Once the drawing is done, the listener should return true for the OverlayEffect to draw it to the output Surface. If it returns false, the frame will be dropped.

OverlayEffect invokes the listener on the getHandler thread.

See also
Frame