# SurfaceProcessor

Artifact: [androidx.camera:camera-core](https://developer.android.com/jetpack/androidx/releases/camera) [View Source](https://cs.android.com/search?q=file:androidx/camera/core/SurfaceProcessor.java+class:androidx.camera.core.SurfaceProcessor) Added in [1.3.0](https://developer.android.com/jetpack/androidx/releases/camera#1.3.0)

*** ** * ** ***


```
public interface SurfaceProcessor
```

<br />

*** ** * ** ***

Interface to implement a GPU-based post-processing effect.

This interface is for implementing a GPU effect for the `https://developer.android.com/reference/androidx/camera/core/Preview``https://developer.android.com/reference/androidx/camera/core/UseCase`. Both the input and the output of the implementation are `https://developer.android.com/reference/android/view/Surface.html`s. It's recommended to use graphics API such as OpenGL or Vulkan to access the `https://developer.android.com/reference/android/view/Surface.html`.

If the implementation fails to process frames, for example, fails to allocate the resources, it should throw a `https://developer.android.com/reference/androidx/camera/core/ProcessingException` in either `https://developer.android.com/reference/androidx/camera/core/SurfaceProcessor#onInputSurface(androidx.camera.core.SurfaceRequest)` or `https://developer.android.com/reference/androidx/camera/core/SurfaceProcessor#onOutputSurface(androidx.camera.core.SurfaceOutput)` to notify CameraX. If the implementation encounters an error after the pipeline is running, it should invalidate the input `https://developer.android.com/reference/android/view/Surface.html` by calling `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest#invalidate()`, then throwing a `https://developer.android.com/reference/androidx/camera/core/ProcessingException` when `https://developer.android.com/reference/androidx/camera/core/SurfaceProcessor#onInputSurface(androidx.camera.core.SurfaceRequest)` is invoked again.

Once the implementation throws an exception, CameraX will treat it as an unrecoverable error and abort the pipeline. If the `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput#getTargets()` is `https://developer.android.com/reference/androidx/camera/core/CameraEffect#PREVIEW()`, CameraX will not propagate the error to the app. It's the implementation's responsibility to notify the app.

Since camera session closure is a complicated asynchronous process, some devices may pass a few frames for drawing even after the `resultListener` parameter from the `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest#provideSurface(android.view.Surface,java.util.concurrent.Executor,androidx.core.util.Consumer<androidx.camera.core.SurfaceRequest.Result>)` method reports that surface is safe for releasing. Users should ignore any new frames that come after CameraX reports surface can be released through the `resultListener` parameter.

See the following sample code for further reference.

```
class SurfaceProcessorImpl implements SurfaceProcessor, SurfaceTexture.OnFrameAvailableListener {

    Consumer<Exception> mErrorListener;
    List<SurfaceTexture> mReleasedSurfaceTextures = new ArrayList<>();

    SurfaceProcessorImpl(@NonNull Consumer<Exception> errorListener) {
        mErrorListener = errorListener;
    }

    void onInputSurface(@NonNull SurfaceRequest request) throws ProcessingException {
        try {
            // Setup the input stream, e.g. a Surface and a SurfaceTexture.

            request.provideSurface(surface, mGlExecutor, result -> {
                surfaceTexture.setOnFrameAvailableListener(null);
                surfaceTexture.release();
                surface.release();
                mReleasedSurfaceTextures.add(surfaceTexture);
            });

            surfaceTexture.setOnFrameAvailableListener(this, mGlHandler);
        } catch (Exception e) {
            // Notify the app before throwing a ProcessingException.
            mErrorListener.accept(e)
            throw new ProcessingException(e);
        }
    }

    void onOutputSurface(@NonNull SurfaceRequest request) throws ProcessingException {
        try {
            // Setup the output streams.
        } catch (Exception e) {
            // Notify the app before throwing a ProcessingException.
            mErrorListener.accept(e)
            throw new ProcessingException(e);
        }
    }

    @Override
    public void onFrameAvailable(@NonNull SurfaceTexture surfaceTexture) {
        if (mReleasedSurfaceTextures.contains(surfaceTexture)) {
            // Ignore frame update if this surface texture is already released.
            return;
        }

        // Draw frame from the surfaceTexture to output surface(s).
    }
}
```

## Summary

| ### Public methods |
|---|---|
| `abstract void` | `https://developer.android.com/reference/androidx/camera/core/SurfaceProcessor#onInputSurface(androidx.camera.core.SurfaceRequest)(@https://jspecify.dev/docs/api/org/jspecify/annotations/NonNull.html https://developer.android.com/reference/androidx/camera/core/SurfaceRequest request)` Invoked when CameraX requires an input `https://developer.android.com/reference/android/view/Surface.html` for reading original frames. |
| `abstract void` | `https://developer.android.com/reference/androidx/camera/core/SurfaceProcessor#onOutputSurface(androidx.camera.core.SurfaceOutput)(@https://jspecify.dev/docs/api/org/jspecify/annotations/NonNull.html https://developer.android.com/reference/androidx/camera/core/SurfaceOutput surfaceOutput)` Invoked when CameraX provides output Surface(s) for drawing processed frames. |

## Public methods

### onInputSurface

Added in [1.3.0](https://developer.android.com/jetpack/androidx/releases/camera#1.3.0)

```
abstract void onInputSurface(@NonNull SurfaceRequest request)
```

Invoked when CameraX requires an input `https://developer.android.com/reference/android/view/Surface.html` for reading original frames.

CameraX requests `https://developer.android.com/reference/android/view/Surface.html`s when the upstream pipeline is reconfigured. For example, when `https://developer.android.com/reference/androidx/camera/core/UseCase`s are bound to lifecycle.

With OpenGL, the implementation should create a `https://developer.android.com/reference/android/view/Surface.html` backed by `https://developer.android.com/reference/android/graphics/SurfaceTexture.html` with the size of `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest#getResolution()`, then listen for the `https://developer.android.com/reference/android/graphics/SurfaceTexture.html#setOnFrameAvailableListener(android.graphics.SurfaceTexture.OnFrameAvailableListener)` to get the incoming frames. The `https://developer.android.com/reference/android/view/Surface.html` should not be released until the callback provided in `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest#provideSurface(android.view.Surface,java.util.concurrent.Executor,androidx.core.util.Consumer<androidx.camera.core.SurfaceRequest.Result>)` is invoked. CameraX may request new input `https://developer.android.com/reference/android/view/Surface.html` before releasing the existing one.

If the implementation encounters errors in creating the input `https://developer.android.com/reference/android/view/Surface.html`, it should throw an `https://developer.android.com/reference/androidx/camera/core/ProcessingException` to notify CameraX.

The implementation can replace a previously provided `https://developer.android.com/reference/android/view/Surface.html` by invoking `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest#invalidate()`. Once invoked, CameraX will restart the camera pipeline and call `https://developer.android.com/reference/androidx/camera/core/SurfaceProcessor#onInputSurface(androidx.camera.core.SurfaceRequest)` again with another `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest`.

The value of the `https://developer.android.com/reference/android/graphics/SurfaceTexture.html#getTransformMatrix(float[])` will need an additional transformation. CameraX calculates the additional transformation based on `https://developer.android.com/reference/androidx/camera/core/UseCase` configurations such as `https://developer.android.com/reference/androidx/camera/core/ViewPort` and target rotation, and provide the value via `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput#updateTransformMatrix(float[],float[])`. Code sample:

```
// Create Surface based on the request.
SurfaceTexture surfaceTexture = SurfaceTexture(textureName);
surfaceTexture.setDefaultBufferSize(request.resolution.width, request.resolution.height);
Surface surface = Surface(surfaceTexture);

// Provide the Surface to CameraX, and cleanup when it's no longer used.
surfaceRequest.provideSurface(surface, executor, result -> {
    surfaceTexture.setOnFrameAvailableListener(null)
    surfaceTexture.release()
    surface.release()
});

// Listen to the incoming frames.
surfaceTexture.setOnFrameAvailableListener(surfaceTexture -> {
    // Process the incoming frames and draw to the output Surface from #onOutputSurface
}, handler);
```

| Parameters |
|---|---|
| `@https://jspecify.dev/docs/api/org/jspecify/annotations/NonNull.html https://developer.android.com/reference/androidx/camera/core/SurfaceRequest request` | a request to provide `https://developer.android.com/reference/android/view/Surface.html` for input. |

| Throws |
|---|---|
| `https://developer.android.com/reference/androidx/camera/core/ProcessingException` | if the implementation fails to fulfill the `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest`. |

| See also |
|---|---|
| `https://developer.android.com/reference/androidx/camera/core/SurfaceRequest` |   |

### onOutputSurface

Added in [1.3.0](https://developer.android.com/jetpack/androidx/releases/camera#1.3.0)

```
abstract void onOutputSurface(@NonNull SurfaceOutput surfaceOutput)
```

Invoked when CameraX provides output Surface(s) for drawing processed frames.

CameraX provides the output `https://developer.android.com/reference/android/view/Surface.html`s when the downstream pipeline is reconfigured, for example, when `https://developer.android.com/reference/androidx/camera/core/UseCase`s are bound or the preview viewfinder is reset.

The provided `https://developer.android.com/reference/android/view/Surface.html`s are for drawing processed frames. The implementation must get the `https://developer.android.com/reference/android/view/Surface.html` via `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput#getSurface(java.util.concurrent.Executor,androidx.core.util.Consumer<androidx.camera.core.SurfaceOutput.Event>)` and provide a `https://developer.android.com/reference/androidx/core/util/Consumer` listening to the end-of-life event of the `https://developer.android.com/reference/android/view/Surface.html`. Then, the implementation should call `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput#close()` after it stops drawing to the `https://developer.android.com/reference/android/view/Surface.html`. CameraX may provide new output `https://developer.android.com/reference/android/view/Surface.html` before requesting to close the existing one.

If the implementation encounters an error and cannot consume the `https://developer.android.com/reference/android/view/Surface.html`, it should throw an `https://developer.android.com/reference/androidx/camera/core/ProcessingException` to notify CameraX.

When drawing to the `https://developer.android.com/reference/android/view/Surface.html`, the implementation should apply an additional transformation to the input `https://developer.android.com/reference/android/view/Surface.html` by calling `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput#updateTransformMatrix(float[],float[])` with the value of `https://developer.android.com/reference/android/graphics/SurfaceTexture.html#getTransformMatrix(float[])`} from the input `https://developer.android.com/reference/android/view/Surface.html`.

| Parameters |
|---|---|
| `@https://jspecify.dev/docs/api/org/jspecify/annotations/NonNull.html https://developer.android.com/reference/androidx/camera/core/SurfaceOutput surfaceOutput` | contains a `https://developer.android.com/reference/android/view/Surface.html` for drawing processed frames. |

| Throws |
|---|---|
| `https://developer.android.com/reference/androidx/camera/core/ProcessingException` | if the implementation fails to consume the `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput`. |

| See also |
|---|---|
| `https://developer.android.com/reference/androidx/camera/core/SurfaceOutput` |   |