Optimize your camera app on foldable devices with Jetpack WindowManager

1. Before you begin

What's special about foldables?

Foldables are once-in-a-generation innovations. They provide unique experiences, and with them come unique opportunities to delight your users with differentiated features like tabletop UI for hands-free usage.

Prerequisites

Basic knowledge of developing Android apps
Basic knowledge of Hilt Dependency Injection framework

What you'll build

In this codelab, you build a camera app with optimized layouts for foldable devices.

You start with a basic camera app that does not react to any device posture or take advantage of the better rear camera for enhanced selfies. You update the source code in order to move the preview to the smaller display when the device is unfolded and react to the phone being set in tabletop mode.

While the camera app is the most convenient use case for this API, both of the features you learn in this codelab can be applied to any app.

What you'll learn

How to use Jetpack Window Manager to react to posture changing
How to move your app to the smaller display of a foldable

What you'll need

A recent version of Android Studio
A foldable device or foldable emulator

2. Get set up

Get the starting code

If you have Git installed, you can simply run the command below. To check whether Git is installed, type git --version in the terminal or command line and verify that it executes correctly.

git clone https://github.com/android/large-screen-codelabs.git

Optional: If you do not have Git, you can click the following button to download all the code for this codelab:

Open the first module

In Android Studio, open the first module under /step1.

Screenshot of Android Studio showing the code related to this codelab

If you're asked to use the latest Gradle version, go ahead and update it.

3. Run and observe

Run the code on module step1.

As you can see, this is a simple camera app. You can toggle between the front and back camera and you can adjust the aspect ratio. However, the first button from the left currently does nothing—but it's going to be the entry point for the Rear Selfie mode.

Now, try to put the device in a half-opened position, in which the hinge is not completely flat or closed but forms a 90-degree angle.

As you can see, the app does not respond to different device postures and so layout does not change, leaving the hinge in the middle of the viewfinder.

4. Learn about the Jetpack WindowManager

The Jetpack WindowManager library helps app developers build optimized experiences for foldable devices. It contains the FoldingFeature class that describes a fold in a flexible display or a hinge between two physical display panels. Its API provides access to important information related to the device:

state() returns FLAT if the hinge is opened at 180 degrees or HALF_OPENED otherwise.
orientation() returns FoldingFeature.Orientation.HORIZONTAL if the FoldingFeature width is greater than the height; otherwise, returns FoldingFeature.Orientation.VERTICAL.
bounds() provides the boundaries of the FoldingFeature in a Rect format.

The FoldingFeature class contains additional information, like occlusionType() or isSeparating(), but this codelabe doesn't explore those in depth.

Starting from version 1.2.0-beta01, the library uses the WindowAreaController, an API that enables Rear Display Mode to move the current window to the display that is aligned with the rear camera, which is great for taking selfies with the rear camera and many other use cases!

Add dependencies

In order to use Jetpack WindowManager in your app, you need to add the following dependencies to your module-level build.gradle file:

step1/build.gradle

def work_version = '1.2.0'
implementation "androidx.window:window:$work_version"
implementation "androidx.window:window-java:$work_version"
implementation "androidx.window:window-core:$work_version"

Now you can access both the FoldingFeature and WindowAreaController classes in your app. You use them to build the ultimate foldable camera experience!

5. Implement the Rear Selfie mode

Start with Rear Display mode.

The API that allows this mode is the WindowAreaController, which provides the information and behavior around moving windows between displays or display areas on a device.

It lets you query the list of WindowAreaInfo that are currently available to be interacted with.

Using the WindowAreaInfo you can access the WindowAreaSession, an interface to represent an active window area feature and the status of availability for a specific WindowAreaCapability.

Declare these variables in your MainActivity:

step1/MainActivity.kt

private lateinit var windowAreaController: WindowAreaController
private lateinit var displayExecutor: Executor
private var rearDisplaySession: WindowAreaSession? = null
private var rearDisplayWindowAreaInfo: WindowAreaInfo? = null
private var rearDisplayStatus: WindowAreaCapability.Status =
    WindowAreaCapability.Status.WINDOW_AREA_STATUS_UNSUPPORTED
private val rearDisplayOperation = WindowAreaCapability.Operation.OPERATION_TRANSFER_ACTIVITY_TO_AREA

And initialize them in the onCreate() method:

step1/MainActivity.kt

displayExecutor = ContextCompat.getMainExecutor(this)
windowAreaController = WindowAreaController.getOrCreate()

lifecycleScope.launch(Dispatchers.Main) {
  lifecycle.repeatOnLifecycle(Lifecycle.State.STARTED) {
    windowAreaController.windowAreaInfos
      .map{info->info.firstOrNull{it.type==WindowAreaInfo.Type.TYPE_REAR_FACING}}
      .onEach { info -> rearDisplayWindowAreaInfo = info }
      .map{it?.getCapability(rearDisplayOperation)?.status?:  WindowAreaCapability.Status.WINDOW_AREA_STATUS_UNSUPPORTED }
      .distinctUntilChanged()
      .collect {
           rearDisplayStatus = it
           updateUI()
      }
  }
}

Now implement the updateUI() function to enable or disable the rear selfie button, depending on the current status:

step1/MainActivity.kt

private fun updateUI() {
    if(rearDisplaySession != null) {
        binding.rearDisplay.isEnabled = true
        // A session is already active, clicking on the button will disable it
    } else {
        when(rearDisplayStatus) {
            WindowAreaCapability.Status.WINDOW_AREA_STATUS_UNSUPPORTED -> {
                binding.rearDisplay.isEnabled = false
                // RearDisplay Mode is not supported on this device"
            }
            WindowAreaCapability.Status.WINDOW_AREA_STATUS_UNAVAILABLE -> {
                binding.rearDisplay.isEnabled = false
                // RearDisplay Mode is not currently available
            }
            WindowAreaCapability.Status.WINDOW_AREA_STATUS_AVAILABLE -> {
                binding.rearDisplay.isEnabled = true
                // You can enable RearDisplay Mode
            }
            WindowAreaCapability.Status.WINDOW_AREA_STATUS_ACTIVE -> {
                binding.rearDisplay.isEnabled = true
                // You can disable RearDisplay Mode
            }
            else -> {
                binding.rearDisplay.isEnabled = false
                // RearDisplay status is unknown
            }
        }
    }
}

This last step is optional, but it's very useful to learn all the possible states of a WindowAreaCapability.

Now implement the function toggleRearDisplayMode, which will close the session, if the capability is already active, or call the transferActivityToWindowArea function:

step1/CameraViewModel.kt

private fun toggleRearDisplayMode() {
    if(rearDisplayStatus == WindowAreaCapability.Status.WINDOW_AREA_STATUS_ACTIVE) {
        if(rearDisplaySession == null) {
            rearDisplaySession = rearDisplayWindowAreaInfo?.getActiveSession(rearDisplayOperation)
        }
        rearDisplaySession?.close()
    } else {
        rearDisplayWindowAreaInfo?.token?.let { token ->
            windowAreaController.transferActivityToWindowArea(
                token = token,
                activity = this,
                executor = displayExecutor,
                windowAreaSessionCallback = this
            )
        }
    }
}

Notice the usage of the MainActivity as a WindowAreaSessionCallback.

The Rear Display API works with a listener approach: when you request to move the content to the other display, you initiate a session that is returned through the listener's onSessionStarted() method. When you instead want to return to the inner (and bigger) display, you close the session, and you get a confirmation in the onSessionEnded() method. To create such a listener, you need to implement the WindowAreaSessionCallback interface.

Modify the MainActivity declaration so that it implements the WindowAreaSessionCallback interface:

step1/MainActivity.kt

class MainActivity : AppCompatActivity(), WindowAreaSessionCallback

Now, implement the onSessionStarted and onSessionEnded methods inside the MainActivity. Those callback methods are extremely useful to get notified of the session status and update the app accordingly.

But this time, for simplicity, just check in the function body if there are any errors and log the state.

step1/MainActivity.kt

override fun onSessionEnded(t: Throwable?) {
    if(t != null) {
        Log.d("Something was broken: ${t.message}")
    }else{
        Log.d("rear session ended")
    }
}

override fun onSessionStarted(session: WindowAreaSession) {
    Log.d("rear session started [session=$session]")
}

Build and run the app. If you then unfold your device and tap on the rear display button, you are prompted with a message like this:

Select "Switch screens now" to see your content moved to the outer display!

6. Implement the Tabletop mode

Now it's time to make your app fold-aware: you move your content on the side or above the hinge of the device based on the orientation of the fold. To do so, you'll be acting inside the FoldingStateActor so that your code is decoupled from the Activity for easier readability.

The core part of this API consists in the WindowInfoTracker interface, which is created with a static method that requires an Activity:

step1/CameraCodelabDependencies.kt

@Provides
fun provideWindowInfoTracker(activity: Activity) =
        WindowInfoTracker.getOrCreate(activity)

You don't need to write this code as it's already present, but it is useful to understand how the WindowInfoTracker is built.

To listen for any window change, listen to these changes in the onResume() method of your Activity:

step1/MainActivity.kt

lifecycleScope.launch {
    foldingStateActor.checkFoldingState(
         this@MainActivity, 
         binding.viewFinder
    )
}

Now, open the FoldingStateActor file, as it's time to fill in the checkFoldingState() method.

As you already saw, it runs in the RESUMED phase of your Activity, and it leverages the WindowInfoTracker in order to listen to any layout change.

step1/FoldingStateActor.kt

windowInfoTracker.windowLayoutInfo(activity)
      .collect { newLayoutInfo ->
         activeWindowLayoutInfo = newLayoutInfo
         updateLayoutByFoldingState(cameraViewfinder)
      }

By using the WindowInfoTracker interface, you can call windowLayoutInfo() in order to collect a Flow of WindowLayoutInfo that contains all the available information in DisplayFeature.

The last step is to react to these changes and move the content accordingly. You do this inside the updateLayoutByFoldingState() method, one step at a time.

Make sure the activityLayoutInfo contains some DisplayFeature properties, and that at least one of them is a FoldingFeature, otherwise you don't want to do anything:

step1/FoldingStateActor.kt

val foldingFeature = activeWindowLayoutInfo?.displayFeatures
            ?.firstOrNull { it is FoldingFeature } as FoldingFeature?
            ?: return

Calculate the position of the fold in order to make sure the device position is impacting your layout and is not outside the bounds of your hierarchy:

step1/FoldingStateActor.kt

val foldPosition = FoldableUtils.getFeaturePositionInViewRect(
            foldingFeature,
            cameraViewfinder.parent as View
        ) ?: return

Now, you're sure that you have a FoldingFeature that impacts your layout, so you need to move your content.

Check if the FoldingFeature is HALF_OPEN or else you just restore the position of your content. If it is HALF_OPEN, you need to run another check and act differently based on the orientation of the fold:

step1/FoldingStateActor.kt

if (foldingFeature.state == FoldingFeature.State.HALF_OPENED) {
    when (foldingFeature.orientation) {
        FoldingFeature.Orientation.VERTICAL -> {
            cameraViewfinder.moveToRightOf(foldPosition)
        }
        FoldingFeature.Orientation.HORIZONTAL -> {
            cameraViewfinder.moveToTopOf(foldPosition)
        }
    }
} else {
    cameraViewfinder.restore()
}

If the fold is VERTICAL, you move your content to the right, otherwise you move it on top of the fold position.

Build and run your app, and then unfold your device and place it in tabletop mode to see the content move accordingly!

7. Congratulations!

In this codelab you learned about some capabilities that are unique to foldable devices, such as the Rear Display Mode or Tabletop mode and how to unlock them by using Jetpack WindowManager.

You are ready to implement great user experiences for your camera app.

Reference

Camera preview

Optimize your camera app on foldable devices with Jetpack WindowManager

1. Before you begin

What's special about foldables?

Prerequisites

What you'll build

What you'll learn

What you'll need

2. Get set up

Get the starting code

Open the first module

3. Run and observe

4. Learn about the Jetpack WindowManager

Add dependencies

step1/build.gradle

5. Implement the Rear Selfie mode

step1/MainActivity.kt

step1/MainActivity.kt

step1/MainActivity.kt

step1/CameraViewModel.kt

step1/MainActivity.kt

step1/MainActivity.kt

6. Implement the Tabletop mode

step1/CameraCodelabDependencies.kt

step1/MainActivity.kt

step1/FoldingStateActor.kt

step1/FoldingStateActor.kt

step1/FoldingStateActor.kt

step1/FoldingStateActor.kt

7. Congratulations!

Further reading

Reference