Input compatibility on large screens

On large screen devices, users often interact with apps using a keyboard, mouse, trackpad, stylus, or gamepad. To enable your app to accept input from external devices, do the following:

  • Test basic keyboard support, such as Ctrl+Z for undo, Ctrl+C for copy, and Ctrl+S for save. See Handle keyboard actions for a list of default keyboard shortcuts.
  • Test advanced keyboard support, for example, Tab key and arrow key keyboard navigation, Enter key text entry confirmation, and Spacebar play and pause in media apps.
  • Test basic mouse interactions, including right-click for context menu, icon changes on hover, and mouse wheel or trackpad scroll events on custom components.
  • Test app-specific input devices such as a stylus, game controllers, and music app MIDI controllers.
  • Consider advanced input support that could make the app stand out in desktop environments, for example, touchpad as a cross‑fader for DJ apps, mouse capture for games, and keyboard shortcuts for keyboard‑centric users.

Keyboard

The way your app responds to keyboard input contributes to the large screen user experience. There are three kinds of keyboard input: navigation, keystrokes, and shortcuts.

Keyboard navigation is rarely implemented in touch‑centric apps, but users expect it when they're using an app and have their hands on a keyboard. Keyboard navigation can be essential on phones, tablets, foldables, and desktop devices for users with accessibility needs.

For many apps, arrow key and tab navigation are handled automatically by the Android framework. For example, a Button is focusable by default, and keyboard navigation should generally work without any additional code. To enable keyboard navigation for views that are not focusable by default, mark them as focusable, which can be done programmatically or in XML:

Kotlin

yourView.isFocusable = true

Java

yourView.setFocusable(true);

Alternatively you can set the focusable attribute in your layout file:

android:focusable="true"

To learn more, see Focus Handling.

When focus is enabled, the Android framework creates a navigational mapping for all focusable views based on their position. This usually works as expected and no further development is needed. When the default mapping is not correct for an app's needs, the mapping can be overridden as follows:

Kotlin

// Arrow keys
yourView.nextFocusLeftId = R.id.view_to_left
yourView.nextFocusRightId = R.id.view_to_right
yourView.nextFocusTopId = R.id.view_above
yourView.nextFocusBottomId = R.id.view_below
// Tab key
yourView.nextFocusForwardId = R.id.next_view

Java

// Arrow keys
yourView.setNextFocusLeftId(R.id.view_to_left);
yourView.setNextFocusRightId(R.id.view_to_left);
yourView.setNextFocusTopId(R.id.view_to_left);
yourView.setNextFocusBottomId(R.id.view_to_left);
// Tab key
yourView.setNextFocusForwardId(R.id.next_view);

Test access to every UI element of your app using the keyboard only. Frequently used elements should be accessible without mouse or touch input.

Remember, keyboard support might be essential for users with accessibility needs.

Keystrokes

For text input that would be handled by an on screen virtual keyboard (IME), such as for an EditText , apps should behave as expected on large screen devices with no additional development work. For keystrokes that cannot be anticipated by the framework, apps need to handle the behavior themselves. This is especially true for apps with custom views.

Some examples are chat apps that use the Enter key to send a message, media apps that start and stop playback with the Spacebar, and games that control movement with the w, a, s, and d keys.

Most apps override the onKeyUp() callback and add the expected behavior for each received keycode:

Kotlin

override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean {
    return when (keyCode) {
        KeyEvent.KEYCODE_ENTER -> {
            sendChatMessage()
            true
        }
        KeyEvent.KEYCODE_SPACE -> {
            playOrPauseMedia()
            true
        }
        else -> super.onKeyUp(keyCode, event)
    }
}

Java

@Override
public boolean onKeyUp(int keyCode, KeyEvent event) {
    if (keyCode == KeyEvent.KEYCODE_ENTER) {
        sendMessage();
        return true;
    } else if (KeyEvent.KEYCODE_SPACE){
        playOrPauseMedia();
        return true;
    } else {
        return super.onKeyUp(keyCode, event);
    }
}

An onKeyUp event occurs when a key is released. Using the callback prevents apps from needing to process multiple onKeyDown events if a key is held down or released slowly. Games and apps that need to detect the moment a key is pressed or whether the user is holding a key down can listen for the onKeyDown event and handle repeated onKeyDown events themselves.

For more information, see Handle keyboard actions.

Shortcuts

Common keyboard shortcuts that include the Ctrl, Alt, Shift, and Meta keys are expected when using a hardware keyboard. If an app doesn't implement shortcuts, the experience can feel frustrating to users. Advanced users also appreciate shortcuts for frequently used app‑specific tasks. Shortcuts make an app easier to use and differentiate it from apps that don't have shortcuts.

Some common shortcuts include Ctrl+S (save), Ctrl+Z (undo), and Ctrl+Shift+Z (redo). For a list of default shortcuts, see Handle keyboard actions.

Shortcuts can be enabled by implementing dispatchKeyShortcutEvent() to intercept all key combinations (Alt, Ctrl, Shift, and Meta) for a given keycode. To check for a specific modifier key, use:

Kotlin

override fun dispatchKeyShortcutEvent(event: KeyEvent): Boolean {
  return when (event.keyCode) {
    KeyEvent.KEYCODE_O -> {
      openFile() // Ctrl+O, Shift+O, Alt+O
      true
    }
    KeyEvent.KEYCODE_Z-> {
      if (event.isCtrlPressed) {
        if (event.isShiftPressed) {
          redoLastAction() // Ctrl+Shift+Z pressed
          true
        } else {
          undoLastAction() // Ctrl+Z pressed
          true
        }
      }
    }
    else -> {
      return super.dispatchKeyShortcutEvent(event)
    }
  }
}

Java

@Override
public boolean dispatchKeyShortcutEvent(KeyEvent event) {
  if (event.getKeyCode() == KeyEvent.KEYCODE_O) {
      openFile(); // Ctrl+O, Shift+O, Alt+O
      return true;
  } else if(event.getKeyCode() == KeyEvent.KEYCODE_Z) {
      if (event.isCtrlPressed()) {
          if (event.isShiftPressed()) {
              redoLastAction();
              return true;
          }
          else {
              undoLastAction();
              return true;
          }
      }
  }
  return super.dispatchKeyShortcutEvent(event);
}

Separating shortcut code from other keystroke handling (such as onKeyUp() and onKeyDown()) accepts modifier keys by default without having to manually implement modifier key checks in every case. Allowing all modifier key combinations can also be more convenient for users who are accustomed to different keyboard layouts and operating systems.

However, you can also implement shortcuts in onKeyUp() by checking for KeyEvent.isCtrlPressed(), KeyEvent.isShiftPressed(), or KeyEvent.isAltPressed(). This can be easier to maintain if the modified key behavior is more of a change to an app behavior than a shortcut. For example, in games when W means "walk forward" and Shift+W means "run forward".

Kotlin

override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean {
  return when(keyCode) {
    KeyEvent.KEYCODE_W-> {
      if (event.isShiftPressed) {
        if (event.isCtrlPressed) {
          flyForward() // Ctrl+Shift+W pressed
          true
        } else {
          runForward() // Shift+W pressed
          true
        }
      } else {
        walkForward() // W pressed
        true
      }
    }
    else -> super.onKeyUp(keyCode, event)
  }
}

Java

@Override
public boolean onKeyUp(int keyCode, KeyEvent event) {
    if (keyCode == KeyEvent.KEYCODE_W) {
        if (event.isShiftPressed()) {
            if (event.isCtrlPressed()) {
                flyForward(); // Ctrl+Shift+W pressed
                return true;
            } else {
                runForward(); // Shift+W pressed
                return true;
            }
        } else {
            walkForward();
            return true;
        }
    }
    return super.onKeyUp(keyCode, event);
}

See also Keyboard Shortcuts Helper.

Stylus

Many large screen devices come with a stylus. Android apps handle styluses as touchscreen input. Some devices might also have a USB or Bluetooth drawing table, like the Wacom Intuos. Android apps can receive bluetooth input but not USB input.

A stylus event is reported as a touchscreen event by View#onTouchEvent() or View#onGenericMotionEvent() and contains a MotionEvent#getSource() of type SOURCE_STYLUS.

The MotionEvent object contains information about the event:

Historical points

Android batches input events and delivers them once per frame. A stylus can report events at much higher frequencies than the display. When creating drawing apps, check for events that may be in the recent past by using the getHistorical APIs:

Palm rejection

When users draw, write, or interact with your app using a stylus, they sometimes touch the screen with the palm of their hands. The touch event (set to ACTION_DOWN or ACTION_POINTER_DOWN) can be reported to your app before the system recognizes and disregards the inadvertent palm touch.

Android cancels palm touch events by dispatching a MotionEvent. If your app receives ACTION_CANCEL, cancel the gesture. If your app receives ACTION_POINTER_UP, check whether FLAG_CANCELED is set. If so, cancel the gesture.

Do not check for just FLAG_CANCELED. On Android 13 (API level 33) and higher, the system sets FLAG_CANCELED for ACTION_CANCEL events, but the system does not set the flag on lower Android versions.

Android 12

On Android 12 (API level 32) and lower, detection of palm rejection is possible only for single‑pointer touch events. If a palm touch is the only pointer, the system cancels the event by setting ACTION_CANCEL on the motion event object. If other pointers are down, the system sets ACTION_POINTER_UP, which is insufficient for detecting palm rejection.

Android 13

On Android 13 (API level 33) and higher, if a palm touch is the only pointer, the system cancels the event by setting ACTION_CANCEL and FLAG_CANCELED on the motion event object. If other pointers are down, the system sets ACTION_POINTER_UP and FLAG_CANCELED.

Whenever your app receives a motion event with ACTION_POINTER_UP, check for FLAG_CANCELED to determine whether the event indicates palm rejection (or other event cancellation).

Note-taking apps

ChromeOS has a special intent that surfaces registered note‑taking apps to users. To register an app as a noteὝtaking app, add the following to your app manifest:

<intent-filter>
    <action android:name="org.chromium.arc.intent.action.CREATE_NOTE" />
    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>

When an app is registered with the system, the user can select it as the default note‑taking app. When a new note is requested, the app should create an empty note ready for stylus input. When the user wishes to annotate an image (such as a screenshot or downloaded image), the app launches with ClipData containing one or more items with content:// URIs. The app should create a note that uses the first attached image as a background image and enter a mode where the user can draw on the screen with a stylus.

Test note-taking intents without a stylus

[TBD remove section.]

To test whether an app responds correctly to note‑taking intents without an active stylus, use the following method to display the note‑taking options on ChromeOS:

  1. Switch to dev mode and make the device writable
  2. Press Ctrl+Alt+F2 to open a terminal
  3. Run the command sudo vi /etc/chrome_dev.conf
  4. Press i to edit and add --ash-enable-palette to a new line at the end of the file
  5. Save by pressing Esc and then typing :, w, q and pressing Enter
  6. Press Ctrl+Alt+F1 to return to the regular ChromeOS UI
  7. Log out, then log back in

A stylus menu should now be on the shelf:

  • Tap the stylus button on the shelf and choose New note. This should open a blank drawing note.
  • Take a screenshot. From the shelf, select stylus button > Capture screen or download an image. There should be the option to Annotate image in the notification. This should launch the app with the image ready to be annotated.

Mouse and touchpad support

Most apps generally need to handle only three large screen–centric events: right-click, hover, and drag and drop.

Right-click

Any actions that cause an app to show a context menu, like touch & hold on a list item, should also react to right‑click events.

To handle right‑click events, apps should register a View.OnContextClickListener:

Kotlin

yourView.setOnContextClickListener {
    showContextMenu()
    true
}

Java

yourView.setOnContextClickListener(v -> {
    showContextMenu();
    return true;
});

For details on constructing context menus, see Create a contextual menu.

Hover

You can make your app layouts feel polished and easier to use by handling hover events. This is especially true for custom views:

Kotlin

// Change the icon to a "hand" pointer on hover.
// Highlight the view by changing the background.
yourView.setOnHoverListener { view, _ ->
    addVisualHighlighting(true)
    view.pointerIcon =
        PointerIcon.getSystemIcon(view.context, PointerIcon.TYPE_HAND)
    true // Listener consumes the event.
}

Java

// Change the icon to a "hand" pointer on hover.
// Highlight the view by changing the background.
yourView.setOnHoverListener((view, event) -> {
    addVisualHighlighting(true);
    view.setPointerIcon(
        PointerIcon.getSystemIcon(view.getContext(), PointerIcon.TYPE_HAND)
    );
    return true; // Listener consumes the event.
});

The two most common examples of this are:

  • Indicating to users whether an element has interactive behavior, such as being clickable or editable, by changing the mouse pointer icon
  • Adding visual feedback to items in a large list or grid when the pointer is hovering over them

Drag and drop

In a multi-window environment, users expect to be able to drag and drop items between apps. This is true for desktop devices as well as tablets, phones, and foldables in split‑screen mode.

Consider whether users are likely to drag items into your app. For example, photo editors should expect to receive photos, audio players should expect to receive audio files, and drawing programs should expect to receive photos.

To add drag and drop support, see Enable drag and drop , and take a look at the Android on ChromeOS — Implementing Drag & Drop blog post.

Special considerations for ChromeOS

Advanced pointer support

Apps that do advanced handling of mouse and touchpad input should implement a View#onGenericMotionEvent() and use [MotionEvent.getSource()][] to distinguish between SOURCE_MOUSE and SOURCE_TOUCHSCREEN.

Examine the MotionEvent object to implement the required behavior:

  • Movement generates ACTION_HOVER_MOVE events.
  • Buttons generate ACTION_BUTTON_PRESS and ACTION_BUTTON_RELEASE events. You can also check the current state of all mouse and trackpad buttons using getButtonState().
  • Mouse wheel scrolling generates ACTION_SCROLL events.

Game controllers

Some large screen Android devices support up to four game controllers. Use the standard Android game controller APIs to handle game controllers (see Support game controllers).

Game controller buttons are mapped to common values following a common mapping. But not all game controller manufacturers follow the same mapping conventions. You can provide a much better experience if you allow users to select different popular controller mappings. See Process gamepad button presses for more information.

Input translation mode

ChromeOS enables an input translation mode by default. For most Android apps, this mode helps apps work as expected in a desktop environment. Some examples include automatically enabling two‑finger scrolling on the touchpad, mouse wheel scrolling, and mapping raw display coordinates to window coordinates. Generally, app developers don't need to implement any of these behaviors themselves.

If an app implements custom input behavior, for example defining a custom two‑finger touchpad pinch action, or these input translations don't provide the input events expected by the app, you can disable the input translation mode by adding the following tag to the Android manifest:

<uses-feature
    android:name="android.hardware.type.pc"
    android:required="false" />

Additional resources