On large screen devices, users more often interact with apps using a keyboard, mouse, trackpad, stylus, or gamepad. To enable your app to accept input from external devices, do the following:
- Test basic keyboard support such as Tab and arrow key keyboard navigation, Enter key text entry confirmation, and Space bar play/pause in media apps
- Add standard keyboard shortcuts where applicable; for example, Ctrl + Z for undo and Ctrl + S for save
- Test basic mouse interactions in the manner of right-click for context menu, icon changes on hover, and mouse wheel or trackpad scroll events on custom views
- Test app-specific input devices such as a stylus for drawing apps, game controllers for games, and MIDI controllers for music apps
- Consider advanced input support that could make the app stand out in desktop environments; for example, touchpad as a cross-fader for DJ apps, mouse capture for games, and extensive keyboard shortcuts for keyboard‑centric users
Keyboard
The way your app responds to keyboard input contributes to a good large screen experience. There are three kinds of keyboard input: navigation, keystrokes, and shortcuts.
Navigation
Keyboard navigation is rarely implemented in touch-centric apps, but users expect it when they are using an app and have their hands on a keyboard. It can also be essential for users with accessibility needs on phones, tablets, foldables, and desktop devices.
For many apps, simple arrow key and tab navigation is all that is needed and is
mostly handled automatically by the Android framework. For example, a view of a
Button
is focusable by default, and keyboard navigation should generally work
without any additional code. To enable keyboard navigation for views that are
not focusable by default, developers should mark them as focusable, which can
be done programmatically or in XML, as shown below. See
Focus Handling
for more information.
Kotlin
yourView.isFocusable = true
Java
yourView.setFocusable(true);
Alternatively you can set the focusable
attribute in your layout file:
android:focusable="true"
Once focus is enabled, the Android framework creates a navigational mapping for all focusable views based on their position. This usually works as expected and no further work is needed. When the default mapping is not correct for an app's needs, it can be overridden as follows:
Kotlin
// Arrow keys yourView.nextFocusLeftId = R.id.view_to_left yourView.nextFocusRightId = R.id.view_to_right yourView.nextFocusTopId = R.id.view_above yourView.nextFocusBottomId = R.id.view_below // Tab key yourView.nextFocusForwardId = R.id.next_view
Java
// Arrow keys yourView.setNextFocusLeftId(R.id.view_to_left); yourView.setNextFocusRightId(R.id.view_to_left); yourView.setNextFocusTopId(R.id.view_to_left); yourView.setNextFocusBottomId(R.id.view_to_left); // Tab key yourView.setNextFocusForwardId(R.id.next_view);
It is good practice to try to access every piece of your app's functionality before each release using the keyboard only. It should be easy to access the most common actions without mouse or touch input.
Remember, keyboard support might be essential for users with accessibility needs.
Keystrokes
For text input that would be handled by an on screen virtual keyboard
(IME) such as an EditText
, apps
should behave as expected on large screen devices with no additional work from
the developer. For keystrokes that cannot be anticipated by the framework, apps
need to handle the behavior themselves. This is especially true for apps with
custom views.
Some examples are chat apps that use the enter key to send a message, media apps that start and stop playback with the space key, and games that control movement with the w, a, s, and d keys.
Most apps override the
onKeyUp()
callback and add the expected behavior for each received keycode, as shown
below:
Kotlin
override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean { return when (keyCode) { KeyEvent.KEYCODE_ENTER -> { sendChatMessage() true } KeyEvent.KEYCODE_SPACE -> { playOrPauseMedia() true } else -> super.onKeyUp(keyCode, event) } }
Java
@Override public boolean onKeyUp(int keyCode, KeyEvent event) { if (keyCode == KeyEvent.KEYCODE_ENTER) { sendMessage(); return true; } else if (KeyEvent.KEYCODE_SPACE){ playOrPauseMedia(); return true; } else { return super.onKeyUp(keyCode, event); } }
An onKeyUp
event occurs when a key is released. Using this callback prevents
apps from needing to process multiple onKeyDown
events if a key is held down
or released slowly. Games and apps that want to know the moment a key is
pressed or that expect users to hold down keyboard keys can look for the
onKeyDown()
event and handle the repeated onKeyDown
events themselves.
For more information on providing keyboard support, see Handle keyboard actions.
Shortcuts
Common Ctrl, Alt, and Shift-based shortcuts are expected when using a hardware keyboard. If an app does not implement them, the experience can feel frustrating to users. Advanced users also appreciate shortcuts for frequently used app-specific tasks. Shortcuts make an app easier to use and differentiate it from apps that don't have shortcuts.
Some common shortcuts include Ctrl + S (save), Ctrl + Z (undo), and Ctrl + Shift + Z (redo). For an example of some more advanced shortcuts, see the list of VLC Media Player shortcut keys.
Shortcuts can be implemented using
dispatchKeyShortcutEvent()
.
This intercepts all meta-key combinations (Alt, Ctrl, and Shift) for a given
keycode. To check for a specific meta-key, use
KeyEvent.isCtrlPressed()
,
KeyEvent.isShiftPressed()
,
KeyEvent.isAltPressed()
,
or
KeyEvent.hasModifiers()
.
Separating shortcut code from other keystroke handling (such as onKeyUp()
and onKeyDown()
) can make code maintenance easier and enables the default
acceptance of meta-keys without having to manually implement meta-key checks in
every case. Allowing all meta-key combinations can also be more convenient for
users who are accustomed to different keyboard layouts and operating systems.
Kotlin
override fun dispatchKeyShortcutEvent(event: KeyEvent): Boolean { return when (event.keyCode) { KeyEvent.KEYCODE_O -> { openFile() // Ctrl+O, Shift+O, Alt+O true } KeyEvent.KEYCODE_Z-> { if (event.isCtrlPressed) { if (event.isShiftPressed) { redoLastAction() // Ctrl+Shift+Z pressed true } else { undoLastAction() // Ctrl+Z pressed true } } } else -> { return super.dispatchKeyShortcutEvent(event) } } }
Java
@Override public boolean dispatchKeyShortcutEvent(KeyEvent event) { if (event.getKeyCode() == KeyEvent.KEYCODE_O) { openFile(); // Ctrl+O, Shift+O, Alt+O return true; } else if(event.getKeyCode() == KeyEvent.KEYCODE_Z) { if (event.isCtrlPressed()) { if (event.isShiftPressed()) { redoLastAction(); return true; } else { undoLastAction(); return true; } } } return super.dispatchKeyShortcutEvent(event); }
You can also implement shortcuts in onKeyUp()
by checking for
KeyEvent.isCtrlPressed()
,
KeyEvent.isShiftPressed()
,
or
KeyEvent.isAltPressed()
in the same manner as above. This can be easier to maintain if the
meta-behavior is more of a modification to an app behavior than a shortcut.
For example, when W means "walk forward" and Shift + W means "run forward".
Kotlin
override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean { return when(keyCode) { KeyEvent.KEYCODE_W-> { if (event.isShiftPressed) { if (event.isCtrlPressed) { flyForward() // Ctrl+Shift+W pressed true } else { runForward() // Shift+W pressed true } } else { walkForward() // W pressed true } } else -> super.onKeyUp(keyCode, event) } }
Java
@Override public boolean onKeyUp(int keyCode, KeyEvent event) { if (keyCode == KeyEvent.KEYCODE_W) { if (event.isShiftPressed()) { if (event.isCtrlPressed()) { flyForward(); // Ctrl+Shift+W pressed return true; } else { runForward(); // Shift+W pressed return true; } } else { walkForward(); return true; } } return super.onKeyUp(keyCode, event); }
Stylus
Many large screen devices come with a stylus, and Android apps handle that as touchscreen input. Some devices might also have a USB or Bluetooth drawing table, like the Wacom Intuos. Android apps can receive bluetooth input, but won't work with USB input.
A stylus event is reported as a touchscreen event via
View.onTouchEvent()
or
View.onGenericMotionEvent()
,
and contains a
MotionEvent.getSource()
of type
SOURCE_STYLUS
.
The MotionEvent
will also contain additional data:
MotionEvent.getToolType()
returnsTOOL_TYPE_FINGER
, TOOL_TYPE_STYLUS, orTOOL_TYPE_ERASER
depending on the tool that made contact with the surfaceMotionEvent.getPressure()
reports the physical pressure applied to the stylus pen (if supported)MotionEvent.getAxisValue()
withMotionEvent.AXIS_TILT
andMotionEvent.AXIS_ORIENTATION
which can be used to read the physical tilt and orientation of the stylus (if supported)
Historical points
Android batches input events and delivers them once per frame. A stylus pen
can report events at much higher frequencies than the display. When creating
drawing apps, it is important to check for events that may be in the recent
past by using the getHistorical
APIs:
MotionEvent.getHistoricalX()
MotionEvent.getHistoricalY()
MotionEvent.getHistoricalPressure()
MotionEvent.getHistoricalAxisValue()
Palm rejection
Most large screen devices attempt to recognize when a user's palm is resting
on the touchscreen. However, this is not always possible. Sometimes a touch
event might be reported to the app before the OS recognizes it as a palm. In
that case, touches are canceled by reporting an ACTION_CANCEL
event.
This event tells the app that certain touches are invalid and it should undo all interactions caused by those touches. For example, a drawing app might temporarily draw new lines as soon as they are received to provide the lowest latency, but only commit them permanently to the canvas once the touch series is finished cleanly. If touch events are canceled in the meantime, the temporary lines can be easily erased.
Note-taking apps
Chrome OS has a special intent that surfaces registered note-taking apps to users. To register an app as a note-taking app, add the following to the Android manifest:
<intent-filter>
<action android:name="org.chromium.arc.intent.action.CREATE_NOTE" />
<category android:name="android.intent.category.DEFAULT" />
</intent-filter>
When an app is registered, the user can select it as the default note-taking
app. When a new note is requested, the app should create an empty note ready
for stylus input. When the user wishes to annotate an image (such as a
screenshot or downloaded image), the app launches with ClipData
containing
one or more items with content://
URIs. The app should create a note that
uses the first attached image as a background image and enter a mode where
the user can draw on the screen with a stylus.
Test note-taking intents without a stylus
To test if an app responds correctly to note-taking intents without an active stylus, use the following method to display the note-taking options on Chrome OS:
- Switch to dev mode and make the device writable
- Press Ctrl + Alt + F2 to open a terminal
- Run the command
sudo vi /etc/chrome_dev.conf
- Press
i
to edit and add--ash-enable-palette
to a new line at the end of the file - Save by pressing Esc and then typing :, w, q and pressing Enter
- Press Ctrl + Alt + F1 to return to the regular Chrome OS UI
- Log out and back in
There should now be a stylus menu on the shelf:
- Tap the stylus button on the shelf and choose New note. This should open a blank drawing note.
- Take a screenshot. From the shelf, select stylus button > Capture screen or download an image. There should be the option to "Annotate image" in the notification. This should launch the app with the image ready to be annotated.
Mouse and touchpad support
Most apps generally only need to handle three large screen–centric events: right-click, hover, and drag and drop.
Right-click
Any actions that cause an app to show a context menu, like touch & hold on a
list item, should also react to right-click events. To handle right-click
events, apps should register a
View.OnContextClickListener
.
For details on constructing a context menu, see
Creating Contextual Menus.
Kotlin
yourView.setOnContextClickListener { showContextMenu() true }
Java
yourView.setOnContextClickListener(v -> { showContextMenu(); return true; });
Hover
Developers can make their app layouts feel polished and easier to use by handling hover events. This is especially true for custom views. The two most common examples of this are:
- Indicating to users if an element has interactive behavior, such as being clickable or editable, by changing the mouse pointer icon
- Adding visual feedback to items in a large list or grid when the pointer is hovering over them
Kotlin
// Change the icon to a "hand" pointer on hover, // Highlight the view by changing the background. yourView.setOnHoverListener { view, _ -> addVisualHighlighting(true) view.pointerIcon = PointerIcon.getSystemIcon(view.context, PointerIcon.TYPE_HAND) false // listener did not consume the event. }
Java
yourView.setOnHoverListener((view, event) -> { addVisualHighlighting(true); view.setPointerIcon(PointerIcon .getSystemIcon(view.getContext(), PointerIcon.TYPE_HAND)); return true; });
Drag and drop
In a multi-window environment, users expect to be able to drag and drop items between apps. This is true for desktop devices as well as tablets, phones, and foldables in split-screen mode.
Developers should consider whether users are likely to drag items into their app. Some common examples include: photo editors should expect to receive photos, audio players should expect to receive audio files, and drawing programs should expect to receive photos.
To add drag and drop support, follow the Android Drag and drop documentation and take a look at this Chrome OS blog post.
Special considerations for Chrome OS
- Remember to request permission via
requestDragAndDropPermissions
to access items dragged in from outside the app - An item must have the
View.DRAG_FLAG_GLOBAL
flag in order to be dragged out to other applications
Advanced pointer support
Apps that do advanced handling of mouse and touchpad input should follow the
Android documentation for
View.onGenericMotionEvent()
and use
MotionEvent.getSource()
to distinguish between
SOURCE_MOUSE
and
SOURCE_TOUCHSCREEN
.
Examine the MotionEvent
to implement the required behavior:
- Movement generates
ACTION_HOVER_MOVE
events. - Buttons generate
ACTION_BUTTON_PRESS
andACTION_BUTTON_RELEASE
events. You can also check the current state of all mouse/trackpad buttons usinggetButtonState()
. - Mouse wheel scrolling generates
ACTION_SCROLL
events.
Game controllers
Some large screen Android devices support up to four game controllers. Developers should use the standard Android game controller APIs to handle them (see Support game controllers).
Buttons are mapped to common values following a common mapping. Unfortunately, not all game controller manufacturers follow the same mapping conventions. You can provide a much better experience if you allow users to select different popular controller mappings. See Process gamepad button presses for more information.
Input translation mode
Chrome OS enables an input translation mode by default. For most Android apps this mode helps apps work as expected in a desktop environment. Some examples include automatically enabling two-finger scrolling on the touchpad, mouse wheel scrolling, and mapping raw display coordinates to window coordinates. Generally, app developers do not need to implement any of these behaviors themselves.
If an app implements custom input behavior, for example defining a custom two-finger touchpad pinch action, or these input translations do not provide the input events expected by the app, you can disable the input translation mode by adding the following tag to the Android manifest:
<uses-feature
android:name="android.hardware.type.pc"
android:required="false" />