Semantics in Compose

A composition describes the UI of your app and is produced by running composables. The composition is a tree-structure that consists of the composables that describe your UI.

Next to the Composition, there exists a parallel tree, called the semantics tree. This tree describes your UI in an alternative manner that is understandable for Accessibility services and for the Testing framework. Accessibility services use the tree to describe the app to users with a specific need. The testing framework uses the tree to interact with your app and make assertions about it. The Semantics tree does not contain the information on how to draw your composables, but it contains information about the semantic meaning of your composables.

If your app consists of composables and modifiers from the Compose foundation and material library, the Semantics tree is automatically filled and generated for you. However when you're adding custom low-level composables, you have to manually provide its semantics. There might also be situations where your tree does not correctly or fully represent the meaning of the elements on the screen, in which case you can adapt the tree.

Consider for example this custom calendar composable:

In this example, the entire calendar is implemented as a single low-level composable, using the Layout composable and drawing directly to the Canvas. If you don't do anything else, accessibility services won't receive enough information about the content of the composable and the user's selection within the calendar. For example, if a user clicks on the day containing 17, the accessibility framework only receives the description information for the whole calendar control. In this case, the TalkBack accessibility service would announce "Calendar" or, only slightly better, "April Calendar" and the user would be left to wonder what day was selected. To make this composable more accessible, you'll need to add semantic information manually.

Semantics properties

All nodes in the UI tree with some semantic meaning have a parallel node in the Semantics tree. The node in the Semantics tree contains those properties that convey the meaning of the corresponding composable. For example, the Text composable contains a semantic property text, because that's the meaning of that composable. An Icon contains a contentDescription property (if set by the developer) that conveys in text what the meaning of the Icon is. Composables and modifiers that are built on top of the Compose foundation library already set the relevant properties for you. Optionally, set or override the properties yourself with the semantics and clearAndSetSemantics modifiers. For example, add custom accessibility actions to a node, provide an alternative state description for a toggleable element, or indicate that a certain text composable should be considered as a heading.

To visualize the Semantics tree, use the Layout Inspector Tool or use the printToLog() method inside tests. This prints the current Semantics tree inside Logcat.

class MyComposeTest {

    @get:Rule
    val composeTestRule = createComposeRule()

    @Test
    fun MyTest() {
        // Start the app
        composeTestRule.setContent {
            MyTheme {
                Text("Hello world!")
            }
        }
        // Log the full semantics tree
        composeTestRule.onRoot().printToLog("MY TAG")
    }
}
SemanticsSnippets.kt

The output of this test would be:

    Printing with useUnmergedTree = 'false'
    Node #1 at (l=0.0, t=63.0, r=221.0, b=120.0)px
     |-Node #2 at (l=0.0, t=63.0, r=221.0, b=120.0)px
       Text = '[Hello world!]'
       Actions = [GetTextLayoutResult]

Consider how semantics properties convey the meaning of a composable. Consider a Switch. This is how it looks to the user:

To describe the meaning of this element, you could say the following: "This is a Switch, which is a toggleable element in its 'On' state. You can click it to interact with it."

This is exactly what the semantics properties are used for. The semantics node of this Switch element contains the following properties, as visualized with the Layout Inspector:

The Role indicates the type of element. The StateDescription describes how the "On" state should be referenced. By default this is a localized version of the word "On", but this can be made more specific (for example, "Enabled") based on the context. The ToggleableState is the current state of the Switch. The OnClick property references the method used to interact with this element. For a full list of semantics properties, check out the SemanticsProperties object. For a full list of possible Accessibility Actions, check out the SemanticsActions object.

Keeping track of the semantics properties of each composable in your app unlocks a lot of powerful possibilities. Some examples:

• Talkback uses the properties to read aloud what's shown on the screen and lets the user smoothly interact with it. For the Switch composable, Talkback might say: "On; Switch; double tap to toggle". The user can double tap their screen to toggle the Switch off.
• The Testing framework uses the properties to find nodes, interact with them, and make assertions. A sample test for the Switch could be:

  val mySwitch = SemanticsMatcher.expectValue(
      SemanticsProperties.Role, Role.Switch
  )
  composeTestRule.onNode(mySwitch)
      .performClick()
      .assertIsOff()
  SemanticsSnippets.kt

Merged and unmerged Semantics tree

As mentioned before, each composable in the UI tree might have zero or more semantics properties set. When a composable has no semantics properties set, it isn't included as part of the Semantics tree. That way, the Semantics tree contains only the nodes that actually contain semantic meaning. However, sometimes to convey the correct meaning of what is shown on the screen, it is also useful to merge certain sub-trees of nodes and treat them as one. That way you can reason about a set of nodes as a whole, instead of dealing with each descendant node individually. As a rule of thumb, each node in this tree represents a focusable element when using Accessibility services.

An example of such a composable is Button. You can reason about a button as a single element, even though it may contain multiple child nodes:

Button(onClick = { /*TODO*/ }) {
    Icon(
        imageVector = Icons.Filled.Favorite,
        contentDescription = null
    )
    Spacer(Modifier.size(ButtonDefaults.IconSpacing))
    Text("Like")
}
SemanticsSnippets.kt

In the Semantics tree, the properties of the button's descendants are merged, and the button is presented as a single leaf node in the tree:

Composables and modifiers can indicate that they want to merge their descendants' semantics properties by calling Modifier.semantics (mergeDescendants = true) {}. Setting this property to true indicates that the semantics properties should be merged. In the Button example, the Button composable uses the clickable modifier internally that includes this semantics modifier. Therefore, the descendant nodes of the button are merged. Read the accessibility documentation to learn more about when you should change merging behavior in your composable.

Several modifiers and composables in the Foundation and Material Compose libraries have this property set. For example, the clickable and toggleable modifiers will automatically merge their descendants. Also, the ListItem composable will merge its descendants.

Inspect the trees

The semantics tree is in fact two different trees. There's a merged Semantics tree, which merges descendant nodes when mergeDescendants is set to true. There's also an unmerged Semantics tree, which does not apply the merging, but keeps every node intact. Accessibility services use the unmerged tree and apply their own merging algorithms, taking into consideration the mergeDescendants property. The testing framework uses the merged tree by default.

You can inspect both trees with the printToLog() method. By default, and as in the earlier examples, the merged tree is logged. To print the unmerged tree instead, set the useUnmergedTree parameter of the onRoot() matcher to true:

composeTestRule.onRoot(useUnmergedTree = true).printToLog("MY TAG")
SemanticsSnippets.kt

The Layout Inspector lets you display both the merged and the unmerged Semantics tree, by selecting the preferred one in the view filter:

For each node in your tree, the Layout Inspector shows both the Merged Semantics and the Semantics set on that node in the properties panel:

By default, matchers in the Testing Framework use the merged Semantics tree. That's why you can interact with a Button by matching the text shown inside it:

composeTestRule.onNodeWithText("Like").performClick()
SemanticsSnippets.kt

Override this behavior by setting the useUnmergedTree parameter of the matchers to true, as with the onRoot matcher.

Merge behavior

When a composable indicates that its descendants should be merged, how does this merging happen exactly?

Each semantics property has a defined merging strategy. For example, the ContentDescription property adds all descendant ContentDescription values to a list. Check the merging strategy of a semantics property by checking its mergePolicy implementation in SemanticsProperties.kt. Properties can take on the parent or child value, merge the values into a list or string, not allow merging at all and throw an exception instead, or any other custom merging strategy.

An important note is that descendants that themselves have set mergeDescendants = true are not included in the merge. Take a look at an example:

Here is a clickable list item. When the user presses the row, the app navigates to the article detail page, where the user can read the article. Inside the list item, there is a button to bookmark the article, which forms a nested clickable element, so the button shows up separately in the merged tree. The rest of the content in the row is merged:

Adapt the Semantics tree

As mentioned before, you can override or clear certain semantics properties or change the merging behavior of the tree. This is particularly relevant when you're creating your own custom components. Without setting the correct properties and merge behavior, your app might not be accessible, and tests might behave differently than you expect. To read more about some common use cases where you should adapt the Semantics tree, read the accessibility documentation. If you want to learn more about testing, check the testing guide.

Additional resources

• Accessibility: Essential concepts and techniques common to all Android app development
• Build Accessible Apps: Key steps you can take to make your app more accessible
• Principles for improving app accessibility: Key principles to keep in mind when working to make your app more accessible
• Testing for Accessibility: Test principles and tools for Android accessibility