Semantics in Compose

A Composition describes the UI of your app and is produced by running composables. The Composition is a tree-structure that consists of the composables that describe your UI.

Next to the Composition, there exists a parallel tree, called the Semantics tree. This tree describes your UI in an alternative manner that is understandable for Accessibility services and for the Testing framework. Accessibility services use the tree to describe the app to users with a specific need. The Testing framework uses it to interact with your app and make assertions about it. The Semantics tree does not contain the information on how to draw your composables, but it contains information about the semantic meaning of your composables.

Figure 1. A typical UI hierarchy and its semantics tree.

If your app consists of composables and modifiers from the Compose foundation and material library, the Semantics tree is automatically filled and generated for you. However when you’re adding custom low-level composables, you will have to manually provide its semantics. There might also be situations where your tree does not correctly or fully represent the meaning of the elements on the screen, in which case you can adapt the tree.

Consider for example this custom calendar composable:

Figure 2. A custom calendar composable with selectable day elements.

In this example, the entire calendar is implemented as a single low-level composable, using the Layout composable and drawing directly to the Canvas. If you do not do anything else, accessibility services will not receive enough information about the content of the composable and the user's selection within the calendar. For example, if a user clicks on the day containing 17, the accessibility framework only receives the description information for the whole calendar control. In this case, the TalkBack accessibility service would simply announce "Calendar" or, only slightly better, "April Calendar" and the user would be left to wonder what day was selected. To make this composable more accessible, you’ll need to add semantic information manually.

Semantics properties

All nodes in the UI tree with some semantic meaning have a parallel node in the Semantics tree. The node in the Semantics tree contains those properties that convey the meaning of the corresponding composable. For example, the Text composable contains a semantic property text, because that’s the meaning of that composable. An Icon contains a contentDescription property (if set by the developer) that conveys in text what the meaning of the Icon is. Composables and modifiers that are built on top of the Compose foundation library already set the relevant properties for you. Optionally, you can set or override the properties yourself with the semantics and clearAndSetSemantics modifiers. For example, you can add custom accessibility actions to a node, provide an alternative state description for a toggleable element, or indicate that a certain text composable should be considered as a heading.

To visualize the Semantics tree, we can use the Layout Inspector Tool or use the printToLog() method inside our tests. This will print the current Semantics tree inside Logcat.

class MyComposeTest {

    @get:Rule
    val composeTestRule = createComposeRule()

    @Test
    fun MyTest() {
        // Start the app
        composeTestRule.setContent {
            MyTheme {
                Text("Hello world!")
            }
        }
        // Log the full semantics tree
        composeTestRule.onRoot().printToLog("MY TAG")
    }
}SemanticsSnippets.kt

The output of this test would be:

    Printing with useUnmergedTree = 'false'
    Node #1 at (l=0.0, t=63.0, r=221.0, b=120.0)px
     |-Node #2 at (l=0.0, t=63.0, r=221.0, b=120.0)px
       Text = '[Hello world!]'
       Actions = [GetTextLayoutResult]

Let’s look at an example to see how semantics properties are used to convey the meaning of a composable. Let’s think about a Switch. This is how it looks to the user:

Figure 3. A Switch in its “On” and “Off” state.

To describe the meaning of this element, you could say the following: “This is a Switch, which is a toggleable element, currently in its 'On' state. You can click it to interact with it.”

This is exactly what the semantics properties are used for. The semantics node of this Switch element contains the following properties, as visualized with the Layout Inspector:

Figure 4. Layout Inspector showing the Semantics properties of a Switch composable.

The Role indicates what type of element we’re looking at. The StateDescription describes how the “On” state should be referenced. By default this is simply a localized version of the word “On”, but this can be made more specific (for example, “Enabled”) based on the context. The ToggleableState is the current state of the Switch. The OnClick property references the method used to interact with this element. For a full list of semantics properties, check out the SemanticsProperties object. For a full list of possible Accessibility Actions, check out the SemanticsActions object.

Keeping track of the semantics properties of each composable in your app unlocks a lot of powerful possibilities. Some examples:

Talkback uses the properties to read aloud what’s shown on the screen and lets the user smoothly interact with it. For our Switch, it might say: “On; Switch; double tap to toggle”. The user can double tap their screen to toggle the Switch off.

The Testing framework uses the properties to find nodes, interact with them, and make assertions. A sample test for our Switch could be:

val mySwitch = SemanticsMatcher.expectValue(
    SemanticsProperties.Role, Role.Switch
)
composeTestRule.onNode(mySwitch)
    .performClick()
    .assertIsOff()SemanticsSnippets.kt

Merged and unmerged Semantics tree

As mentioned before, each composable in the UI tree might have zero or more semantics properties set. When a composable has no semantics properties set, it isn’t included as part of the Semantics tree. That way, the Semantics tree contains only the nodes that actually contain semantic meaning. However, sometimes to convey the correct meaning of what is shown on the screen, it is also useful to merge certain sub-trees of nodes and treat them as one. That way we can reason about a set of nodes as a whole, instead of dealing with each descendant node individually. As a rule of thumb, each node in this tree represents a focusable element when using Accessibility services.

An example of such a composable is Button. We’d like to reason about the Button as a single element, even though it may contain multiple child nodes:

Button(onClick = { /*TODO*/ }) {
    Icon(
        imageVector = Icons.Filled.Favorite,
        contentDescription = null
    )
    Spacer(Modifier.size(ButtonDefaults.IconSpacing))
    Text("Like")
}SemanticsSnippets.kt

In our Semantics tree, the properties of the Button’s descendants are merged, and the Button is presented as a single leaf node in the tree:

Composables and modifiers can indicate that they want to merge their descendants' semantics properties by calling Modifier.semantics (mergeDescendants = true) {}. Setting this property to true indicates that the semantics properties should be merged. In our Button example, the Button composable uses the clickable modifier internally that includes this semantics modifier. Therefore, the descendant nodes of the Button will be merged. Read the accessibility documentation to learn more about when you should change merging behavior in your composable.

Several modifiers and composables in the Foundation and Material Compose libraries have this property set. For example, the clickable and toggleable modifiers will automatically merge their descendants. Also, the ListItem composable will merge its descendants.

Inspecting the trees

When talking about the Semantics tree, we are actually talking about two different trees. There’s a merged Semantics tree, which merges descendant nodes when mergeDescendants is set to true. There’s also an unmerged Semantics tree, which does not apply the merging, but keeps every node intact. Accessibility services use the unmerged tree and apply their own merging algorithms, taking into consideration the mergeDescendants property. The Testing framework uses the merged tree by default.

You can inspect both trees with the printToLog() method. By default, and as in the earlier examples, the merged tree will be logged. To print the unmerged tree instead, set the useUnmergedTree parameter of the onRoot() matcher to true:

composeTestRule.onRoot(useUnmergedTree = true).printToLog("MY TAG")SemanticsSnippets.kt

The Layout Inspector allows you to display both the merged and the unmerged Semantics tree, by selecting the preferred one in the view filter:

Figure 5. Layout Inspector view options, allowing both the display of the merged and the unmerged Semantics tree.

For each node in your tree, the Layout Inspector shows both the Merged Semantics and the Semantics set on that node in the properties panel:

By default, matchers in the Testing Framework use the merged Semantics tree. That’s why you can interact with a Button by matching the text shown inside it:

composeTestRule.onNodeWithText("Like").performClick()SemanticsSnippets.kt

You can override this behavior by setting the useUnmergedTree parameter of the matchers to true, as we did before with the onRoot matcher.

Merging behavior

When a composable indicates that its descendants should be merged, how does this merging happen exactly?

Each semantics property has a defined merging strategy. For example, the ContentDescription property adds all descendant ContentDescription values to a list. You can check the merging strategy of a semantics property by checking its mergePolicy implementation in SemanticsProperties.kt. Properties can choose to always pick the parent or child value, merge the values into a list or string, not allow merging at all and throw an exception instead, or any other custom merging strategy.

An important note is that descendants that themselves have set mergeDescendants = true are not included in the merge. Let’s take a look at an example:

Figure 6. List item with image, some text, and a bookmark icon.

Here we have a clickable list item. When the user presses the row, the app navigates to the article detail page, where the user can read the article. Inside the list item, there is a button to bookmark this article. In this case we have a nested clickable element, so the button will show up separately in the merged tree. The rest of the content in the row is merged:

Figure 7. The merged tree contains multiple texts in a list inside the Row node. The unmerged tree contains separate nodes for each Text composable.

Adapting the Semantics tree

As mentioned before, you can override or clear certain semantics properties, or change the merging behavior of the tree. This is particularly relevant when you’re creating your own custom components. Without setting the correct properties and merge behavior, your app might not be accessible, and tests might behave differently than you expect. To read more about some common use cases where you should adapt the Semantics tree, read the accessibility documentation. If you want to learn more about testing, check the Testing Guide.