Analyze traces in bulk

Once you have collected multiple traces using ProfilingManager, exploring them individually to find performance problems becomes impractical. Bulk trace analysis lets you query a dataset of traces simultaneously to:

  • Identify common performance regressions.
  • Calculate statistical distributions (for example, P50, P90, P99 latency).
  • Find patterns across several traces.
  • Find outlier traces to understand and debug performance issues.

This section demonstrates how to use the Perfetto Python Batch Trace Processor to analyze startup metrics across a set of locally stored traces and locate outlier traces for deeper analysis.

Design the query

First, create a PerfettoSQL query that extracts the metric you want to compare. The following example measures app-controlled startup latency from activityStart to the first rendered frame (Choreographer#doFrame), as shown in Figure 1.

A timeline view of a trace, highlighting the duration from the activityStart event to the first Choreographer#doFrame event.
Figure 1. Trace section from activityStart to the first frame generated.
CREATE OR REPLACE PERFETTO FUNCTION find_slices(pattern STRING) RETURNS
TABLE (name STRING, ts LONG, dur LONG) AS
SELECT name,ts,dur FROM slice WHERE name GLOB $pattern;

CREATE OR REPLACE PERFETTO FUNCTION generate_start_to_end_slices(startSlicePattern STRING, endSlicePattern STRING, inclusive BOOL) RETURNS
TABLE(name STRING, ts LONG, dur LONG) AS
SELECT name, ts, MIN(startToEndDur) as dur
FROM
  (SELECT S.name as name, S.ts as ts, E.ts + IIF($inclusive, E.dur, 0) - S.ts as startToEndDur
  FROM find_slices($startSlicePattern) as S CROSS JOIN find_slices($endSlicePattern) as E
  WHERE startToEndDur > 0)
GROUP BY name, ts;

SELECT ts,name,dur from generate_start_to_end_slices('activityStart','*Choreographer#doFrame [0-9]*', true)

You can execute the query within the Perfetto UI and then use the query results to generate a debug track (Figure 2) and visualize it within the timeline (Figure 3).

Perfetto UI query results panel showing how to create a debug track for a startup query.
Figure 2. Create a debug track for a startup query.
A timeline view in the Perfetto UI showing a generated debug track for a startup query.
Figure 3. Generated debug track for a startup query.

Set up the Python environment

Install Python on your local machine and its required libraries:

pip install perfetto pandas plotly

Create the bulk trace analysis script

The following sample script executes the query in multiple traces using Perfetto's Python BatchTraceProcessor.

from perfetto.batch_trace_processor import BatchTraceProcessor
import glob
import plotly.express as px

# Collect all trace files in the local directory
traces = glob.glob('*.perfetto-trace')

if not traces:
    print("No .perfetto-trace files found in the current directory.")
    exit(1)

if __name__ == '__main__':
    # Process all traces in parallel to aggregate metrics across runs
    with BatchTraceProcessor(traces) as btp:
        query = """
        CREATE OR REPLACE PERFETTO FUNCTION find_slices(pattern STRING) RETURNS
        TABLE (name STRING, ts LONG, dur LONG) AS
        SELECT name,ts,dur FROM slice WHERE name GLOB $pattern;

        CREATE OR REPLACE PERFETTO FUNCTION generate_start_to_end_slices(startSlicePattern STRING, endSlicePattern STRING, inclusive BOOL) RETURNS
        TABLE(name STRING, ts LONG, dur LONG) AS
        SELECT name, ts, MIN(startToEndDur) as dur
        FROM
          (SELECT S.name as name, S.ts as ts, E.ts + IIF($inclusive, E.dur, 0) - S.ts as startToEndDur
          FROM find_slices($startSlicePattern) as S CROSS JOIN find_slices($endSlicePattern) as E
          WHERE startToEndDur > 0)
        GROUP BY name, ts;

        SELECT ts,name,dur / 1000000 as dur_ms from generate_start_to_end_slices('activityStart','*Choreographer#doFrame [0-9]*', true)
        """
        df = btp.query_and_flatten(query)
        # Plot the distribution of startup times, tracking trace file paths on
        # hover
        violin = px.violin(df, x='dur_ms', hover_data='_path', title='startup time', points='all')
        violin.show()

Understand the script

When you run the Python script, it performs the following actions:

  1. The script searches in your local directory for all Perfetto traces suffixed with .perfetto-trace and uses them as source traces for analysis.
  2. It runs a bulk trace query that computes the subset of startup time corresponding to the time from the activityStart trace slice to the first frame generated by your app.
  3. It plots the latency in milliseconds using a violin plot to visualize the distribution of startup times.

Interpret the results

A violin plot showing the distribution of queried startup latencies.
Figure 4. Violin plot of queried startup latencies.

Running the script generates a violin plot showing a bimodal distribution with two distinct latency peaks (Figure 4). Hover over individual data points in the plot to inspect the _path filename and compare traces from each peak (Figure 5):

  • High-latency traces (right peak): Opening a trace from this group in the Perfetto UI reveals an extra MyFlaggedFeature slice running during startup (Figure 6).
  • Low-latency traces (left peak): Opening a trace from the faster group confirms that MyFlaggedFeature is absent (Figure 7), indicating that a feature flag enabled for a subset of users caused the startup regression.
A plot highlighted high latency trace.
Figure 5. High latency data point in a violin plot.
A trace highlighting high latency startup due to MyFlaggedFeature slice.
Figure 6. High latency trace startup with an additional MyFlaggedFeature slice.
A trace highlighting low latency startup without MyFlaggedFeature slice.
Figure 7. Low latency trace startup.

This example demonstrates one of the many ways you can use bulk trace analysis. Other use cases include extracting statistics from the field to gauge impact, detecting regressions, and more.