Memory & LMK Expert Hints

  • To investigate low memory kills, look for the "lmk_kill_occurred" ftrace event; a high count indicates severe memory pressure.
  • When you see a burst of "lmk_kill_occurred" events, query for processes with high CPU wall_duration in the "sched_slice" table during the same time window. Runaway processes consuming CPU exacerbate memory pressure.
  • To understand a process's memory impact, inspect its "anon_rss" (Anonymous Resident Set Size) from memory counters like "mem.info".
  • Check system-wide memory stats for a high or rapidly increasing "swap_used" value.
  • To confirm memory thrashing, look for high CPU usage by the "kswapd" kernel thread.
  • To find the direct trigger for LMKD, look for "memory_pressure" trace events from Pressure Stall Information (PSI).
  • To detect 'lost' memory for processes using hardware accelerators (like a TPU), query the counter table for RssFile values. If it drops significantly while hardware is active, check for kswapd0 scheduling slices.
  • When analyzing memory shared with hardware, query the dma_heap_stat and dmabuf_total_size counters (or ion_total_size for older devices) for a more accurate picture than RSS.
  • Be aware that RssFile can over-report memory if the same physical page is mapped multiple times.
  • When investigating OOMs, establish a baseline by querying process memory counters (e.g., mem.rss) and compare the median and 95th percentile against its history.
  • If bitmaps are a major memory consumer, check for outliers in bitmap count by querying android.graphics.Bitmap instances across traces.
  • To find what is holding onto an object (like a bitmap), trace its retainer path back to a GC root by querying heap_graph_reference. Pay close attention to custom application classes.
  • Query the android_bitmaps table to check the width and height properties of bitmaps.
  • Check for the presence of software bitmaps, which consume memory on both the app heap and in graphics memory.
  • When analyzing bitmaps, look for duplicates by checking for multiple android.graphics.Bitmap objects with identical properties.