Use the latency statistics feature of memsli and sched_sli
When containerized workloads experience memory pressure or CPU scheduling contention, identifying which cgroup is responsible and how severe the latency is can be difficult without per-cgroup observability. memsli and sched_sli are Alibaba Cloud Linux 3 kernel features that expose per-cgroup latency histograms: memsli for memory operations (via the memory subsystem) and sched_sli for CPU scheduling (via the cpuacct subsystem). This topic explains how to enable, disable, and query these features on an Alibaba Cloud Linux 3 instance.
Prerequisites
Before you begin, ensure that you have:
An instance running Alibaba Cloud Linux 3
A kernel version of 5.10.134-12 or later
How it works
memsli and sched_sli hook into Control Groups (cgroups) to measure how long memory and scheduling operations take in each cgroup:
memsli tracks latency for memory operations such as direct memory reclamation, direct memory compaction, and memory swap-out or swap-in. All latency data is exposed as read-only files under
/sys/fs/cgroup/memory/.sched_sli tracks latency for CPU scheduling operations such as runqueue wait time and I/O block time. All latency data is exposed as read-only files under
/sys/fs/cgroup/cpuacct/.
Both features report latency as a histogram: each output line shows how many events fell within a given latency range, plus a total(ms) field (cumulative latency across all events). sched_sli interfaces also include an nr field (total event count). A non-zero count in a high-latency bucket, or a rising total(ms), indicates contention worth investigating.
Enable or disable memsli
memsli is enabled by default.
To disable it:
echo 0 | sudo tee /proc/memsli/enabledTo re-enable it:
echo 1 | sudo tee /proc/memsli/enabled
Enable or disable sched_sli
sched_sli is disabled by default.
To enable it:
echo 1 | sudo tee /proc/cpusli/sched_lat_enabledTo disable it:
echo 0 | sudo tee /proc/cpusli/sched_lat_enabled
sched_sli requires the cpu and cpuacct subsystems to be mounted to the same directory. In Alibaba Cloud Linux 3, both are mounted to /sys/fs/cgroup/ by default.
Query memsli latency statistics
All memsli interfaces are read-only files under /sys/fs/cgroup/memory/.
| Interface | Description |
|---|---|
memory.direct_compact_latency | Latency in direct memory compaction when a memory cgroup (memcg) allocates memory |
memory.direct_reclaim_memcg_latency | Latency in direct memory reclamation when memory usage in a memcg exceeds its limit |
memory.direct_reclaim_global_latency | Latency in global direct memory reclamation due to insufficient global memory during memcg memory allocation |
memory.direct_swapout_memcg_latency | Latency in direct memory swap-out for anonymous pages when memcg memory usage exceeds its limit and memcg memory is reclaimed |
memory.direct_swapout_global_latency | Latency in global direct memory swap-out for anonymous pages due to insufficient global memory during memcg memory allocation |
memory.direct_swapin_latency | Latency in direct memory swap-in from swap space when memcgs reclaim memory |
Example: query direct memory reclamation latency
sudo cat /sys/fs/cgroup/memory/memory.direct_reclaim_memcg_latencySample output:
0-1ms: 0
1-5ms: 0
5-10ms: 0
10-100ms: 0
100-500ms: 0
500-1000ms: 0
>=1000ms: 0
total(ms): 0Each line shows the number of reclamation events that completed within that latency range. total(ms) is the cumulative latency across all events — use it to detect sustained pressure even when individual event counts are low.
Query sched_sli latency statistics
All sched_sli interfaces are read-only files under /sys/fs/cgroup/cpuacct/.
| Interface | Description |
|---|---|
cpuacct.cgroup_wait_latency | Latency for processes waiting to be scheduled in the runqueue |
cpuacct.wait_latency | Latency of processes waiting for scheduling in the runqueue |
cpuacct.block_latency | Latency from when uninterruptible processes dequeue to when they are awakened |
cpuacct.ioblock_latency | Latency from when uninterruptible processes waiting for I/O dequeue to when they are awakened |
Example: query I/O block latency
sudo cat /sys/fs/cgroup/cpuacct/cpuacct.ioblock_latencySample output:
0-1ms: 0
1-4ms: 0
4-7ms: 0
7-10ms: 0
10-20ms: 0
20-30ms: 0
30-40ms: 0
40-50ms: 0
50-100ms: 0
100-500ms: 0
500-1000ms: 0
1000-5000ms: 0
5000-10000ms: 0
>=10000ms: 0
total(ms): 0
nr: 0Each line shows the number of I/O block events that completed within that latency range. total(ms) is the cumulative I/O block latency. nr is the total number of events recorded. A large total(ms) with a small nr indicates a few extremely long I/O stalls; a large nr with events spread across high-latency buckets indicates systemic I/O contention.