How to prevent application performance fluctuation caused by cgroups-Alibaba Cloud Linux(Alinux)-阿里云帮助中心

Control groups (cgroups) are a Linux kernel feature that can be used to allocate, manage, and monitor resources on a system. If the hierarchies or number of cgroups is not appropriate for the system, application performance may degrade. This topic describes the issue of application performance jitter caused by cgroups and how to prevent it from occurring.

Problem description

In container-based scenarios, applications may occasionally experience increased request latency. Concurrently, the `sys` CPU utilization on the container's host reaches 30% or more. The `sys` metric is the percentage of CPU time spent in the kernel space. For example, when you run the top command to view Linux performance data, the `sy` value is 30% or higher.

top - 16:21:19 up 22 days,  1:59,  3 users,  load average: 40.02, 30.02, 25.06
Tasks: 193 total,   3 running, 190 sleeping,   0 stopped,   0 zombie
%Cpu(s):  20.0 us,  30.0 sy,  0.0 ni, 45.0 id,  5.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 24130808 total,  2071424 free,  1832512 used, 20226872 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 19737776 avail Mem

Possible causes

For example, if you run the cat /proc/cgroups command to view the status of all cgroups, the output may show that the number of cgroups for memory is as high as 2,040.

#subsys_name  hierarchy num_cgroups enabled
cpuset  2     200     1
cpu     2     200     1
cpuacct 2     200     1
blkio   6     62      1
memory  10    2040    1
devices 5     59      1

In this case, you can use perf to analyze and identify the cause of the issue.

Run the following command to install perf:
```
yum install perf -y
```
Run the following command to sample and analyze the processes of the system:
```
perf record -a -g sleep 10
```

Run the following command to view the analysis results:

perf report

The following output is an example:

31.04%  [kernel]                                      [k] memcg_stat_show
18.61%  [kernel]                                      [k] memcg_sum_events.isra.22
9.41%  [kernel]                                      [k] mem_cgroup_iter
6.94%  [kernel]                                      [k] css_next_descendant_pre
6.11%  [kernel]                                      [k] _find_next_bit
3.96%  [kernel]                                      [k] mem_cgroup_usage.part.43
1.75%  [kernel]                                      [k] find_next_bit
1.38%  [kernel]                                      [k] mem_cgroup_node_nr_lru_pages

The analysis results indicate that most of the Linux kernel runtime is spent in the memcg_stat_show function. This occurs because an excessive number of cgroups are associated with memory, which causes the kernel to spend a long time traversing them.

In addition, excessive numbers of cgroups corresponding to cpuacct and cpu may also affect the performance of Completely Fair Scheduler (CFS) and the load balancing mechanism of the Linux kernel.

Solutions

When you perform operations and maintenance (O&M) on Linux instances, follow these suggestions to prevent application performance jitter caused by cgroups.

Keep the cgroup hierarchy to 10 levels or less.
Limit the total number of cgroups to 1,000 or less. Keep the number of cgroups as low as possible.

You can also use the operating system console's Zombie Memcg Reclaim feature to periodically reclaim zombie Memcgs on nodes. This practice prevents problems caused by Memcg leaks.