Solutions for system stuttering or out-of-memory issues caused by Linux memcg residuals
This topic describes the causes of and solutions for system stuttering or out-of-memory (OOM) issues caused by Linux memcg residuals.
Problem description
After the system runs for a long time, it may stutter or run out of memory.
To confirm whether the issue is caused by memcg residuals, check for both of the following indicators:
A large amount of kernel memory is consumed by unreclaimable
unreclaimable slabandpercpuallocations.The number of
memcginstances is large, but the number of folders in/sys/fs/cgroup/memory/is relatively small.
Cause
A design flaw in the kmem feature of the Linux memcg subsystem causes this issue. When memcg kmem is enabled, deleting a memcg leaves some internal kernel data structures behind. Because these structures are unreclaimable, they accumulate over time and continuously consume physical memory until the system stutters or runs out of memory.
This issue affects the following systems:
Alibaba Cloud Linux 3 with a kernel version earlier than 5.10.134-12.2
Alibaba Cloud Linux 2 with a kernel version earlier than 4.19.91-26.6
Other mainstream Linux distributions
Solution
Disable the memcg kmem feature
Disable the
memcg kmemfeature.sudo grubby --update-kernel=ALL --args='cgroup.memory=nokmem'Restart the system.
sudo rebootVerify that the feature is disabled.
cat /proc/cmdlineThe output should contain
cgroup.memory=nokmem, as shown in the following screenshot.
If the issue persists, enable the periodic background reclaim feature for
memcg.ImportantThis feature is available only on Alibaba Cloud Linux systems. It consumes some CPU resources, so evaluate your workload before enabling it. If the feature causes other issues, disable it.
Enable background reclaim:
sudo sh -c 'echo 1 > /sys/kernel/mm/memcg_reaper/reap_background'Disable background reclaim:
sudo sh -c 'echo 0 > /sys/kernel/mm/memcg_reaper/reap_background'