Solutions for system stuttering or out-of-memory issues caused by Linux memcg residuals

Updated at:

This topic describes the causes of and solutions for system stuttering or out-of-memory (OOM) issues caused by Linux memcg residuals.

Problem description

After the system runs for a long time, it may stutter or run out of memory.

To confirm whether the issue is caused by memcg residuals, check for both of the following indicators:

  • A large amount of kernel memory is consumed by unreclaimable unreclaimable slab and percpu allocations.

  • The number of memcg instances is large, but the number of folders in /sys/fs/cgroup/memory/ is relatively small.

Cause

A design flaw in the kmem feature of the Linux memcg subsystem causes this issue. When memcg kmem is enabled, deleting a memcg leaves some internal kernel data structures behind. Because these structures are unreclaimable, they accumulate over time and continuously consume physical memory until the system stutters or runs out of memory.

This issue affects the following systems:

  • Alibaba Cloud Linux 3 with a kernel version earlier than 5.10.134-12.2

  • Alibaba Cloud Linux 2 with a kernel version earlier than 4.19.91-26.6

  • Other mainstream Linux distributions

Solution

Disable the memcg kmem feature

  1. Disable the memcg kmem feature.

    sudo grubby --update-kernel=ALL --args='cgroup.memory=nokmem'
  2. Restart the system.

    sudo reboot
  3. Verify that the feature is disabled.

    cat /proc/cmdline

    The output should contain cgroup.memory=nokmem, as shown in the following screenshot.

    image

  4. If the issue persists, enable the periodic background reclaim feature for memcg.

    Important

    This feature is available only on Alibaba Cloud Linux systems. It consumes some CPU resources, so evaluate your workload before enabling it. If the feature causes other issues, disable it.

    • Enable background reclaim:

      sudo sh -c 'echo 1 > /sys/kernel/mm/memcg_reaper/reap_background'
    • Disable background reclaim:

      sudo sh -c 'echo 0 > /sys/kernel/mm/memcg_reaper/reap_background'