Use the SysOM FastOOM feature

Enable FastOOM

1. Go to the Operating System Console - Component Management page.

2. Install SysOM for the target ECS instance.

3. Return to the Component Management page and click Details.

4. Click Create Configuration to configure the FastOOM feature.

   Enter a Configuration Name and select Enable Custom FastOOM.

   Node-level FastOOM

   Enter the configuration information and click OK. You can use the regular expression validation box to test if your expression correctly matches a target string.

   For example, enter the regular expression .*sh$ in the left input box and the test string bash in the right input box. If the match is successful, a green checkmark appears on the right.

   The following table describes the configuration parameters.

   Parameter	Description	Example
   Enable node-level FastOOM	Select this option to enable node-level FastOOM. When enabled, node-level FastOOM monitors node memory pressure. When the node approaches an OOM state, it terminates processes based on the following configurations.	None
   Memory pressure level	Specifies the level of memory pressure at which to terminate processes. Options include Medium, High, and Auto. A Medium level means FastOOM intervenes and terminates processes sooner. The Auto option uses a proprietary Alibaba Cloud Linux kernel feature to prevent system stalls from memory reclamation. For offline workloads, we recommend setting this to High. For latency-sensitive workloads, we recommend setting this to Medium.	None
   Kill entire process group on OOM	If you select this option, FastOOM terminates all processes in the process group of the selected process.	None
   Preferred processes to kill	When the conditions for FastOOM to terminate a process are met, it prioritizes terminating the processes specified here. You can enter a list of regular expressions, separated by commas (,).	Java,^python indicates that processes whose names contain the substring "Java" or start with "python" are prioritized for termination.
   Processes to avoid killing	When the conditions for FastOOM to terminate a process are met, you can protect specific processes from termination. This ensures FastOOM does not terminate matched processes. You can enter a list of regular expressions, separated by commas (,).	.*sh$,mysql indicates that processes whose names end with "sh" or contain the substring "mysql" will not be terminated by user space FastOOM.

   Note

   • In scenarios with sudden memory spikes that exceed total system memory, the node-level FastOOM policy may not be able to intervene before a system OOM event occurs. Therefore, the FastOOM termination policy cannot guarantee that a matched process will not be terminated by the kernel OOM Killer.

   • If you do not configure policies for preferred or avoided processes, FastOOM uses the same algorithm as the kernel OOM Killer. It selects a process to terminate based on its memory consumption and itsoom_score_adj value.

   • If you select the "Auto" memory pressure option for node-level FastOOM, it activates the kernel-level FastOOM feature in Alibaba Cloud Linux. If the kernel version does not support this feature, it defaults to the user space FastOOM at the Medium pressure level. (This option is supported on Alibaba Cloud Linux kernel versions 5.10.134-017 and later that use cgroup v1).

   Pod-level FastOOM

   Click +, enter the configuration information, and click OK.

   Note

   To create multiple Pod-level FastOOM policies, click + again and enter the new configuration.

   Note

   On each node, a maximum of 10 Pods can be matched by the Pod name regular expression. If more than 10 Pods match, the policy does not apply to the excess Pods.

   Parameter	Description	Example
   Pod name	Use a regular expression to match Pod names. We recommend applying this policy to long-running, stable Pods.	sysom matches Pods whose names contain the substring "sysom".
   Namespace	The namespace where the matched Pods are located.	None
   Memory pressure level	Specifies the level of memory pressure at which to terminate processes. Options include Medium and High. A Medium level means FastOOM intervenes sooner. For offline workloads, we recommend High. For latency-sensitive workloads, we recommend Medium.	None
   Kill entire process group on OOM	If you select this option, FastOOM terminates all processes in the process group of the selected process.	None
   Preferred processes to kill	When the conditions for FastOOM to terminate a process are met, it prioritizes terminating the specified processes within the Pod. You can enter a list of regular expressions, separated by commas (,).	Java,^python indicates that processes whose names contain the substring "Java" or start with "python" are prioritized for termination.
   Processes to avoid killing	When the conditions for FastOOM to terminate a process are met, you can protect specific processes within the Pod from termination. This ensures that FastOOM does not terminate matched processes. You can enter a list of regular expressions, separated by commas (,).	.*sh$,mysql indicates that processes whose names end with "sh" or contain the substring "mysql" will not be terminated by FastOOM.

5. In the left-side navigation pane, click System Management. On the Managed tab, select the nodes where you want to apply the FastOOM feature, and then click Change Components.

6. Select the SysOM component configuration that you created and click Submit.

Modify FastOOM

1. Go to the Operating System Console - Component Management page.

2. Click Details for the SysOM component.

3. In the list of configurations, find the configuration you want to change and click Modify in the Actions column.

   The configuration list shows all configurations in a table, including their names and the number of associated nodes.

4. Modify the configuration information and click OK.

Disable FastOOM

1. Go to the Operating System Console - Component Management page.

2. Click Details for the SysOM component.

3. In the list of configurations, find the configuration you want to change and click Modify in the Actions column.

   The list shows the name of each configuration and the number of associated nodes.

   Note

   If you no longer need a configuration, find it in the list, click Delete in the Actions column, and then click OK in the confirmation message. If you do this, you can skip the remaining steps.

4. Disable FastOOM.

   • Disable the FastOOM feature

     Clear the Enable Custom FastOOM checkbox and click OK.

   • Disable a Pod-level FastOOM policy

     Click the delete icon for the policy, and then click OK.

Resolve system jitter with node-level FastOOM

Before an OOM event, systems often experience high CPU usage, high system load, and high IO wait, sometimes even stalling completely. This can cause service jitter or prolonged unresponsiveness. The problem is especially severe when total memory usage nears the threshold for a kernel OOM event. This example shows how to solve this problem by configuring the node-level user-space OOM feature.

Symptoms

An instance's memory usage begins to increase rapidly, approaching the system's total memory but not yet exceeding it. Because the limit has not been exceeded, the kernel OOM Killer is not triggered, and the system is in anear-OOM state.

Using the top command, you can see that the system's CPU sys usage, IO wait, and system load all remain high. The kswapd0 thread consumes significant CPU for memory reclamation.

top - 15:52:05 up 9 days,  5:02,  0 users,  load average: 27.02, 9.73, 3.86
Tasks: 192 total,   2 running, 138 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us, 43.0 sy,  0.0 ni,  0.5 id, 55.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7650272 total,   124224 free,  7399088 used,   126960 buff/cache
KiB Swap:        0 total,        0 free,        0 used.    45000 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   82 root      20   0       0      0      0 R  56.8  0.0   5:44.74 kswapd0
 3078 root      20   0  767676  25400      0 S  14.1  0.3  18:13.97 kube-proxy
 4731 root      20   0  231612  13952      0 R   2.0  0.2 161:01.80 AliYunDunMonito
4039635 65532   20   0  736560  10292      0 S   2.0  0.1   0:00.90 kube-rbac-proxy
 3462 root      20   0  735260  15652      0 S   1.4  0.2  18:35.64 node-cache
 4361 root      20   0 1300652  86524      0 S   1.4  1.1  39:08.82 ilogtail
4039689 root    20   0 2218752  57740   2084 D   1.4  0.8   0:01.77 unity-mon
 2138 root      20   0 1303484  67668      0 D   1.1  0.9 216:42.64 kubelet
 2483 root      20   0 2438684  10476      0 S   1.1  0.1  65:04.99 argusagent
 3078 root      20   0  767676  25400      0 S   1.1  0.3  18:13.39 kube-proxy
 3908 root      20   0  147952  34328      0 S   1.1  0.4  41:06.41 AliYunDun

The system overview in the Operating System Console shows that user-space packet receive latency occurred at the same time as the OOM event, causing service jitter.

The abnormal event list shows a Notice-level event for Userspace Packet Receive Latency and a Danger-level event for Node OOM Prediction and Detection, which indicates that system memory usage reached 7.62 GB with only 112.48 MB remaining.

Solution

In this scenario, use the FastOOM feature in the Alibaba Cloud Linux console to prevent near-OOM stalls. Enable the node-level FastOOM feature from the component configuration page. Because the service is latency-sensitive, set the memory pressure level to Medium. Then, configure the policy to exempt the service application (started with python, with a process name that contains the "python" substring) from being terminated by the OOM killer, and prioritize terminating an irrelevant logging program.

Here, Preferred processes to kill is set to logcollect.

After this is enabled, when the node's memory level enters a near-OOM state, user space FastOOM intervenes early and terminates processes according to your configuration, which releases memory and prevents the system from stalling. You can view the records of FastOOM interventions in the system overview in the Operating System Console.

The record shows that FastOOM terminated kube-rbac-proxy (8 MiB memory usage), node_exporter (8 MiB memory usage), unity-mon (50 MiB memory usage), and logcollect (1024 MiB memory usage).

Because theoom_score_adj value for processes likekube-rbac-proxy andnode_exporter is set close to 999, FastOOM matches the kernel policy and terminates these processes first. However, the amount of memory released by terminating these processes is small, and the system remains in anear-OOM state. FastOOM then terminates thelogcollect process, which was configured as a preferred kill target, preventing the system from experiencing jitter in thenear-OOM state.

Use Pod-level kill policies

Symptoms

The kernel provides the unique interface /proc/<PID>/oom_score_adj to control the OOM Killer's process selection strategy. Setting the value to -1000 completely exempts the process, which means it will not be killed by the OOM Killer. Setting the value to 1000 prioritizes the process to be killed. In a Kubernetes environment, if you want to control the OOM policies for multiple business scenarios running in a Pod, you need to modify the oom_score_adj of specific processes in the Pod. For short-lived daemon processes, it is difficult to modify the oom_score_adj.

Solution

The Operating System Console provides a flexible Pod-level OOM termination policy, allowing you to flexibly configure which processes within a Pod's containers to avoid or prioritize for termination when an OOM event occurs.

1. For example, a DaemonSet named test-alinux deploys corresponding Pods to each node in a cluster.

   The DaemonSet details page shows that the namespace is default and that three Pods are currently in the Running state on different nodes.

2. Configure the Pod-level FastOOM policy.

   • To match the corresponding Pods, set the Pod name to test-alinux (the regular expression will match Pods named test-alinux-xxx on different nodes) and the namespace to default.

   • To control the OOM termination policy, set the memory pressure level to High. This makes the timing of the user-space OOM trigger close to that of the kernel OOM trigger.

   • Configure the policy to prioritize terminating specific processes while protecting business-critical processes and the Pod's primary process (PID 1). This helps prevent Pod restarts and service interruptions.

   In the configuration, set Preferred processes to kill to malloc and Processes to avoid killing to unity,sh.

3. After the configuration is applied to the corresponding nodes, an OOM event occurs when the memory usage of a container within a Pod exceeds its limit. You can view the records of FastOOM interventions in the system overview in the Operating System Console.

   The health history details show a node-level user-space OOM event where FastOOM terminated the malloc process (399 MiB memory usage) and the test-malloc process (418 MiB memory usage).