Mitigate Linux memory fragmentation

Long-running Linux systems can develop memory fragmentation that causes performance jitters during high-order allocations. Use the following methods to reduce fragmentation proactively.

Symptoms

Services on the instance experience intermittent long response times or system call latency spikes. The sys CPU usage increases. The buddy system shows a shortage of high-order memory (blocks with order > 3). Run cat /proc/buddyinfo to check — starting from the fourth column, each column shows the free page count for that order:

cat /proc/buddyinfo 
Node 0, zone      DMA      1      0      0   1   2    1    1     0     1      1     3 
Node 0, zone    DMA32   3173    856    529   0   0    0    0     0     0      0     0 
Node 0, zone   Normal  19030   8688   7823   0   0    0    0     0     0      0     0

Cause

Over time, large contiguous physical memory blocks fragment into smaller non-contiguous blocks. When a service requests a large contiguous allocation, the kernel triggers memory compaction — a slow process that causes performance jitters. A typical fragmentation-related kernel stack trace:

0xffffffff8118f9cb compaction_alloc  ([kernel.kallsyms])
0xffffffff811c88a9 migrate_pages  ([kernel.kallsyms])
0xffffffff811901ee compact_zone  ([kernel.kallsyms])
0xffffffff8119041b compact_zone_order  ([kernel.kallsyms])
0xffffffff81190735 try_to_compact_pages  ([kernel.kallsyms])
0xffffffff81631cb4 __alloc_pages_direct_compact  ([kernel.kallsyms])
0xffffffff811741d5 __alloc_pages_nodemask  ([kernel.kallsyms])
0xffffffff811b5a79 alloc_pages_current  ([kernel.kallsyms])
0xffffffff811c0005 new_slab  ([kernel.kallsyms])
0xffffffff81633848 __slab_alloc  ([kernel.kallsyms])
0xffffffff811c5291 __kmalloc_node_track_caller  ([kernel.kallsyms])
0xffffffff8151a8c1 __kmalloc_reserve.isra.30  ([kernel.kallsyms])
0xffffffff8151b7cd alloc_sib  ([kernel.kallsyms])
0xffffffff815779e9 sk_stream_alloc_skb  ([kernel.kallsyms])
0xffffffff8157872d tcp_sendmsg  ([kernel.kallsyms])
0xffffffff815a26b4 inet_sendmsg  ([kernel.kallsyms])
0xffffffff81511017 sock_aio_write  ([kernel.kallsyms])
0xffffffff811df729 do_sync_readv_writev  ([kernel.kallsyms])
0xffffffff811e0cfe do_readv_writev  ([kernel.kallsyms])

Solutions

Configure flexible processing policies with the SysOM FastOOM feature in the operating system console to prevent unresponsive systems. Additionally, Use the following methods to mitigate memory fragmentation:

Adjust the min watermark
Set the min watermark to 1%–3% of total memory (2% recommended). This triggers asynchronous reclaim earlier when free memory is low. Run:
```
sysctl -w vm.min_free_kbytes = memtotal_kbytes * 2%
```
Replace memtotal_kbytes * 2% with 2% of your instance's total memory in KB.
Adjust the gap between the min and low watermarks
The watermark_scale_factor kernel parameter controls the gap between the min and low watermarks, helping absorb burst allocations. Default gap: 0.1% of total memory; minimum: half the min watermark. To adjust watermark_scale_factor, run:
```
sysctl -w vm.watermark_scale_factor = value
```
value is a scale factor that adjusts the gap between the min watermark and low watermark.
Periodically perform memory compaction
During off-peak hours, manually trigger asynchronous memory compaction:
```
echo 1 > /proc/sys/vm/compact_memory
```
Periodically drop the cache
If the previous methods are insufficient, drop the cache during a planned maintenance window. This frees memory pages for reallocation and effectively reduces fragmentation, but causes temporary performance jitters. Run:
```
echo 3 > /proc/sys/vm/drop_caches
```