Best practices for avoiding Split Locks-Elastic Compute Service(ECS)-阿里云帮助中心

Detect and prevent Split Locks — atomic operations that cross cache lines — to avoid system-wide performance degradation in concurrent or HPC workloads.

How it works

When an atomic operation's operand spans two cache lines, the processor locks the bus to ensure atomicity. All other cores pause memory access until the operation completes.

A Split Lock occurs when both conditions are met:

Atomic operation: An assembly instruction with a LOCK prefix executes.
Cross-cache-line access: The operand address is not aligned and spans two cache lines.

Split Lock example

The buf member in struct counter occupies 62 bytes. Due to alignment rules, the 8-byte c member may start near byte 62 of a 64-byte cache line, spanning two lines. When __sync_fetch_and_add atomically increments c, a Split Lock is triggered.

#include <stdio.h>
#include <sys/mman.h>
#pragma pack(push,2)
struct counter
{
    char buf[62];
    long long c;
};
#pragma pack(pop)
int main () {
    struct counter *p;
    int size = sizeof(struct counter);
    int prot = PROT_READ | PROT_WRITE;
    int flags = MAP_PRIVATE | MAP_ANONYMOUS;
    p = (struct counter *) mmap(0, size, prot, flags, -1, 0);
    while(1) {
        __sync_fetch_and_add(&p->c, 1);
    }
    return 0;
}

Impact of Split Locks

Global performance degradation: A Split Lock blocks memory access for all processes on the system, not just the triggering process.
Increased latency: Frequent Split Locks increase memory access latency and cause system performance jitter.

Detect Split Locks

Important

Detection is supported on g9i, c9i, r9i, g8i, c8i, and r8i instances.

For example, on an ecs.g8i.xlarge instance, a counts value greater than 0 indicates Split Locks have occurred.

perf stat -e cpu/event=0x2c,umask=0x10/ -a -I 1000

[root@i-xxx ~]# perf stat -e cpu/event=0x2c,umask=0x10/ -a -I 1000
#           time             counts unit events
     1.007775384             13,111      cpu/event=0x2c,umask=0x10/
     2.009778256             13,808      cpu/event=0x2c,umask=0x10/
     3.010775132             13,689      cpu/event=0x2c,umask=0x10/

If a memory access does not cross a cache line, the corresponding count is 0.

[root@ixxxxxx ~]# perf stat -e cpu/event=0x2c,umask=0x10/ -a -I 1000
#           time             counts unit events
     1.001129037                  0      cpu/event=0x2c,umask=0x10/
     2.002289568                  0      cpu/event=0x2c,umask=0x10/
     3.003416893                  0      cpu/event=0x2c,umask=0x10/

Avoid Split Locks

Align atomic variables to their natural boundaries.

// Recommended: Align to 64 bytes to avoid crossing cache lines and false sharing.
alignas(64) atomic<uint64_t> counter;

// For 128-bit atomic types, align to 16 bytes.
alignas(16) atomic<__int128> big_counter;

Place large atomic variables at the beginning of a struct.

// Not recommended: The atomic variable can be shifted by preceding members, causing it to cross a cache line.
struct BadExample {
    char a;                    // Occupies 1 byte
    atomic<__int128> val;     // May cross a cache line
};

// Recommended: Place the member with the strictest alignment requirement first and declare it explicitly.
struct GoodExample {
    alignas(16) atomic<__int128> val;
    char a;
};

Split large atomic types into smaller ones. If true 128-bit atomicity is not required, use two 64-bit atomic operations instead.
```
struct PaddedCounter {
    alignas(64) atomic<uint64_t> low;
    alignas(64) atomic<uint64_t> high;
};
```

Use aligned pointers for atomic operations.

// Incorrect: malloc does not guarantee 16-byte alignment, especially with older libc versions.
void* ptr = malloc(sizeof(atomic<__int128>));
atomic<__int128>* p = new(ptr) atomic<__int128>;

// Correct: Use aligned_alloc.
void* aligned_ptr = aligned_alloc(16, sizeof(atomic<__int128>));
atomic<__int128>* p = new(aligned_ptr) atomic<__int128>;

Verify alignment with static_assert.

static_assert(alignof(atomic<__int128>) >= 16, "128-bit atomic must be 16-byte aligned");

Do not use atomic types in packed structs.

#pragma pack(push, 1)
struct Packed {
    uint8_t flag;
    atomic<uint64_t> counter; // Incorrect: Even an 8-byte type can cross a cache line due to the compact layout.
};
#pragma pack(pop)