Enable container memory QoS

Advantages over open-source Kubernetes memory QoS

The upstream Kubernetes memory QoS feature (Kubernetes 1.22+) supports only cgroup v2, requires manual kubelet configuration, and lacks per-pod or per-namespace granularity.

ack-koordinator improves on the upstream implementation in two ways:

• Broader kernel compatibility: Supports cgroup v1 and v2, backed by Alinux kernel features such as memcg backend asynchronous reclamation and minimum watermark rating. See Overview of kernel features and interfaces.

• Fine-grained configuration: Configure memory QoS per pod, namespace, or cluster through pod annotations or ConfigMaps.

Configuration mechanism

ack-koordinator uses four cgroup parameters for memory QoS. Each maps to configuration options in Advanced parameters:

Configuration priority

When multiple configuration sources apply to the same pod, ack-koordinator uses the following priority order (highest first):

1. Pod annotation (koordinator.sh/memoryQOS)

2. Namespace-level ConfigMap (ack-slo-pod-config)

3. Cluster-level ConfigMap (ack-slo-config)

QoS class mapping

If a pod lacks the koordinator.sh/qosClass label, ack-koordinator maps Kubernetes QoS classes automatically:

Run the test

1. Create redis-demo.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: redis-demo-config
   data:
     redis-config: |
       appendonly yes
       appendfsync no
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: redis-demo
     labels:
       koordinator.sh/qosClass: 'LS'
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "auto"}'
   spec:
     containers:
     - name: redis
       image: redis:5.0.4
       command:
         - redis-server
         - "/redis-master/redis.conf"
       env:
       - name: MASTER
         value: "true"
       ports:
       - containerPort: 6379
       resources:
         limits:
           cpu: "2"
           memory: "6Gi"
         requests:
           cpu: "2"
           memory: "2Gi"
       volumeMounts:
       - mountPath: /redis-master-data
         name: data
       - mountPath: /redis-master
         name: config
     volumes:
       - name: data
         emptyDir: {}
       - name: config
         configMap:
           name: redis-demo-config
           items:
           - key: redis-config
             path: redis.conf
     nodeName: # Set to the name of the node running Redis.
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: redis-demo
   spec:
     ports:
     - name: redis-port
       port: 6379
       protocol: TCP
       targetPort: 6379
     selector:
       name: redis-demo
     type: ClusterIP

2. Deploy Redis:

   kubectl apply -f redis-demo.yaml

3. Simulate memory overcommitment with the Stress tool. Create stress-demo.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: stress-demo
     labels:
       koordinator.sh/qosClass: 'BE'
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "auto"}'
   spec:
     containers:
       - args:
           - '--vm'
           - '2'
           - '--vm-bytes'
           - 11G
           - '-c'
           - '2'
           - '--vm-hang'
           - '2'
         command:
           - stress
         image: polinux/stress
         imagePullPolicy: Always
         name: stress
     restartPolicy: Always
     nodeName: # Set to the same node as redis-demo.

4. Deploy the stress workload:

   kubectl apply -f stress-demo.yaml

5. Verify the global minimum watermark before running the benchmark.

   Important

   In memory overcommitment scenarios, a low global minimum watermark causes the OOM killer to run before memory reclamation. For a 32 GB node, set this value to at least 4,000,000 KB.

   cat /proc/sys/vm/min_free_kbytes

   Expected output:

   4000000

6. Deploy memtier-benchmark to send requests to the Redis pod:

   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       name: memtier-demo
     name: memtier-demo
   spec:
     containers:
       - command:
           - memtier_benchmark
           - '-s'
           - 'redis-demo'
           - '--data-size'
           - '200000'
           - "--ratio"
           - "1:4"
         image: 'redislabs/memtier_benchmark:1.3.0'
         name: memtier
     restartPolicy: Never
     nodeName: # Set to the name of the node sending requests.

7. Check the benchmark results:

   kubectl logs -f memtier-demo

8. To compare, disable memory QoS on both pods and repeat the test:

   apiVersion: v1
   kind: Pod
   metadata:
     name: redis-demo
     labels:
       koordinator.sh/qosClass: 'LS'
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "none"}'
   spec:
     ...
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: stress-demo
     labels:
       koordinator.sh/qosClass: 'BE'
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "none"}'

Test results

Enabling memory QoS reduced Redis latency by 7.9% and increased throughput by 8.7% under memory overcommitment.

Is the memory QoS configuration from ack-slo-manager still valid after upgrading to ack-koordinator?

Yes. ack-koordinator is backward compatible with the annotation-based protocol used in ack-slo-manager 0.8.0 and earlier:

• alibabacloud.com/qosClass — sets the QoS class

• alibabacloud.com/memoryQOS — configures memory QoS

The following table shows which protocols each version supports:

Support for the alibabacloud.com protocol ended on July 30, 2023. Migrate to the koordinator.sh protocol.