Use p2p acceleration

Background

When many nodes in a large-scale container cluster pull an image at the same time, the network bandwidth of the container image registry can become a bottleneck, which slows down image pulls. The p2p acceleration feature uses the bandwidth of compute nodes to distribute images. This reduces pressure on the container image registry, significantly accelerates image pulls, and shortens application deployment time. In a test on a 1,000-node cluster pulling a 1 GB image, p2p acceleration reduced the image pull time by more than 95% compared to standard image pulls over a 10 Gbps network. Additionally, the new p2p acceleration solution is 30% to 50% faster than the previous version. By default, the new solution supports using p2p acceleration when you load container images on demand. For more information, see Load resources of a container image on demand.

You can use p2p acceleration in the following scenarios:

• ACK clusters

• On-premises clusters or clusters from third-party cloud service providers

Prerequisites

A p2p acceleration agent is installed.

• If you want to use p2p acceleration in an ACK cluster, see Install a P2P acceleration agent in an ACK cluster.

• If you want to use p2p acceleration in an on-premises cluster or a cluster from a third-party cloud service provider, see Install a P2P acceleration agent in an on-premises cluster or a cluster of a third-party cloud service provider.

Limitations

When you enable p2p acceleration, the p2p acceleration agent uses a webhook to replace your container image address with a p2p image address. For example, if your original image address is test****vpc.cn-hangzhou.cr.aliyuncs.com/docker-builder/nginx:latest, the new p2p-accelerated image address is test****vpc.distributed.cn-hangzhou.cr.aliyuncs.com:65001/docker-builder/nginx:latest.

The webhook also automatically generates an image pull secret for the accelerated image address by copying the original image pull secret. The creation of the p2p image pull secret and the replacement of the image address are asynchronous processes. To prevent image pull failures, apply the required image pull secret before you deploy the workload. Alternatively, you can manually create an image pull secret for p2p image pulls using the domain test-registry-vpc.distributed.cn-hangzhou.cr.aliyuncs.com:65001, and then deploy the workload.

Enable p2p acceleration

• Add the p2p acceleration label to an application workload.

  The following example shows how to add a label to a Deployment. Run the following command to edit the Deployment:

  kubectl edit deploy <Deployment-Name>

  Add the k8s.aliyun.com/image-accelerate-mode: p2p label to the Deployment file.

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: test
    labels:
      app: nginx
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          # Enable p2p.
          k8s.aliyun.com/image-accelerate-mode: p2p
          app: nginx
      spec:
        # Your ACR instance image pull secret.
        imagePullSecrets:
        - name: test-registry
        containers:
        # Your ACR instance image.
        - image: test-registry-vpc.cn-hangzhou.cr.aliyuncs.com/docker-builder/nginx:latest
          name: test
          command: ["sleep", "3600"]

• Add the p2p acceleration label to a namespace

  • Add the p2p acceleration label by using the console.

    1. Log on to the ACK console. In the left-side navigation pane, choose Clusters.

    2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Namespaces and Quotas.

    3. On the Namespace page, find the target namespace and click Edit in the Actions column.

    4. In the Edit the namespace dialog box, click +Labels, set Variable Key to k8s.aliyun.com/image-accelerate-mode, set Variable Value to p2p, and then click Confirm.

  • Add the p2p acceleration label by using the CLI.

    kubectl label namespaces <YOUR-NAMESPACE> k8s.aliyun.com/image-accelerate-mode=p2p

Verify p2p acceleration

Run the following command to view the pod:

kubectl get po <Pod-Name> -oyaml

Expected output:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    # Injects p2p annotations automatically.
    k8s.aliyun.com/image-accelerate-mode: p2p
    k8s.aliyun.com/p2p-config: '...'
spec:
  containers:
   # Injects the image to the p2p endpoint.
   - image: test-registry-vpc.distributed.cn-hangzhou.cr.aliyuncs.com:65001/docker-builder/nginx:latest
  imagePullSecrets:
  - name: test-registry
  # Injects the image pull secret for the p2p endpoint.
  - name: acr-credential-test-registry-p2p

The pod is injected with the p2p-related annotation, p2p-accelerated image address, and the corresponding image pull credential. This confirms that p2p acceleration is enabled.

(Optional) Enable client metric collection

p2p:

  v2:
    # Component for P2P v2.
    image: registry-vpc.__ACK_REGION_ID__.aliyuncs.com/acs/dadi-agent
    imageTag: v0.1.2-72276d4-aliyun

    # The maximum number of layers that can be concurrently downloaded by each node proxy.
    proxyConcurrencyLimit: 128

    # The server port to communicate with P2P nodes.
    p2pPort: 65002

    cache:
      # The disk cache capacity in bytes. Default value: 4 GB.
      capacity: 4294967296
      # Set this parameter to 1 if you are using high-performance disks, such as ESSD PL2/PL3, on your ECS instance.
      aioEnable: 0
    exporter:
      # Set this parameter to true if you want to collect component metrics.
      enable: false
      port: 65003

    # The limit for downstream throughput.
    throttleLimitMB: 512

ExporterConfig:
  enable: true # Specifies whether to enable metrics.
  port: 65006  # The listening port.
  standaloneExporterPort: true # Specifies whether to expose a standalone port. If this parameter is set to false, metrics are exported through the HTTP service port.

# HELP DADIP2P_Alive 
# TYPE DADIP2P_Alive gauge
DADIP2P_Alive{node="192.168.69.172:65005",mode="agent"} 1.000000 1692156721833

# HELP DADIP2P_Read_Throughtput Bytes / sec
# TYPE DADIP2P_Read_Throughtput gauge
DADIP2P_Read_Throughtput{node="192.168.69.172:65005",type="pread",mode="agent"} 0.000000 1692156721833
DADIP2P_Read_Throughtput{node="192.168.69.172:65005",type="download",mode="agent"} 0.000000 1692156721833
DADIP2P_Read_Throughtput{node="192.168.69.172:65005",type="peer",mode="agent"} 0.000000 1692156721833
DADIP2P_Read_Throughtput{node="192.168.69.172:65005",type="disk",mode="agent"} 0.000000 1692156721833
DADIP2P_Read_Throughtput{node="192.168.69.172:65005",type="http",mode="agent"} 0.000000 1692156721833

# HELP DADIP2P_QPS 
# TYPE DADIP2P_QPS gauge
DADIP2P_QPS{node="192.168.69.172:65005",type="pread",mode="agent"} 0.000000 1692156721833
DADIP2P_QPS{node="192.168.69.172:65005",type="download",mode="agent"} 0.000000 1692156721833
DADIP2P_QPS{node="192.168.69.172:65005",type="peer",mode="agent"} 0.000000 1692156721833
DADIP2P_QPS{node="192.168.69.172:65005",type="disk",mode="agent"} 0.000000 1692156721833
DADIP2P_QPS{node="192.168.69.172:65005",type="http",mode="agent"} 0.000000 1692156721833

# HELP DADIP2P_MaxLatency us
# TYPE DADIP2P_MaxLatency gauge
DADIP2P_MaxLatency{node="192.168.69.172:65005",type="pread",mode="agent"} 0.000000 1692156721833
DADIP2P_MaxLatency{node="192.168.69.172:65005",type="download",mode="agent"} 0.000000 1692156721833
DADIP2P_MaxLatency{node="192.168.69.172:65005",type="peer",mode="agent"} 0.000000 1692156721833
DADIP2P_MaxLatency{node="192.168.69.172:65005",type="disk",mode="agent"} 0.000000 1692156721833
DADIP2P_MaxLatency{node="192.168.69.172:65005",type="http",mode="agent"} 0.000000 1692156721833

# HELP DADIP2P_Count Bytes
# TYPE DADIP2P_Count gauge
DADIP2P_Count{node="192.168.69.172:65005",type="pread",mode="agent"} 0.000000 1692156721833
DADIP2P_Count{node="192.168.69.172:65005",type="download",mode="agent"} 0.000000 1692156721833
DADIP2P_Count{node="192.168.69.172:65005",type="peer",mode="agent"} 0.000000 1692156721833
DADIP2P_Count{node="192.168.69.172:65005",type="disk",mode="agent"} 0.000000 1692156721833
DADIP2P_Count{node="192.168.69.172:65005",type="http",mode="agent"} 0.000000 1692156721833

# HELP DADIP2P_Cache 
# TYPE DADIP2P_Cache gauge
DADIP2P_Cache{node="192.168.69.172:65005",type="allocated",mode="agent"} 4294967296.000000 1692156721833
DADIP2P_Cache{node="192.168.69.172:65005",type="used",mode="agent"} 4294971392.000000 1692156721833

# HELP DADIP2P_Label 
# TYPE DADIP2P_Label gauge

DADIP2P_Count{node="11.238.108.XXX:9877",type="http",mode="agent"} 4248808352.000000 1692157615810
Total HTTP request traffic processed by the agent service: 4,248,808,352 bytes.

DADIP2P_Cache{node="11.238.108.XXX:9877",type="used",mode="agent"} 2147487744.000000 1692157615810
Current cache usage of the agent: 2,147,487,744 bytes.

DeployConfig:
  mode: agent
  logDir: /dadi-p2p/log
  logAudit: true
  logAuditMode: stdout # Sends logs to the console. Set the value to file to write logs to /dadi-p2p/log/audit.log.

2022/08/30 15:44:52|AUDIT|th=00007FBA247C5280|download[pathname=/https://cri-pi840la*****-registry.oss-cn-hangzhou.aliyuncs.com/docker/registry/v2/blobs/sha256/dd/dd65726c224b09836aeb6ecebd6baf58c96be727ba86da14e62835569896008a/data][offset=125829120][size=2097152][latency=267172]
....
2022/08/30 15:44:55|AUDIT|th=00007FBA2EFEAEC0|http:pread[pathname=/https://cri-pi840lacia*****-registry.oss-cn-hangzhou.aliyuncs.com/docker/registry/v2/blobs/sha256/dd/dd65726c224b09836aeb6ecebd6baf58c96be727ba86da14e62835569896008a/data][offset=127467520][size=65536][latency=21]

download[pathname=mytest][offset=0][size=65536][latency=26461]
  ## The latency when the p2p agent downloads the data segment [0,65536) of the mytest file from the upstream source is 26,461 μs.
rpc:pread[pathname=mytest][offset=0][size=65536][latency=2]
  ## The latency when the p2p agent returns the data segment [0,65536) of the mytest file to the downstream client is 2 μs.
http:pread[pathname=mytest][offset=0][size=65536][latency=26461]
  ## The latency when the proxy downloads the data segment [0,65536) of the mytest file from the upstream source is 26,461 μs.