DocsICP FilingConsole
Documentation
Home Page Container Compute Service User Guide PPU Zhenwu 810E Best practices Accelerate CI image builds with caching in ACS serverless pods

Accelerate CI image builds with caching in ACS serverless pods

Updated at:
Copy as MD
Product
My Favorites

This topic describes how to use a cache to accelerate CI image builds in ACS serverless pod scenarios.

Docker-in-Docker

Docker provides an official Docker-in-Docker (DinD) image that allows you to use the Docker client to build images inside a container. You can also integrate it with frameworks such as GitLab Runner or Tekton to create CI/CD workflows. The community provides the following image: docker pull docker.io/library/docker:27-dind.

apiVersion: v1
kind: Pod
metadata:
  name: dind-demo
spec:
  containers:
  - name: dind
    image: docker:27-dind
    securityContext:
      privileged: true
Note

The DinD method requires the container to start in privileged mode.

Integrate Docker-in-Docker with GitLab Runner

image

You can run build jobs by using the shell executor or the docker executor.

Shell executor

The shell executor treats a single machine like a VM. This allows you to store and reuse all build caches on the machine by mounting storage, such as a cloud disk.

The following example shows a shell executor configuration.

[[runners]]
  name = "my-dind-runner"
  url = "YOUR_GITLAB_URL"
  token = "YOUR_RUNNER_TOKEN"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.shell]
    [runners.shell.dind]
      enabled = true
      tls_verify = false
      image = "docker:19.03.12-dind"
      no_entrypoint = true
      disable_cache = false
      shm_size = 0

The following example shows a gitlab-ci.yaml file.

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build-image:
  stage: build
  script:
    - docker build -t my-app .
    - docker tag my-app $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

Docker executor

The docker executor provides better isolation because each build runs in a new DinD container. However, reusing the cache is more challenging and requires explicit referencing.

The following example shows a docker executor configuration.

[[runners]]
  url = "https://gitlab.com/"
  token = REGISTRATION_TOKEN
  executor = "docker"
  [runners.docker]
    tls_verify = true
    image = "docker:19.03.12"
    privileged = true
    disable_cache = false
    volumes = ["/certs/client", "/cache"]

The following example shows a gitlab-ci.yaml file.

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from
$CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag
$CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
Note

This example uses a remote image repository as a cache with the --cache-from flag. Alternatively, you can use a mount-based approach and reference a cache with --import-cache. For more information, see https://docs.docker.com/build/cache/backends/local/.

Build images with Kaniko in ACS

Kaniko is an image build tool that can build images within a container using the runtime interface without relying on the Docker daemon. This makes it highly scalable for ACS serverless pod scenarios.

The following example simulates a continuous delivery process that clones code and builds images using NAS as shared storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-nas
  namespace: default
  annotations:
    csi.alibabacloud.com/mountpoint: xxxx-mxm66.cn-beijing.nas.aliyuncs.com
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: alibaba-cloud-nas
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test-nas
    namespace: default
  csi:
    driver: nasplugin.csi.alibabacloud.com
    volumeAttributes:
      path: /
      server: xxx-mxm66.cn-beijing.nas.aliyuncs.com
      volumeHandle: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
  persistentVolumeReclaimPolicy: Retain
  storageClassName: alibaba-cloud-nas
  volumeMode: Filesystem
apiVersion: v1
kind: Pod
metadata:
  name: clone
spec:
  restartPolicy: Never
  containers:
    - name: clone
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/tekton-pipeline-git-init:latest
      command: ["/ko-app/git-init"]
      args:
        - "-url=https://gitee.com/AliyunContainerService/tekton-demo.git"
        - "-revision=master"
        - "-path=/workspace/code/tekton-demo"
        - "-depth=1"
        - "-terminationMessagePath=/workspace/code/terminatingmessage"
      volumeMounts:
        - name: dockerfile-storage
          mountPath: /workspace
  volumes:
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: test-nas
Note

Mount the shared storage to the /workspace path and clone the code into it. Subsequent build tasks can read from this location directly.

Build without a cache

A standard build that reads code from the shared storage path takes 126 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  restartPolicy: Never
  containers:
    - name: kaniko
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.0"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

Build with an ACR remote cache

kaniko supports caching build layers to a remote image repository, such as ACR, by using the --cache-repo flag. By default, kaniko caches the RUN and COPY layers. You can also specify to cache only the RUN or COPY layers. With layer caching, the build process took 55 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-with-acr
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-with-acr
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

Build with ACR and a warmed base image

The Kaniko project includes a sub-project called kaniko-warmer, which caches a base image to a local directory. By combining this with shared storage like NAS, you can cache the base image for multi-application or concurrent build scenarios. This reduces the build time to 51 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmer
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmer
      image: registry.cn-beijing.aliyuncs.com/acs-demo-ns/kaniko-warmer:latest
      args:
        - "--cache-dir=/workspace/cache"
        - "--image=registry.cn-hangzhou.aliyuncs.com/knative-sample/golang:1.12"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
---
apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmed-up
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmed-up
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
        - "--cache-dir=/workspace/cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
Note

The optimal solution for a production environment depends on specific factors, such as the programming languages used (for example, Go, Python, or Java) and whether development occurs in parallel across multiple applications and environments. For example, if your primary tech stack is Java and you have parallel development, you can use Extreme NAS as a shared cache to accelerate Maven (MVN) processes and builds. Alternatively, you can build the cache at a higher level in your application logic and download a compressed MVN package for each build. This approach combines an existing cache with incremental downloads. The best strategy requires an analysis of your specific business scenario.

Build images with BuildKit in ACS

Build with an ACR remote cache

Use --export-cache to push the cache to a repository and --import-cache to retrieve it for a build.

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.6,push=true"
        - "--export-cache"
        - "type=inline"
        - "--import-cache"
        - "type=registry,ref=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.5"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

Build with a local cache

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.12,push=true"
        - "--export-cache"
        - "type=local,mode=max,compression=gzip,dest=/workspace/buildkit-cache"
        - "--import-cache"
        - "type=local,src=/workspace/buildkit-cache"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/buildkit-cache
          subPath: buildkit-cache
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

The cache is stored on NAS. A build that reuses this cache takes about one minute. This is slightly slower than Kaniko when reusing layers.

Integrate ACR for builds in ACS

Step 1: Create image repository and select code source

This example shows how to use Alibaba Cloud DevOps Codeup to host your code and integrate it with ACR as a code source.

Step 2: Configure build rules

In the Build Settings, you can enable Automatically build images on code change and configure builds to trigger based on code commits that match a regular expression.

Step 3: Reuse build cache

ACR enables image caching by default and stores the cache in the current repository. ACR automatically reuses this cache in subsequent builds, which can improve build efficiency by up to three times.

Previous：Deploy the Kubeflow Community Edition on ACKNext：Develop custom container images for ACS PPU