DocsICP FilingConsole
Documentation
Home Page Container Compute Service User Guide PPU Zhenwu 810E Best practices Accelerate CI image builds with caching

Accelerate CI image builds with caching

Updated at:
Copy as MD
Product
My Favorites

This topic describes how to accelerate CI image builds for ACS Serverless Pods using a cache.

Docker-in-docker

Docker provides an official dind image that lets you build images inside a container using the Docker client. You can integrate this with frameworks like GitLab Runner or Tekton to create CI/CD workflows. The community-provided image is: docker pull docker.io/library/docker:27-dind.

apiVersion: v1
kind: Pod
metadata:
  name: dind-demo
spec:
  containers:
  - name: dind
    image: docker:27-dind
    securityContext:
      privileged: true
Note

The dind method requires the container to start in privileged mode.

Integrate Docker-in-docker with GitLab Runner

image

You can run build jobs using a shell executor or a docker executor.

Shell executor

A shell executor runs all jobs on the same machine, allowing you to store and reuse build caches on a mounted cloud disk.

The following code shows a shell executor configuration.

[[runners]]
  name = "my-dind-runner"
  url = "YOUR_GITLAB_URL"
  token = "YOUR_RUNNER_TOKEN"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.shell]
    [runners.shell.dind]
      enabled = true
      tls_verify = false
      image = "docker:19.03.12-dind"
      no_entrypoint = true
      disable_cache = false
      shm_size = 0

The following code shows a gitlab-ci.yaml file.

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build-image:
  stage: build
  script:
    - docker build -t my-app .
    - docker tag my-app $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

Docker executor

A docker executor provides improved isolation, as each build runs in a new dind container. The drawback is that reusing the cache requires explicit configuration.

The following code shows a docker executor configuration.

[[runners]]
  url = "https://gitlab.com/"
  token = REGISTRATION_TOKEN
  executor = "docker"
  [runners.docker]
    tls_verify = true
    image = "docker:19.03.12"
    privileged = true
    disable_cache = false
    volumes = ["/certs/client", "/cache"]

The following code shows a gitlab-ci.yaml file.

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from
$CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag
$CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
Note

In this example, we use the --cache-from option to use a remote image repository as a cache. You can also use a mounted volume and the --import-cache option to reference a cache. For more information, see Docker documentation for the local cache backend.

Build images in ACS with Kaniko

Kaniko is an image build tool that builds images from a Dockerfile within a container, eliminating the need for a Docker daemon. This makes it well-suited for large-scale builds in environments like ACS Serverless Pods.

The following example simulates a continuous delivery pipeline that uses NAS as shared storage to clone code and build images.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-nas
  namespace: default
  annotations:
    csi.alibabacloud.com/mountpoint: xxxx-mxm66.cn-beijing.nas.aliyuncs.com
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: alibaba-cloud-nas
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test-nas
    namespace: default
  csi:
    driver: nasplugin.csi.alibabacloud.com
    volumeAttributes:
      path: /
      server: xxx-mxm66.cn-beijing.nas.aliyuncs.com
      volumeHandle: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
  persistentVolumeReclaimPolicy: Retain
  storageClassName: alibaba-cloud-nas
  volumeMode: Filesystem
apiVersion: v1
kind: Pod
metadata:
  name: clone
spec:
  restartPolicy: Never
  containers:
    - name: clone
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/tckton-pipeline-git-init:latest
      command: ["/ko-app/git-init"]
      args:
        - "-url=https://gitee.com/AliyunContainerService/tekton-demo.git"
        - "-revision=master"
        - "-path=/workspace/code/tekton-demo"
        - "-depth=1"
        - "-terminationMessagePath=/workspace/code/terminatingmessage"
      volumeMounts:
        - name: dockerfile-storage
          mountPath: /workspace
  volumes:
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: test-nas
Note

Mount the shared storage to the /workspace path and clone the code into it. Subsequent build tasks can then read the code directly from this storage.

Build without a cache

For a standard build without a cache, the process reads the code from shared storage and builds the image, taking 126 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  restartPolicy: Never
  containers:
    - name: kaniko
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.0"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

image

Build with an ACR remote cache

The Kaniko --cache-repo option lets you cache build layers in a remote image repository, such as ACR. By default, it caches the RUN and COPY layers, but you can configure it to cache only one or the other. After layer caching is enabled, the build process takes 55 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-with-acr
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-with-acr
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

image

Build with ACR and a warm base image

The Kaniko project includes a sub-project called kaniko-warmer. This tool can cache a base image to a local directory. When combined with NAS shared storage, you can cache the base image for use in multi-application or concurrent builds. This build process takes 51 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmer
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmer
      image: registry.cn-beijing.aliyuncs.com/acs-demo-ns/kaniko-warmer:latest
      args:
        - "--cache-dir=/workspace/cache"
        - "--image=registry.cn-hangzhou.aliyuncs.com/knative-sample/golang:1.12"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
---
apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmed-up
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmed-up
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
        - "--cache-dir=/workspace/cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

image

Note

The optimal solution depends on your production environment. Factors to consider include your programming languages (such as Go, Python, or Java) and whether multiple applications are developed in parallel. For example, if your primary tech stack is Java and development occurs in parallel, you can use Extreme NAS as a shared cache to accelerate builds and optimize the Maven (MVN) process. Another strategy is to pre-warm a custom cache, for instance, by downloading a compressed MVN package for each build and adopting an incremental download method. The best approach depends on your specific use case.

Build images in ACS with BuildKit

Build with an ACR remote cache

Use the --export-cache option to push the cache to the same repository and the --import-cache option to import the cache.

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.6,push=true"
        - "--export-cache"
        - "type=inline"
        - "--import-cache"
        - "type=registry,ref=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.5"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

image

Build with a local cache

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.12,push=true"
        - "--export-cache"
        - "type=local,mode=max,compression=gzip,dest=/workspace/buildkit-cache"
        - "--import-cache"
        - "type=local,src=/workspace/buildkit-cache"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/buildkit-cache
          subPath: buildkit-cache
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

This example uses NAS for the cache. With caching enabled, the build takes about 62 seconds. Compared to Kaniko, layer reuse is slightly slower.

Kaniko cache:

image

BuildKit cache:

image

Build images with ACR's automated build feature

1. Create an image repository and select source

This example uses Alibaba Cloud Codeup to host the source code and trigger ACR builds.

image

2. Set build rules

You can configure build trigger rules, such as automatically building an image on a code commit. You can also use regular expressions to specify which commits trigger a build.

image

3. Reuse the build cache

In Default Mode, ACR automatically caches image build layers in the current image repository. Subsequent builds automatically use this cache.

image

image

Tests show that a build takes 64 seconds without caching and 21 seconds with caching.

Previous：Deploy the Kubeflow Community Edition on ACKNext：Develop custom container images for ACS PPU