AI 助理
文档备案控制台
官方文档
输入文档关键字查找
首页

ACS Serverless Pod场景使⽤缓存加速CI镜像构建

更新时间：
复制为 MD 格式
我的收藏

本文介绍如何在ACS Serverless Pod场景中使用缓存加速CI镜像构建。

Docker In Docker

Docker官方提供dind方式的镜像，支持用户在容器内使用docker客户端进行镜像的构建。并且可以通过集成如gitlab runner或tekton等框架的方式进行CI/CD的工作流集成。社区提供镜像：docker pull docker.io/library/docker:27-dind。

apiVersion: v1
kind: Pod
metadata:
  name: dind-demo
spec:
  containers:
  - name: dind
    image: docker:27-dind
    securityContext:
      privileged: true
说明

dind的方式需要容器以privileged方式启动。

使⽤Gitlab Runner集成Docker-In-Docker

image

您可以通过shell executor或者docker executor的方式执行构建job。

shell executor

使用shell executor的好处是将单机作为VM模式使用，所有构建缓存都可以在单机上通过挂载云盘等方式进行缓存和复用。

以下为shell executor。

[[runners]]
  name = "my-dind-runner"
  url = "YOUR_GITLAB_URL"
  token = "YOUR_RUNNER_TOKEN"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.shell]
    [runners.shell.dind]
      enabled = true
      tls_verify = false
      image = "docker:19.03.12-dind"
      no_entrypoint = true
      disable_cache = false
      shm_size = 0

以下为gitlab-ci.yaml。

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build-image:
  stage: build
  script:
    - docker build -t my-app .
    - docker tag my-app $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

docker executor

docker executor的好处是隔离性更优，每次构建都会启动一个dind容器来执行，缺点是缓存复用问题，需要我们显式开启缓存引用。

以下为docker executor。

[[runners]]
  url = "https://gitlab.com/"
  token = REGISTRATION_TOKEN
  executor = "docker"
  [runners.docker]
    tls_verify = true
    image = "docker:19.03.12"
    privileged = true
    disable_cache = false
    volumes = ["/certs/client", "/cache"]

以下为gitlab-ci.yaml。

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from
$CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag
$CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
说明

这⾥我们通过--cache-from的⽅式使⽤远程镜像仓库作为缓存，当然也可以基于挂载的⽅式使⽤--import-cache来引用缓存 https://docs.docker.com/build/cache/backends/local/。

ACS中通过kaniko构建镜像

Kaniko是⼀个镜像构建⼯具，⽀持在容器内进⾏镜像构建（使⽤运⾏时接⼝，不依赖docker-daemon），因此可以⼤规模应⽤于ACS Serverless Pod场景。

以下示例模拟客户持续交付流程，通过NAS作为共享存储进⾏代码克隆和镜像构建。

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-nas
  namespace: default
  annotations:
    csi.alibabacloud.com/mountpoint: xxxx-mxm66.cn-beijing.nas.aliyuncs.com
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: alibaba-cloud-nas
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test-nas
    namespace: default
  csi:
    driver: nasplugin.csi.alibabacloud.com
    volumeAttributes:
      path: /
      server: xxx-mxm66.cn-beijing.nas.aliyuncs.com
      volumeHandle: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
  persistentVolumeReclaimPolicy: Retain
  storageClassName: alibaba-cloud-nas
  volumeMode: Filesystem
apiVersion: v1
kind: Pod
metadata:
  name: clone
spec:
  restartPolicy: Never
  containers:
    - name: clone
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/tckton-pipeline-git-init:latest
      command: ["/ko-app/git-init"]
      args:
        - "-url=https://gitee.com/AliyunContainerService/tekton-demo.git"
        - "-revision=master"
        - "-path=/workspace/code/tekton-demo"
        - "-depth=1"
        - "-terminationMessagePath=/workspace/code/terminatingmessage"
      volumeMounts:
        - name: dockerfile-storage
          mountPath: /workspace
  volumes:
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: test-nas
说明

挂载共享存储到/workspace路径，执⾏代码克隆⾄共享存储中，后续构建任务直接读取。

Build Without Cache

使用普通构建模式，读取共享存储路径的代码并进行构建，构建过程耗时126秒。

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  restartPolicy: Never
  containers:
    - name: kaniko
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.0"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

image

Build With ACR Remote Cache

kaniko支持通过--cache-repo方式指定将构建层缓存至远端镜像仓库（本文为ACR），默认会将RUN&COPY层进行缓存，可以按需指定仅缓存RUN或COPY层。使用层缓存后构建过程耗时55秒。

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-with-acr
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-with-acr
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

image

Build With ACR and Base Image Warmed Up

kaniko项⽬中提供⼀个kaniko-warm的⼦项⽬，可以将基础镜像缓存⾄本地⽬录，结合共享存储NAS的特性，可以将其缓存⾄共享存储中，提供给多应用构建或并发构建场景使⽤。构建过程耗时51秒。

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmer
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmer
      image: registry.cn-beijing.aliyuncs.com/acs-demo-ns/kaniko-warmer:latest
      args:
        - "--cache-dir=/workspace/cache"
        - "--image=registry.cn-hangzhou.aliyuncs.com/knative-sample/golang:1.12"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
---
apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmed-up
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmed-up
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
        - "--cache-dir=/workspace/cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

image

说明

具体到客户的⽣产环境，⽐如是否有多语⾔（go、python、java等），是否有多应⽤多环境并⾏研发，会存在不同的解决⽅案。⽐如当涉及到主流技术栈为java，存在并⾏研发时，考虑到MVN过程的效率优化，我们可以使⽤极速NAS作为共享缓存加速构建，或者业务在更上层构建缓存，每次构建时下载MVN压缩包，采⽤存量缓存+增量下载的⽅式，需要结合业务场景来看。

ACS中通过buildkit构建镜像

Build With ACR Remote Cache

通过--export-cache将缓存推送至同仓库，--import-cache来使用缓存。

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.6,push=true"
        - "--export-cache"
        - "type=inline"
        - "--import-cache"
        - "type=registry,ref=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.5"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

image

Build With Local Cache

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.12,push=true"
        - "--export-cache"
        - "type=local,mode=max,compression=gzip,dest=/workspace/buildkit-cache"
        - "--import-cache"
        - "type=local,src=/workspace/buildkit-cache"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/buildkit-cache
          subPath: buildkit-cache
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

缓存我们使⽤了NAS，在复⽤缓存的情况下构建耗时62秒左右，对⽐kaniko来看在复⽤层的场景下略慢。

kaniko缓存：

image

buildkit缓存：

image

ACS集成ACR进⾏进⾏构建

1. 创建镜像仓库并选择代码源

本⽂使⽤云效Codeup托管代码，并集成⾄ACR构建代码源中。

image

2. 设置构建规则

可以设置构建触发规则是否提交代码时⾃动构建，以及根据正则表达式匹配代码提交进⾏代码构建。

image

3. 构建缓存复⽤

默认模式下ACR会对镜像构建进⾏缓存，缓存⾄当前镜像仓库，下⼀次镜像构建时⾃动使⽤镜像缓存。

image

image

测试下来非缓存模式构建64秒，缓存后构建21秒。

上一篇：在ACS上部署社区版Kubeflow下一篇：ACS PPU用户容器镜像开发实践