AI 助理
文档备案控制台
官方文档
输入文档关键字查找
首页

ACS Serverless Pod场景使用缓存加速CI镜像构建

更新时间：
复制为 MD 格式
产品详情
我的收藏

本文介绍如何在ACS Serverless Pod场景中使用缓存加速CI镜像构建。

Docker In Docker

Docker官方提供Docker-in-Docker（DinD）方式的镜像，支持用户在容器内使用docker客户端进行镜像的构建。并且可以通过集成如gitlab runner或tekton等框架的方式进行CI/CD的工作流集成。社区提供镜像：docker pull docker.io/library/docker:27-dind。

apiVersion: v1
kind: Pod
metadata:
  name: dind-demo
spec:
  containers:
  - name: dind
    image: docker:27-dind
    securityContext:
      privileged: true
说明

dind方式需要容器以privileged方式启动。

使用Gitlab Runner集成Docker-In-Docker

image

可通过shell executor或者docker executor的方式执行构建job。

shell executor

使用shell executor的好处是将单机作为VM模式使用，所有构建缓存都可以在单机上通过挂载云盘等方式进行缓存和复用。

以下为shell executor。

[[runners]]
  name = "my-dind-runner"
  url = "YOUR_GITLAB_URL"
  token = "YOUR_RUNNER_TOKEN"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.shell]
    [runners.shell.dind]
      enabled = true
      tls_verify = false
      image = "docker:19.03.12-dind"
      no_entrypoint = true
      disable_cache = false
      shm_size = 0

以下为gitlab-ci.yaml。

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build-image:
  stage: build
  script:
    - docker build -t my-app .
    - docker tag my-app $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

docker executor

docker executor的好处是隔离性更优，每次构建都会启动一个dind容器执行，缺点是存在缓存复用问题，需要显式开启缓存引用。

以下为docker executor。

[[runners]]
  url = "https://gitlab.com/"
  token = REGISTRATION_TOKEN
  executor = "docker"
  [runners.docker]
    tls_verify = true
    image = "docker:19.03.12"
    privileged = true
    disable_cache = false
    volumes = ["/certs/client", "/cache"]

以下为gitlab-ci.yaml。

default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from
$CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag
$CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
说明

这里通过--cache-from的方式使用远程镜像仓库作为缓存，当然也可以基于挂载的方式使用--import-cache引用缓存 https://docs.docker.com/build/cache/backends/local/。

ACS中通过kaniko构建镜像

Kaniko是一个镜像构建工具，支持在容器内进行镜像构建（使用运行时接口，不依赖docker-daemon），因此可以大规模应用于ACS Serverless Pod场景。

以下示例模拟持续交付流程，通过NAS作为共享存储进行代码克隆和镜像构建。

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-nas
  namespace: default
  annotations:
    csi.alibabacloud.com/mountpoint: xxxx-mxm66.cn-beijing.nas.aliyuncs.com
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: alibaba-cloud-nas
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test-nas
    namespace: default
  csi:
    driver: nasplugin.csi.alibabacloud.com
    volumeAttributes:
      path: /
      server: xxx-mxm66.cn-beijing.nas.aliyuncs.com
      volumeHandle: nas-36ab0c3c-ef49-48de-ad19-4a4d1e54ac4a
  persistentVolumeReclaimPolicy: Retain
  storageClassName: alibaba-cloud-nas
  volumeMode: Filesystem
apiVersion: v1
kind: Pod
metadata:
  name: clone
spec:
  restartPolicy: Never
  containers:
    - name: clone
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/tckton-pipeline-git-init:latest
      command: ["/ko-app/git-init"]
      args:
        - "-url=https://gitee.com/AliyunContainerService/tekton-demo.git"
        - "-revision=master"
        - "-path=/workspace/code/tekton-demo"
        - "-depth=1"
        - "-terminationMessagePath=/workspace/code/terminatingmessage"
      volumeMounts:
        - name: dockerfile-storage
          mountPath: /workspace
  volumes:
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: test-nas
说明

挂载共享存储到/workspace路径，执行代码克隆至共享存储中，后续构建任务直接读取。

Build Without Cache

使用普通构建模式，读取共享存储路径的代码并进行构建，构建过程耗时126秒。

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  restartPolicy: Never
  containers:
    - name: kaniko
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.0"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

Build With ACR Remote Cache

kaniko支持通过--cache-repo方式指定将构建层缓存至远端镜像仓库（本文为ACR），默认会将RUN和COPY层进行缓存，可以按需指定仅缓存RUN或COPY层。使用层缓存后构建过程耗时55秒。

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-with-acr
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-with-acr
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas

Build With ACR and Base Image Warmed Up

kaniko项目中提供一个kaniko-warm的子项目，可以将基础镜像缓存至本地目录，结合共享存储NAS的特性，可以将其缓存至共享存储中，提供给多应用构建或并发构建场景使用。构建过程耗时51秒。

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmer
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmer
      image: registry.cn-beijing.aliyuncs.com/acs-demo-ns/kaniko-warmer:latest
      args:
        - "--cache-dir=/workspace/cache"
        - "--image=registry.cn-hangzhou.aliyuncs.com/knative-sample/golang:1.12"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
---
apiVersion: v1
kind: Pod
metadata:
  name: kaniko-warmed-up
spec:
  restartPolicy: Never
  containers:
    - name: kaniko-warmed-up
      image: registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/kaniko-executor:v1.8.1
      args:
        - "--dockerfile=/workspace/code/tekton-demo/src/Dockerfile"
        - "--context=dir://code/tekton-demo/src"
        - "--destination=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v1.4"
        - "--cache=true"
        - "--cache-repo=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build-cache"
        - "--cache-dir=/workspace/cache"
      volumeMounts:
        - name: kaniko-secret
          mountPath: /kaniko/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/cache
          subPath: cache
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
说明

具体到客户的生产环境，比如是否有多语言（go、python、java等），是否有多应用多环境并行研发，会存在不同的解决方案。比如当涉及到主流技术栈为java，存在并行研发时，考虑到MVN过程的效率优化，可以使用极速NAS作为共享缓存加速构建，或者业务在更上层构建缓存，每次构建时下载MVN压缩包，采用存量缓存+增量下载的方式，需要结合业务场景进行分析。

ACS中通过buildkit构建镜像

Build With ACR Remote Cache

通过--export-cache将缓存推送至同仓库，通过--import-cache使用缓存。

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.6,push=true"
        - "--export-cache"
        - "type=inline"
        - "--import-cache"
        - "type=registry,ref=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.5"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

Build With Local Cache

apiVersion: v1
kind: Pod
metadata:
  name: buildkit
spec:
  restartPolicy: Never
  containers:
    - name: buildkit
      image: docker.anyhub.us.kg/moby/buildkit:v0.14.1-rootless
      command: ["buildctl-daemonless.sh", "--debug"]
      args:
        - "build"
        - "--progress=plain"
        - "--frontend=dockerfile.v0"
        - "--opt"
        - "filename=Dockerfile"
        - "--local"
        - "context=/workspace/code/tekton-demo/src"
        - "--local"
        - "dockerfile=/workspace/code/tekton-demo/src"
        - "--output"
        - "type=image,name=registry.cn-beijing.aliyuncs.com/demo-acs/demo-build:v2.12,push=true"
        - "--export-cache"
        - "type=local,mode=max,compression=gzip,dest=/workspace/buildkit-cache"
        - "--import-cache"
        - "type=local,src=/workspace/buildkit-cache"
      env:
        - name: BUILDKITD_FLAGS
          value: --oci-worker-no-process-sandbox
        - name: DOCKER_CONFIG
          value: /tmp/.docker/
      securityContext:
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - name: docker-secret
          mountPath: /tmp/.docker
        - name: workspace
          mountPath: /workspace/code
          subPath: code
        - name: workspace
          mountPath: /workspace/buildkit-cache
          subPath: buildkit-cache
        - name: buildkitd
          mountPath: /home/user/.local/share/buildkit
  volumes:
    - name: docker-secret
      secret:
        secretName: docker-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      persistentVolumeClaim:
        claimName: test-nas
    - name: buildkitd
      emptyDir: {}

缓存使用了NAS，在复用缓存的情况下构建耗时1分钟左右，对比kaniko，在复用层的场景下略慢。

ACS集成ACR进行构建

1. 创建镜像仓库并选择代码源

本文使用云效Codeup托管代码，并集成至ACR构建代码源中。

2. 设置构建规则

可在构建设置中开启代码变更自动构建镜像，并根据正则表达式匹配代码提交进行代码构建。

3. 构建缓存复用

ACR 默认开启镜像缓存并存至当前仓库，下次构建自动复用，构建效率预计提升 3 倍。

上一篇：在ACS上部署社区版Kubeflow下一篇：ACS PPU用户容器镜像开发实践