Pipeline cache
Alibaba Cloud DevOps pipelines provide a build cache to accelerate your builds. This topic describes how the pipeline cache works and its common use cases.
1. How the cache works
The Alibaba Cloud DevOps pipeline cache is available in two types:
File cache: Stores files such as JAR packages downloaded to
/root/.m2during the build process.Image build cache: Caches intermediate layers from the image build process to speed up subsequent image builds.
By default, the file cache is stored in the cloud. For each build, cache files are downloaded from the cloud and restored to the build environment. Data is transferred between the build environment and cloud storage over an internal network, ensuring high speeds.
When you use a private build cluster, you can choose to run jobs on a VM or in a container (supported on linux/amd64 machines only). In the job editing panel of the pipeline, you can set Build Cluster (for example, a test build cluster) and Specified Build Node (for example, Linux,amd64), and then select the Build Environment.
If you select Default Environment, Alibaba Cloud DevOps starts a container on the build agent and runs the job in that container. If you select Default VM Environment, the cache settings do not take effect, and files generated during the build are persisted directly on the build agent.
For containerized jobs on a private build agent, the pipeline still defaults to storing the cache in the cloud. This can be slow because data must be transferred over the public internet, especially with limited outbound bandwidth. To address this, Alibaba Cloud DevOps also supports a local cache. This method stores the file cache in a specific directory on the build agent and remounts it for the next job run.
The file caches are job-level, shared across multiple runs of the same job within a pipeline. If you use a private build agent and want jobs from multiple pipelines to use the same cache, you can use the private build cluster local mount configuration. This feature mounts a host directory into the job execution environments across multiple pipelines.
2. File cache
2.1 Job-level cloud cache
On the pipeline editing page, click the Variables and Cache tab and select the Cache settings page. The Cache page provides the following cache directories by default, and you can turn each directory on or off with its switch:
/root/.m2/root/.gradle/caches/root/.npm/root/.yarn/go/pkg/mod/root/.cache
In the Private build cluster cache configuration section, you can select Local Cache (All dependency files are stored on a private build cluster.) or Alibaba Cloud DevOps managed cache (cache files are uploaded to Alibaba Cloud DevOps management storage).
The pipeline enables directory caching for common technology stacks by default, such as /root/.m2 used for Maven builds. During runtime, the pipeline packages the specified directories into a single .tgz file, uploads it to the OSS managed by Alibaba Cloud DevOps pipeline, and then downloads and unzips it to the same directory during the next build. This configuration applies to all jobs in the pipeline. The following log shows the cache download step of a Java build pipeline, in which the cache file is fetched from the cloud and restored to the build environment:
[18:10:37] [INFO] WORK_SPACE=/root/workspace
[18:10:37] [INFO] PROJECT_DIR=/root/workspace/repo1
[18:10:37] [INFO] PLUGIN_DIR=/root/plugins
[18:10:37] [INFO] BUILD_JOB_ID=194842122
[18:10:37] [INFO] LOG_LEVEL=DEBUG
[18:10:37] [DEBUG] ARCHIVE_PATH=/root/.m2 /root/.gradle/caches /root/.npm /root/.yarn /go/pkg/mod /root/.cache
[18:10:37] [USER]CACHE_NAME=global-cache
[18:10:37] [USER]ARCHIVE_PATH=/root/.m2 /root/.gradle/caches /root/.npm /root/.yarn /go/pkg/mod /root/.cache
[18:10:37] [USER]COMMAND=RUN
[18:10:37] [USER]FILE_ID=aone2,xxx/caches/2962805/global-cache-stage1-stage1_job1.tgz
[18:10:37] [USER]ENDPOINT=oss-cn-beijing-internal.aliyuncs.com
[18:10:37] [USER]BUCKET_NAME=execution-component-rdc
[18:10:37]
[18:10:37]
[18:10:37] Fetching cache file aone2/xxx/caches,xxx/global-cache-stage1-stage1_job1.tgz -> /root/workspace/packages/global-cache-stage1-stage1_job1.tgz
[18:10:37] log file is /root/workspace/repo1/ossutil.log
[18:10:37]
Succeed: Total num: 1, size: 1,671,476. OK num: 1(download 1 objects).
[18:10:37]
[18:10:37] average speed 15768000(byte/s)
[18:10:37]
[18:10:37] 0.106430(s) elapsed
[18:10:37] [WARN] Cache file downloaded successfully
[18:10:37] [INFO] /root/workspace/packages/global-cache-stage1-stage1_job1.tgz SIZE 1671476
[18:10:37] Restoring cache directories
[18:10:37] tar: Removing leading `/' from member names
[18:10:37] store oss runtime log: aone2,xxx/caches/2962805/194842122-RUN-ossutil.log
[18:10:37] [SUCCESS] Step run successfullyThe following log shows the cache upload step of a Java build pipeline, in which all cache directories are uploaded successfully:
[18:10:46] [USER]BUCKET_NAME=execution-component-rdc
[18:10:46]
[18:10:46] /root/.m2
[18:10:46] Preparing cache file /root/.m2 -> aone2/xxx/caches/xxx global-cache-stage1-stage1_job1.tgz
[18:10:46] /root/.gradle/caches
[18:10:46] Preparing cache file /root/.gradle/caches -> aone2/xxx/caches/xxx/global-cache-stage1-stage1_job1.tgz
[18:10:46] /root/.npm
[18:10:46] Preparing cache file /root/.npm -> aone2,xxx,caches/xxx global-cache-stage1-stage1_job1.tgz
[18:10:46] /root/.yarn
[18:10:46] Preparing cache file /root/.yarn -> aone2/xxx/caches/xxx /global-cache-stage1-stage1_job1.tgz
[18:10:46] /go/pkg/mod
[18:10:46] Preparing cache file /go/pkg/mod -> aone2,xxx caches/xxx /global-cache-stage1-stage1_job1.tgz
[18:10:46] /root/.cache
[18:10:46] Preparing cache file /root/.cache -> aone2/xxx/caches/xxx/global-cache-stage1-stage1_job1.tgz
[18:10:46] tar: /root/.cache/git/credential/socket: socket ignored
[18:10:46] [INFO] /root/workspace/packages/global-cache-stage1-stage1_job1.tgz SIZE 1671048
[18:10:46] [INFO] try to upload
[18:10:46] log file is /root/workspace/repo1/ossutil.log
[18:10:47]
Succeed: Total num: 1The log ends with [WARN] Cache file uploaded successfully and [SUCCESS] Step run successfully, which indicate that the cache upload step is complete.
You can use the action buttons next to each cached directory to delete it or temporarily disable caching for it. If a directory is disabled or deleted, its files are no longer packaged and uploaded in subsequent builds.
Click Add Cache Directory to add more directories to the file cache.
Click Clear Cache to remove all cloud caches associated with the current pipeline. In the next build, no cache files are downloaded or extracted. However, the pipeline still packages and uploads files from the specified directories after the build completes, creating a new cache for future runs.
2.2 Job-level local cache
When you use a private build agent, run a job in a container, and opt to use a local cache, the file cache is stored on the build agent. The configuration is similar to the cloud cache, except that you must select the local cache option. On the pipeline editing page, click the Variables and Cache tab, and then select Cache from the left-side menu. In the Private build cluster cache configuration section, select Local Cache (All dependency files are stored on a private build cluster.).
To find the local cache directory for a job, get the following information from the build environment allocation log (runner version v0.1.1 or later is required):
[executionStep begins at 2024-03-19 17:55:17]
[Runner Group Info]:
>> RunnerGroupId : xxx
>> RunnerGroupName : xxx
[Runner Info]:
>> ID : xxx
>> Tenant : be-xxx
>> Server : https://pre-devops-build1.aliyuncs.com
>> Version : v0.1.1
>> CommitId : 71fa3d2
>> BuildTime : 20240227175443
>> Workspace : /root/yunxiao/be-xxx/runner
>> InstanceId : xxx
[Build Machine Info]:
>> OS : linux
>> Arch : amd64
>> Hostname : xxx
>> Memory : 2GB physical, 2GB usable
>> CPU : 1 package(s), 1 core(s), 1 hardware thread(s)
[Build Job Info]:
>> ID : xxx
>> ParentID : xxx
>> Namespace : be-xxx
>> Executor : DockerThe job's cache is located in the following paths, which vary based on the build agent's operating system.
Linux / macOS | |
Windows | |
If a pipeline is configured with a local cache but the job is configured to use a public build cluster, this configuration does not take effect. The cloud cache is used instead.
2.3 File cache scope
A pipeline can contain multiple job definitions. Caches are isolated between jobs and between pipelines, and can be shared only among different runs of the same job.
For example, on the Run History tab of a pipeline, three runs (#3, #2, and #1) are all in the Successful state, and each run contains the three stage nodes Job 1, Job 2, and Job 3, with durations of 19 seconds, 20 seconds, and 20 seconds. Job 1 is the job node that has the file cache configured. It saves the file cache according to its configuration across all three runs and restores the cache for its next run, which shows that the cache takes effect across multiple runs of the same pipeline. Job 2 and Job 3 each have their own separate caches.
2.4 Cloud file cache limitations
If the total size of the compressed tgz package exceeds 2 GB, the pipeline will not upload it.
2.5 Cloud file cache cleanup
In addition to manually clearing the cache from the pipeline configuration page, the system automatically clears caches that have not been updated in over 15 days.
2.6 Local file cache cleanup
Using a local cache on a private build cluster generates many log and cache files on the build agent over time, consuming significant disk space. Alibaba Cloud DevOps provides a cache cleanup tool to clear the following items on private build agents:
Stopped containers from containerized builds.
Dangling images (images without tags).
Intermediate images generated by custom environment builds.
Build job run logs older than 3 days.
Build job step logs older than 3 days.
Job cache directories that have not been updated for 15 days.
Run the following command to use the cleanup tool:
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /root/yunxiao:/root/yunxiao registry.cn-beijing.aliyuncs.com/build-steps/cache-cleaner:0.0.3-20230914183240To clear the cache on a regular schedule, we recommend adding the preceding command to a cron job.
2.7 Private build cluster local mount configuration
In addition to the job-isolated cache, Alibaba Cloud DevOps pipelines also support mounting a build agent directory to jobs across multiple pipelines. For example, when you use a containerized build for a Java project, you might want all build jobs to share the host's /root/.m2/ directory to avoid repeated dependency downloads. To achieve this, use the following configuration.
On the pipeline editing page, click the Variables and Cache tab, and then select Cache on the left. The Private build cluster local mount configuration section at the bottom of the page displays the local mount configuration card. Click Modify configuration to edit the mount mapping, and use the switch to enable or disable it. The configuration items are Host Directory (for example, /root/.m2) and Container Build Environment Directory (for example, /root/.m2). This mapping mounts the dependency cache directory on the host to the corresponding directory in the container build environment, so that all dependency files are stored on the private build cluster.
All containerized build jobs that run on the private build agent under the target pipeline will mount the /root/.m2 directory of the host to the /root/.m2 directory in the container.
Note: The job-level cache and the local mount cache can be configured simultaneously.
If a job uses a private build cluster, the private build cluster local mount setting takes effect even though the job-level configuration also has an entry for
/root/.m2. This is because the private build cluster local mount configuration also specifies the same directory. The remaining directories are still cached to the local directory as job-level local caches.If a job uses a public build cluster, the private build cluster cache configuration and private build cluster local mount configuration are ignored. The job-level cloud cache is used instead.
3. Image build cache
3.1 Image build job cache
The Alibaba Cloud DevOps pipeline image build job provides build caching capabilities:
When you use a public build cluster: Alibaba Cloud DevOps pipelines temporarily start a buildkitd sidecar container in the build environment. The image build container then connects to this buildkitd over HTTP and uses BuildKit to build the image. When the build is complete, the buildkitd sidecar container packages the entire BuildKit context and uploads it to the cloud. For the next build, the cache file is downloaded into the sidecar container, which then restores the BuildKit context to provide image caching.
The image build cache is also job-level, meaning caches are isolated between different jobs and pipelines.
When you use a private build cluster: The image build job directly mounts the host's dockerd to build the image. In this case, the image build cache is stored locally on the build agent.
By default, caching is enabled for image builds. If it is not needed, select the No Cache check box at the bottom of the image build job editing panel to disable caching for the job. In the next build, the pipeline does not pull the cache from the cloud to restore the context. When this check box is selected, docker build uses the --no-cache=true parameter, which skips all local cache layers and rebuilds the image from scratch.
3.2 Image build cache limitations
If the cache for an image build job exceeds 5 GB, the pipeline will not upload it.