Alibaba Cloud DevOps pipelines provide a build cache to accelerate your builds. This topic describes how the pipeline cache works and its common use cases.
1. How the cache works
The Alibaba Cloud DevOps pipeline cache is available in two types:
File cache: Stores files such as JAR packages downloaded to
/root/.m2during the build process.Image build cache: Caches intermediate layers from the image build process to speed up subsequent image builds.
By default, the file cache is stored in the cloud. For each build, cache files are downloaded from the cloud and restored to the build environment. Data is transferred between the build environment and cloud storage over an internal network, ensuring high speeds.
When you use a private build cluster, you can choose to run jobs on a VM or in a container (supported on linux/amd64 machines only).
As shown in the preceding figure, if you select Default Environment, Alibaba Cloud DevOps starts a container on the build agent and runs the job in that container. If you select Default VM Environment, the cache settings do not take effect, and files generated during the build are persisted directly on the build agent.
For containerized jobs on a private build agent, the pipeline still defaults to storing the cache in the cloud. This can be slow because data must be transferred over the public internet, especially with limited outbound bandwidth. To address this, Alibaba Cloud DevOps also supports a local cache. This method stores the file cache in a specific directory on the build agent and remounts it for the next job run.
The file caches are job-level, shared across multiple runs of the same job within a pipeline. If you use a private build agent and want jobs from multiple pipelines to use the same cache, you can use the private build cluster local mount configuration. This feature mounts a host directory into the job execution environments across multiple pipelines.
2. File cache
2.1 Job-level cloud cache
On the pipeline editing page, click the Variables and Cache tab and select the Cache settings page.
The pipeline enables directory caching for common technology stacks by default, such as /root/.m2 used for Maven builds. During runtime, the pipeline packages the specified directories into a single .tgz file, uploads it to the OSS managed by Alibaba Cloud DevOps pipeline, and then downloads and unzips it to the same directory during the next build. This configuration applies to all jobs in the pipeline.
You can use the action buttons next to each cached directory to delete it or temporarily disable caching for it. If a directory is disabled or deleted, its files are no longer packaged and uploaded in subsequent builds.
Click Add Cache Directory to add more directories to the file cache.
Click Clear Cache to remove all cloud caches associated with the current pipeline. In the next build, no cache files are downloaded or extracted. However, the pipeline still packages and uploads files from the specified directories after the build completes, creating a new cache for future runs.
2.2 Job-level local cache
When you use a private build agent, run a job in a container, and opt to use a local cache, the file cache is stored on the build agent. The configuration is similar to the cloud cache, but requires selecting the local cache option, as shown in the following figure.
To find the local cache directory for a job, you need to get the following information from the build environment allocation log (runner version v0.1.1 or later is required):
The job's cache is located in the following paths, which vary based on the build agent's operating system.
Linux / macOS | |
Windows | |
If a pipeline is configured with a local cache but the job is configured to use a public build cluster, this configuration does not take effect. The cloud cache is used instead.
2.3 File cache scope
A pipeline can contain multiple job definitions. Caches are isolated between jobs and between pipelines, and can only be shared among different runs of the same job. As shown in the following figure, for this pipeline, Job 1 saves the file cache according to its configuration across all three runs and restores the cache for its next run. Job 2 and Job 3 each have their own separate caches.
2.4 Cloud file cache limitations
If the total size of the compressed tgz package exceeds 2 GB, the pipeline will not upload it.
2.5 Cloud file cache cleanup
In addition to manually clearing the cache from the pipeline configuration page, the system automatically clears caches that have not been updated in over 15 days.
2.6 Local file cache cleanup
Using a local cache on a private build cluster generates many log and cache files on the build agent over time, consuming significant disk space. Alibaba Cloud DevOps provides a cache cleanup tool to clear the following items on private build agents:
Stopped containers from containerized builds.
Dangling images (images without tags).
Intermediate images generated by custom environment builds.
Build job run logs older than 3 days.
Build job step logs older than 3 days.
Job cache directories that have not been updated for 15 days.
Run the following command to use the cleanup tool:
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /root/yunxiao:/root/yunxiao registry.cn-beijing.aliyuncs.com/build-steps/cache-cleaner:0.0.3-20230914183240To clear the cache on a regular schedule, we recommend adding the preceding command to a cron job.
2.7 Private build cluster local mount configuration
In addition to the job-isolated cache, Alibaba Cloud DevOps pipelines also support mounting a build agent directory to jobs across multiple pipelines. For example, when you use a containerized build for a Java project, you might want all build jobs to share the host's /root/.m2/ directory to avoid repeated dependency downloads. To achieve this, use the following configuration:
All containerized build jobs that run on the private build agent under the target pipeline will mount the /root/.m2 directory of the host to the /root/.m2 directory in the container.
Note: The job-level cache and the local mount cache can be configured simultaneously.
If a job uses a private build cluster, the private build cluster local mount setting takes effect even though the job-level configuration also has an entry for
/root/.m2. This is because the private build cluster local mount configuration also specifies the same directory. The remaining directories are still cached to the local directory as job-level local caches.If a job uses a public build cluster, the private build cluster cache configuration and private build cluster local mount configuration are ignored. The job-level cloud cache is used instead.
3. Image build cache
3.1 Image build job cache
The Alibaba Cloud DevOps pipeline image build job provides build caching capabilities:
When you use a public build cluster: Alibaba Cloud DevOps pipelines temporarily start a buildkitd sidecar container in the build environment. The image build container then connects to this buildkitd over HTTP and uses BuildKit to build the image. When the build is complete, the buildkitd sidecar container packages the entire BuildKit context and uploads it to the cloud. For the next build, the cache file is downloaded into the sidecar container, which then restores the BuildKit context to provide image caching.
The image build cache is also job-level, meaning caches are isolated between different jobs and pipelines.
When you use a private build cluster: The image build job directly mounts the host's dockerd to build the image. In this case, the image build cache is stored locally on the build agent.
By default, caching is enabled for image builds. If it is not needed, you can select Do not use cache to disable caching for the job. In the next build, the pipeline does not pull the cache from the cloud to restore the context.
3.2 Image build cache limitations
If the cache for an image build job exceeds 5 GB, the pipeline will not upload it.