Each time a pipeline task runs, a new instance is scheduled. This instance must pull the build code and its required dependencies. If pulling dependencies takes too long, the pipeline task can experience slow builds or timeout failures. The Serverless-cd community provides a cache plugin based on Object Storage Service (OSS) to resolve issues such as slow dependency pulling and build timeouts. This topic describes how to accelerate builds using the OSS cache plugin.
Use the cache plugin
The Serverless-cd community provides a cache plugin that is optimized for the features of Serverless App Center. To use the cache plugin, you must specify an OSS Bucket to store the cache. The cache plugin then retrieves, reads, and updates the cache content in the specified or default OSS Bucket based on the specified key.
After the plugin runs, if the cache key is a hit, the file content that corresponds to the cache key is downloaded to a specified local path.
After the engine runs, the file content from the specified local path is uploaded to the OSS Bucket.
Define the OSS cache configuration in the execution context
You can define the OSS cache configuration in the execution context of a pipeline, pipeline template, or task template. The following is an example.
---
# Submit the pipeline for execution.
kind: Pipeline
name: "p-<% .git.shortCommitId %>-<% .currentTimestampMs %>"
# Description.
description: cached pipeline
spec:
context:
data:
# Cache configuration for the application center. If not provided, the engine automatically performs initialization in the region where the task node is scheduled.
# The default bucket name is: serverless-cd-cache-${region-id}-<% .accountId %>.
cacheConfig:
# Use OSS for caching. If not specified, the platform performs automatic initialization.
# Let the platform perform automatic initialization. This is because the worker node may be scheduled to a region different from the specified OSS region.
# For GitHub repositories, the worker node and the corresponding OSS Bucket are in the Singapore (ap-southeast-1) region.
# For GitLab, Codeup, and Gitee repositories, the worker node and the corresponding OSS Bucket are in the Hangzhou (cn-hangzhou) region.
oss:
# The name of the bucket used for caching. If the bucket does not exist, the engine does not perform automatic initialization.
bucketName: serverless-cd-cache-ap-southeast-1-<% .accountId %>
# The region where the bucket is located. Do not pull caches across regions.
regionId: ap-southeast-1
templateName: mytemplate-<% .git.branch %>
---Use the cache plugin in steps
You can use the cache plugin in the `steps` of the execution context within a pipeline template or task template. Note that the credentials information is optional. The plugin automatically uses the role permissions granted to the application center. The following is an example.
---
kind: PipelineTemplate
name: mytemplate-<% .git.branch %>
description: cached pipelinetemplate
spec:
context:
data:
envName: test
deployFile: s.yaml
tasks:
# Pre-check
- name: pre-check
context:
data:
displayName: "Pre-check"
enable: true
steps:
- plugin: "@serverless-cd/checkout"
- plugin: "@serverless-cd/s-setup"
- run: s plan --local -o json
taskTemplate: serverless-runner-task
runAfters: []
- name: pre-check-approval
context:
data:
dingdingNotification:
enable: false
enable: true
taskTemplate: need-approval
runAfters:
- name: pre-check
# Build and deploy
- name: build-and-deploy
context:
data:
enable: true
steps:
- plugin: "@serverless-cd/checkout"
- plugin: "@serverless-cd/s-setup"
- plugin: "@serverless-cd/cache"
inputs:
# Define a unique identifier for the cache. You can associate it with information such as the application or Git version.
# You can also use "cache-${{hashFile('./package.json')}}".
# Calculate the unique identifier for the cache from the hash value of a local file.
# The corresponding file path for the cache in the bucket is cache-home/${key}. For example, in this case,
# the cache path in the bucket is cache-home/cache-<% .appName %>.
key: "cache-<% .appName %>"
# Configure the cache path based on the requirements of the current build tool.
# For example, for Node.js, you can use the default ~/.npm as the cache path for the pipeline application.
path: ~/.npm
# Execute s-deploy.
- run: |
set -e
s deploy -t --use-local --assume-yes --skip-push
echo "Deployment by Serverless Devs succeeded."
taskTemplate: serverless-runner-task
runAfters:
- name: pre-check-approval
---Cache paths
The required cache path varies depending on the build tool. Refer to the following table or the documentation for your specific tool to determine the correct path.
Management tool | Cache path |
Maven | ~/.m2 |
Gradle | ~/.gradle |
NPM | ~/.npm |
Yarn | ~/.yarn |
gomod | $GOPATH/pkg/mod |
Other caches (including pip3) | ~/.cache |
Create a local cache
If the build dependency source service has poor performance, or if you cannot pull dependencies from the cloud because of security issues, you can use the ossutil tool to upload the cache content from your local development machine to OSS and then use it in the pipeline.
Alternatively, you can log on to the Object Storage Service console and upload the local cache, such as ~/.npm, to the corresponding path in OSS.
The following is an example.
# Use ossutil to create a cache for cloud builds. This command copies the local npm cache to Object Storage Service (OSS).
# The URL for an automatically created bucket is serverless-cd-${region-id}-cache-${uid}.oss-${region-id}.aliyuncs.com
# For GitHub repositories, the worker node and the corresponding OSS Bucket should be deployed in the Singapore (ap-southeast-1) region.
# For GitLab, Codeup, and Gitee repositories, the worker node and the corresponding OSS Bucket should be deployed in the Hangzhou (cn-hangzhou) region.
./ossutil64 cp -r ~/.npm oss://${bucket_url}/cache-home/${cache_key}