Cache settings

Each job node in a Flow pipeline creates a new build environment that is destroyed after the job completes. To avoid downloading dependencies every time, you can use Flow caching to persist and share dependency files across runs.

For example, running npm install in a Node.js build job creates ./node_modules in the project and stores the cache in /root/.npm. Persisting /root/.npm speeds up dependency downloads in later runs.

Configure cache directories

In the pipeline editor, select to configure the Cache Directory.

• By default, Flow caches the following directories. You can enable, disable, or modify these default cache directories as needed.

  Management tool	Cache directory
  Maven	/root/.m2
  Gradle	/root/.gradle/caches
  NPM	/root/.npm
  Yarn	/root/.yarn
  go mod	/go/pkg/mod
  Other cache	/root/.cache

• You can also add custom cache directories. Custom directories must follow these rules:

  • Do not specify the root directory (/), /root, /root/workspace, or any subdirectories of these locations.

    To cache a file or directory within /root/workspace, copy it to another location under /root and set that location as the cache directory.

    The following example shows how to do this:

    In the Process Configuration tab, select the Run Command step, and enter the following script in the command editor ① to copy the file that is generated in the project directory to the cache directory /root/cpj:

    cd /root/cpj
    ls
    cd ${PROJECT_DIR}
    echo "aaa" > a.txt
    cp ${PROJECT_DIR}/a.txt /root/cpj

    The cache directory list contains the following paths, and the Enabled toggle is turned on for every entry:

    • /root/.m2

    • /root/.gradle/caches

    • /root/.npm

    • /root/.yarn

    • /go/pkg/mod

    • /root/.cache

    • /root/cpj (the custom cache directory that was added)

    For Cache configuration for a private build cluster, select Local cache (all dependency files are stored on the private build cluster).

    On the second pipeline run, the log of the Run Command task step shows that cd /root/cpj and ls list a.txt in the cache directory, which confirms that the custom cache takes effect and that the cache file survives between runs.

    [executionStep begins at 2022-04-18 10:59:44]
    [INFO] PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/sbin:/bin
    [10:59:44] [SUCCESS] Credential obtained successfully
    [10:59:44] [WARNING] No working directory specified. The default path /root/workspace/code is used
    [10:59:44] [INFO] Running the step
    [10:59:44] [INFO] PIPELINE_ID=1731311
    [10:59:44] [INFO] PIPELINE_NAME=custom-cache-pipeline
    [10:59:44] [INFO] BUILD_NUMBER=2
    [10:59:44] [INFO] EMPLOYEE_ID=5e71d6f503283833284f91c2
    [10:59:44] [INFO] WORK_SPACE=/root/workspace
    [10:59:44] [INFO] PROJECT_DIR=/root/workspace/code
    [10:59:44] [INFO] PLUGIN_DIR=/root/plugins
    [10:59:44] [INFO] BUILD_JOB_ID=53832096
    [10:59:44] [INFO] Preparing the execution context
    [10:59:44] [INFO] Running user commands
    [10:59:44] [User Command] + cd /root/cpj
    [10:59:44] [User Command] + ls
    [10:59:44] a.txt
    [10:59:44] [User Command] + cd /root/workspace/code
    [10:59:44] [User Command] + echo aaa
    [10:59:44] [User Command] + cp /root/workspace/code/a.txt /root/cpj
    [10:59:44] [SUCCESS] The step ran successfully

  • The path cannot contain ... For example, /root/abc/.. is invalid.

  • The path must be a valid directory path. For example, /root/%\&dfaf is invalid.

Using cache in build jobs

After you configure the cache directories, add parameters to your build commands to use the cache during pipeline runs.

• For npm builds, use the following command to install dependencies, prioritizing packages from the local cache:

  npm install --prefer-offline --no-audit

• For Yarn builds, use the following command to install dependencies, prioritizing packages from the local cache:

  yarn install --prefer-offline

Caching for a private build cluster

A private build cluster offers two caching methods:

• Local cache: Dependency files are stored on the private build cluster. Using the local cache avoids network-related build slowdowns.

• Alibaba Cloud DevOps-managed cache: Each build downloads cached files from the cloud. This works well for a cluster with multiple build agents, because they can reuse cached dependencies.

Cache cleanup on build agents

Over time, a private build cluster accumulates cache files. To prevent disk space issues that can cause pipeline failures, clean up the cache periodically.

The Alibaba Cloud DevOps cache cleanup tool removes the following items from build agents:

• Base environment cleanup:

  • Stopped containers.

  • Dangling images.

  • Intermediate images generated during custom environment builds.

• Cleanup of the /root/yunxiao/*/runner/ directory:

  • Build job run logs that are older than 3 days.

  • Build job step logs that are older than 3 days.

  • Build job cache directories that have not been updated in 15 days.

Run the following command to use the cleanup tool:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /root/yunxiao:/root/yunxiao registry.cn-beijing.aliyuncs.com/build-steps/cache-cleaner:0.0.3-20230914183240

You can add this command to a cronjob to schedule regular cache cleanup.