Pipeline runner

1. Install a runner

The runner acts as a channel for pipeline builds or deployments. The machine where a runner is installed must belong to a build cluster or a host group. Therefore, you must first create a private build cluster or host group.

• Go to the Alibaba Cloud DevOps homepage, navigate to global settings > build cluster management, and click New build cluster. This is used for pipeline build environments.

• Go to the Alibaba Cloud DevOps homepage, navigate to global settings > host group management, and click New host group. This is used for deploying pipeline tasks to hosts.

Adding a host to a private build cluster is similar to adding one to a host group. The following sections use a build cluster as an example to demonstrate runner installation.

1.1 Network connectivity requirements

If you use the Manual Installation method, the host must access the following public network services:

• https://devops-build-new.aliyuncs.com

• https://gray-devops-build.aliyuncs.com

If you use Alibaba Cloud ECS in one of the following regions: China (Beijing), China (Hangzhou), China (Zhangjiakou), China (Shenzhen), China (Shanghai), China (Qingdao), China (Hohhot), Singapore, China (Heyuan), China (Guangzhou), China (Ulanqab), or China (Chengdu), and you install the runner using the Alibaba Cloud ECS method, the runner automatically connects to the pipeline server through an internal endpoint in the corresponding region. The ECS host itself does not need to access the public network.

Note: When adding a host to a build cluster, the host must have public network access regardless of whether you use the Alibaba Cloud ECS or Manual Installation method. This is because the runner may need to pull images from public registries during the build process.

1.2 Host requirements

You can install the pipeline runner on Linux, Windows, and macOS hosts. The requirements are as follows:

• The Linux operating system must support one of the following services: Systemd, Upstart, or System V.

• The Windows operating system must support Windows service.

Both amd64 and arm64 chip architectures are supported for these operating systems.

To install the runner, use the root account on Linux or the Administrator account on Windows. On macOS, you can use any user.

1.3 Install a runner on a host

Go to the Alibaba Cloud DevOps homepage, navigate to global settings > build cluster management, find and open the details page for the target build cluster, and click Add node to add a new build host.

Select your operating system and then choose an installation method:

• Manual Installation. Copy the installation command and run it on your host.

• Alibaba Cloud ECS.

  • If required, follow the prompts to create an ECS service connection for your Alibaba Cloud account.

  • Select the target ECS host. Alibaba Cloud DevOps sends the installation command using Cloud Assistant. After the installation is complete, you can view the installation log on the page.

After the host is added, you can view its information on the Host list tab of the build cluster details page. The information includes Instance ID, IP, Hostname, Region, Operating system, Runner status, and Installation progress. You can also click the Log link next to the installation progress to view the detailed installation log.

For Linux and Windows hosts, you can install the runner using either the Manual Installation or Alibaba Cloud ECS method. For macOS hosts, only the manual installation method is supported.

On all supported operating systems, the runner is installed as a service.

• On Linux, the runner supports the following service types: Systemd, Upstart, and System V. If the host supports multiple service types, the runner uses the first available type in this order. The runner installation log indicates which service type was selected:

[INFO] install runner system service
/usr/local/share/yunxiao-runner/v0.1.1/runner install --tenant=be-xxx
INFO[2024-02-28T11:09:17+08:00] service type linux-systemd
INFO[2024-02-28T11:09:17+08:00] Aliyun yunxiao runner-v0.1.1-be-xxx service, linux-systemd installed
[INFO] start runner service
/usr/local/share/yunxiao-runner/v0.1.1/runner start --tenant=be-xxx
INFO[2024-02-28T11:09:17+08:00] service type linux-systemd
INFO[2024-02-28T11:09:17+08:00] Aliyun yunxiao runner-v0.1.1-be-xxx service started
INFO[2024-02-28T11:09:17+08:00] Aliyun yunxiao runner-v0.1.1-be-xxx service status running

• On Windows, the runner is started by using the Windows service.

• On macOS, the runner is started by using Launchd.

1.4 Runner service naming

Alibaba Cloud DevOps is a multi-tenant service. You can install services for different organizations on the same host.

Each Alibaba Cloud DevOps organization has a unique namespace, which is a 24-character random alphanumeric string, for example, aaaaaaaaaaaaaaaaaaaaaaaa. If the installed runner version is v0.1.0, the service name is runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa. If another organization's namespace is bbbbbbbbbbbbbbbbbbbbbbbb, you can have both services installed on the same machine:

runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
runner-v0.1.1-be-bbbbbbbbbbbbbbbbbbbbbbbb

Although you can install multiple runner versions for the same organization on one host, do not do so by running the installation command multiple times.

In fact, after you install a runner service, it automatically checks the server for newer versions and performs an auto-upgrade. This ensures that only one runner service exists for each organization. Therefore, you do not need to run the runner installation command multiple times on the same host.

2. Configure a runner

After a successful installation, you can view the runner's configuration on the host. You typically do not need to modify the default configuration file. The method for locating the configuration file and applying changes varies by operating system.

2.1 Configurable options

The runner supports the following configuration settings:

• scanInterval: The interval in seconds at which the runner sends a heartbeat to the pipeline server. You typically do not need to change this value.

• concurrency: The maximum number of concurrent tasks allowed.

• autoUpgrade: Specifies whether to enable auto-upgrade. If enabled, the runner attempts to upgrade itself automatically. For more information, see the Auto-upgrade section.

• upgradeInteval: The interval in seconds for checking for runner version updates.

2.2 Modify the configuration file

Assume the following two services exist on the machine:

runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
runner-v0.1.1-be-bbbbbbbbbbbbbbbbbbbbbbbb

Linux

Use the command for your OS to view the services:

#Systemd
$ systemctl | grep "runner-v" | awk '{print $1}'
runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa.service
runner-v0.1.1-be-bbbbbbbbbbbbbbbbbbbbbbbb.service

#Upstart
$ initctl list | grep 'runner-v' | awk '{print $1}'
runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
runner-v0.1.1-be-bbbbbbbbbbbbbbbbbbbbbbbb

#System V
$ service --status-all 2>&1 | grep 'runner-v' | awk '{print $4}'
runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
runner-v0.1.1-be-bbbbbbbbbbbbbbbbbbbbbbbb

Select the service you want to inspect, and then parse its service name and organization namespace:

$ SERVICE_NAME=runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa.service
# If you use System V or Upstart, set SERVICE_NAME=runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
$ NAMESPACE=`echo $SERVICE_NAME | awk -F '-' '{print $3"-"$4}' | awk -F "." '{print $1}'`
$ echo $NAMESPACE
be-aaaaaaaaaaaaaaaaaaaaaaaa

View the service's configuration file:

$ vi /root/yunxiao/${NAMESPACE}/runner/config/config.yml

Windows

# View the services.
PS C:\> Get-Service  | Select-Object -Property Name | findstr runner
runner-be-aaaaaaaaaaaaaaaaaaaaaaaa 
runner-be-bbbbbbbbbbbbbbbbbbbbbbbb

# Set the SERVICE_NAME and NAMESPACE variables.
PS C:\> $Env:SERVICE_NAME = "runner-be-bbbbbbbbbbbbbbbbbbbbbbbb"
PS C:\> $Env:NAMESPACE = "be-aaaaaaaaaaaaaaaaaaaaaaaa"

# View the service's configuration file.
PS C:\> cat C:\yunxiao\data\$Env:NAMESPACE\runner\config\config.yml

macOS

# View the services.
$ launchctl list | grep runner-v | awk '{print $3}'
runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
runner-v0.1.1-be-bbbbbbbbbbbbbbbbbbbbbbbb

# Set the SERVICE_NAME and NAMESPACE variables.
$ SERVICE_NAME=runner-v0.1.1-be-aaaaaaaaaaaaaaaaaaaaaaaa
$ NAMESPACE=`echo $SERVICE_NAME | awk -F '-' '{print $3"-"$4}'`
$ echo $NAMESPACE
be-bbbbbbbbbbbbbbbbbbbbbbbb

# View the service's configuration file.
cat ~/.yunxiao-runner/$NAMESPACE/config/config.yml

After you modify the configuration file, you must restart the runner for the changes to take effect.

3. Maintain a runner

3.1 Manage the runner service

Linux

Set the SERVICE_NAME variable as described in the "Modify the configuration file" section for Linux.

• Systemd

# Start
$ systemctl start $SERVICE_NAME
# Stop
$ systemctl stop $SERVICE_NAME
# Restart
$ systemctl restart $SERVICE_NAME
# Enable
$ systemctl enable $SERVICE_NAME
# Disable
$ systemctl disable $SERVICE_NAME

• Upstart

# Start
$ start $SERVICE_NAME
# Stop
$ stop $SERVICE_NAME
# Restart
$ restart $SERVICE_NAME
# Disable
# Not applicable
# Enable
# Not applicable

• System V

# Start
$ service $SERVICE_NAME start
# Stop
$ service $SERVICE_NAME stop
# Restart
$ service $SERVICE_NAME stop && service $SERVICE_NAME start
# Disable
$ update-rc.d $SERVICE_NAME defaults
# Enable
$ update-rc.d $SERVICE_NAME remove

Windows

Perform these operations in the Windows Services application:

Find the Aliyun yunxiao runner-be-{namespace} service in the list. Right-click the service and select the desired action from the context menu, such as Start, Stop, Pause, Resume, or Restart.

macOS

Set the SERVICE_NAME variable as described in the "Modify the configuration file" section for macOS.

# Start
$ launchctl start $SERVICE_NAME
# Stop
# This command restarts the service. To stop it permanently, you must uninstall it.
# Restart
$ launchctl stop $SERVICE_NAME # The service restarts immediately after being stopped.
# Disable
# Not applicable
# Enable
# Not applicable

3.2 View runner logs

Linux

Set the NAMESPACE variable as described in the "Modify the configuration file" section for Linux.

$ tail -f /root/yunxiao/$NAMESPACE/runner/__flow_logs/runner.log 

Windows

PS C:\> Get-Service  | Select-Object -Property Name | findstr runner
runner-be-aaaaaaaaaaaaaaaaaaaaaaaa
PS C:\> $Env:NAMESPACE = "be-aaaaaaaaaaaaaaaaaaaaaaaa"
PS C:\> Get-Content -Tail 10 C:\yunxiao\data\$Env:NAMESPACE\runner\__flow_logs\runner.log -Wait 

macOS

Set the NAMESPACE variable as described in the "Modify the configuration file" section for macOS.

$ tail -f ~/.yunxiao-runner/$NAMESPACE/__flow_logs/runner.log

3.3 Clean up temporary files

The runner generates temporary files during its operation:

• The runner.log file.

• Temporary storage for build logs.

• Temporary storage for the local cache.

• For a containerized build, the corresponding image will be pulled.

• Intermediate images generated by image build tasks.

The runner has the following automatic cleanup rules for some temporary files:

• The runner.log file has a maximum size of 500 MB. Up to 14 archived log files are kept, and all log files are retained for a maximum of 7 days. Older logs are automatically deleted.

• Temporary storage for build logs is retained for 3 days and then automatically deleted.

Other items require manual cleanup by running the command shown below. Note that this command is for Linux systems; for Windows or macOS, you must adjust the volume paths.

• Cleans up build cache entries that have not been updated in 15 days.

• For containerized builds, removes stopped containers.

• Removes dangling images.

• Removes intermediate images generated by custom environment builds.

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /root/yunxiao:/root/yunxiao registry.cn-beijing.aliyuncs.com/build-steps/cache-cleaner:0.0.3-20230914183240

3.4 Auto-upgrade

The version of a runner is in the format v<major>.<minor>.<patch>. By default, a runner periodically queries the pipeline service for available version updates. The runner performs an auto-upgrade if the minor version number of the latest version on the server is different from that of the local version.

3.5 Manual upgrade

To upgrade the patch version of a runner, you must uninstall the current version before reinstalling the new one.

4. Uninstall a runner

Linux

Set the SERVICE_NAME and NAMESPACE variables as described in the "Modify the configuration file" section for Linux.

• Systemd

$ systemctl stop $SERVICE_NAME
$ systemctl disable $SERVICE_NAME
$ rm /etc/systemd/system/$SERVICE_NAME
$ rm -rf /root/yunxiao/$NAMESPACE
# To remove all runner services for all tenants on this host, run the following command after you complete the preceding steps for each service.
$ rm -rf /root/yunxiao

• Upstart

$ initctl stop $SERVICE_NAME
$ mv /etc/init/$SERVICE_NAME.conf /etc/init/$SERVICE_NAME.conf.disabled
$ rm -rf /root/yunxiao/$NAMESPACE
# To remove all runner services for all tenants on this host, run the following command after you complete the preceding steps for each service.
$ rm -rf /root/yunxiao

• System V

$ service $SERVICE_NAME stop
$ for i in /etc/rc*.d/S*$SERVICE_NAME; do rm -f $i; done
$ rm -rf /etc/init.d/$SERVICE_NAME
$ rm -rf /root/yunxiao/$NAMESPACE
# To remove all runner services for all tenants on this host, run the following command after you complete the preceding steps for each service.
$ rm -rf /root/yunxiao

Windows

Refer to the preceding instructions for viewing services on Windows to set SERVICE_NAME and NAMESPACE, and the instructions for viewing logs on Windows to configure NAMESPACE_SHORT.

• First, stop the runner service in the Services management application.

• Then, run the following commands to delete the corresponding directories.

PS C:\> C:\yunxiao\d\ohce\runner
PS C:\> Remove-Item -Path "C:\yunxiao\data\$Env:NAMESPACE\runner" -Recurse -Force
# To remove all runner services for all tenants on this host, run the following command after you complete the preceding steps for each service.
PS C:\> Remove-Item -Path "C:\yunxiao" -Recurse -Force

macOS

Set the SERVICE_NAME and NAMESPACE variables as described in the "Modify the configuration file" section for macOS.

$ sudo launchctl bootout gui/`id -u`/$SERVICE_NAME
$ rm -f ~/Library/LaunchAgents/$SERVICE_NAME.plist
$ rm -rf ~/.yunxiao-runner/$NAMESPACE
# To remove all runner services for all tenants on this host, run the following command after you complete the preceding steps for each service.
$ rm -rf ~/.yunxiao-runner

5. Other limitations and known issues

Identical config.yml files on two hosts

Do not copy a config.yml file from one host to another and then start the runner service. This causes both runner services to attempt to pull tasks, but the target host might not exist in the intended build cluster or host group, which can cause unexpected build behavior.