Troubleshoot host deployments

Self-hosted instance is offline

Verify that the agent on your server is online. If it is offline, restart it. The basic agent management commands are:

• Check status:

  /home/staragent/bin/staragentctl status

• Restart:

  /home/staragent/bin/staragentctl restart

• Uninstall:

  /home/staragent/bin/staragentctl stop
  rm -rf /home/staragent
  rm /usr/sbin/staragent_sn

Host deployment fails without error logs

If you are using a non-ECS machine, check if the self-hosted instance was created from an image. If so, uninstall the agent, add the self-hosted instance again, and retry the deployment.

Cannot add a non-ECS self-hosted instance to a host group

For non-ECS self-hosted instances, we recommend using the hybrid cloud hosting mode. For more information, see https://help.aliyun.com/document_detail/201140.html. Click New host group. In the Self-hosted Instance · Add Host dialog box, set Add Method to Hybrid Cloud Hosting, to register a non-Alibaba Cloud machine as an Alibaba Cloud-managed instance. Then, select a service authorization and region. You can then view and add your non-ECS self-hosted instance from the Available Hosts list.

Errors during cluster creation or runner installation

The error message is Failed to install runner for instance i-2ze99uh20nyrykgdobvb! (3000005).

In Alibaba Cloud DevOps Flow, you can create a host group or build cluster by either selecting an ECS instance or installing an agent on a self-hosted instance. However, you cannot use both methods to add the same machine to the same Alibaba Cloud DevOps organization.

For example, if you have already added Host A as an ECS instance, you cannot add it again as a self-hosted instance.

• To switch the registration method, you must first uninstall the Flow Runner from the host. Use the following commands to uninstall the Flow Runner:

  # Stop the specified systemd service.
  systemctl stop runner-{version}-{tenant-name}.service
  # Delete the systemd configuration file for the service.
  rm -rf /etc/systemd/system/runner-{version}-{tenant-name}.service
  # Delete the runner configuration directory for the tenant.
  rm -rf /root/yunxiao/{tenant-name}/runner/config

• The deployment command runs successfully when executed manually on the host but fails in the Flow pipeline.

  • Add the relevant environment variables for the command. For example: /source /root/.bash_profile; source /etc/profile.

  • Use absolute paths in deployment scripts (for example, /home/admin/app/deploy.sh) instead of relative paths.

Flow Runner configuration conflicts from ECS image cloning

During a host deployment, if you clone Machine A to create Machine B, Machine B reuses the Flow Runner configuration from Machine A. This can cause channel conflicts, leading to the following issues:

• Incorrect deployment target: Tasks intended for Machine B are incorrectly dispatched to Machine A.

• Root cause: After the image was cloned, the unique identifier of the Flow Runner (such as its registration information) was not reset. As a result, the scheduling system cannot distinguish between Machine A and Machine B.

• Key impact: This conflict disrupts automated workflows and can lead to environment contamination or service overlap.

• Solution:

  Note

  If you clone a machine with a configured Flow Runner, you must reinstall the Flow Runner on the new machine (Machine B) to ensure that Machine A and Machine B have unique channel IDs.

  # Uninstall the Flow Runner on Machine B
  # Check the Flow Runner service name. The name is typically in the format runner-{version}-{tenant-name}.service
  ls -al /etc/systemd/system | grep runner
  systemctl stop runner-{version}-{tenant-name}.service
  rm -rf /etc/systemd/system/runner-{version}-{tenant-name}.service
  rm -rf /root/yunxiao/{tenant-name}/runner/config
  # Delete the runner-related directory and its files on Machine B
  rm -rf /root/yunxiao/ 
  # Verify that the Flow Runner has been completely removed
  ps -ef | grep runner

Deployment failures or frequent disconnections when a single ECS instance belongs to multiple host groups

When the same ECS instance is added to multiple host groups, the Flow Runner services may conflict, causing deployment failures or frequent disconnections.

Cause analysis

This is typically caused by the Flow Runner going offline, running an outdated version, or having configuration conflicts between multiple Runner instances on the same machine.

Quick recovery

Run the following commands on the server to identify and restart the relevant Runner service:

# List all Runner service names
ls -al /etc/systemd/system | grep runner
# Restart the target Runner service
systemctl restart runner-{version}-{tenant-name}.service

Permanent fix

If the issue persists, uninstall the old Flow Runner and install the latest version (such as 0.3.4):

1. Stop the Runner service:

   systemctl stop runner-{version}-{tenant-name}.service

2. Delete the systemd configuration file:

   rm -rf /etc/systemd/system/runner-{version}-{tenant-name}.service

3. Delete the configuration directory:

   rm -rf /root/yunxiao

4. Reinstall the latest Runner plugin from the Alibaba Cloud DevOps console.

Troubleshoot Flow Runner issues

ECS instance shows a "deploy channel error" or appears offline

• On the ECS instance, verify that Cloud Assistant is running correctly. If not, restart it. For more information, see View execution results and troubleshoot common issues.

• If Cloud Assistant is working properly, check if the ECS instance disk is full. If so, free up disk space.

Host deployment fails to download the latest artifact

Check the logs for messages indicating a full disk. If the server's disk is full, free up space.

In addition to disk space, check the following common artifact download issues:

Slow download speed

Artifact download speed is primarily affected by the server's network bandwidth and the package size. To troubleshoot:

1. Check the bandwidth monitoring on your self-hosted instance to verify whether the bandwidth is fully utilized.

2. Run the wget command from the deployment script directly on the server to test the actual download speed and identify the bottleneck.

3. If the bandwidth is saturated, consider increasing bandwidth for the server.

MD5 checksum failure (file md5 is incorrect)

This error typically occurs when another pipeline uses the same deployment download path, causing file conflicts, or when parallel pipeline runs triggered by source code commits overwrite files. To resolve this issue:

1. Change the deployment download path to ensure that different pipelines use isolated paths.

2. Check the pipeline run history to verify whether parallel runs occurred.

3. Add a command to delete the old artifact package at the end of your deployment script to prevent file conflicts.

Download failure or insufficient permissions

In addition to disk space, check the following:

1. Verify whether multiple deployment orders are running simultaneously on the same machine, which can cause file overwrites.

2. Manually run the wget command from the deployment script on the target machine to verify network connectivity and artifact accessibility.

3. If you are using a private artifact repository, verify that the authorized card member has the required permissions. If not, add the member to the corresponding repository in the artifact repository console.

Deployment script exceeds the character limit

Alibaba Cloud DevOps Flow imposes a 2,000-character limit on deployment scripts. When a complex deployment script (such as a Python deployment script) exceeds this limit, do not write the script directly in the deployment configuration field. Instead:

1. Save the script as a shell file and add it to your code repository.

2. In the deployment task configuration, call the shell script file rather than embedding the full script inline.

Roll back multiple tasks

Rollbacks are performed on a per-task basis. If you have configured three tasks, you must roll back each one individually by selecting the task from the drop-down menu in the deployment history.

Deployment fails and the service does not start

Run the deployment commands directly on the server to debug your deployment script and ensure it is free of errors.

1. Alibaba Cloud DevOps executes the commands you configure in the deployment settings. Copy these commands and run them manually on the server. If the failure persists, debug the deployment script. For example, you can create a new .sh file on the server, paste the commands from your deployment script into it, and then execute the file to debug.

2. If the script runs successfully on the server but fails to start the service when run by Alibaba Cloud DevOps, check for relative paths in your script. Replace them with absolute paths and try again.

Docker container is inaccessible after deployment

If a Docker container starts successfully during deployment but the service is not accessible, follow these steps:

1. Verify that the ECS security group allows inbound traffic on the required port (for example, port 80).

2. Check port listening and firewall status on the server:

   netstat -ano | grep {port}
   systemctl status firewalld
   ufw status
   iptables -nvL

3. Compare the commit ID of the image built locally versus the image built in the pipeline, and verify that the deployment script is the same in both environments.

4. Manually run docker run on the ECS instance to start the container and test the service. This helps determine whether the issue is with the container image or the deployment environment.

docker-compose.yml file not found

Alibaba Cloud DevOps does not automatically sync code repository files to the target ECS instance during the deployment stage. If your deployment relies on a docker-compose.yml file, use one of the following solutions:

• Manually upload the docker-compose.yml file to the ECS instance before deployment.

• Add a pipeline step to download the code or artifact that includes the docker-compose.yml file before running docker-compose.

• Use the docker run command directly instead of docker-compose to start the container.

Application starts normally when deployed manually but fails with Alibaba Cloud DevOps

If an application (such as a Spring Boot application) starts successfully when deployed manually on the server but fails when deployed through Alibaba Cloud DevOps, check the temporary deployment script:

1. Locate the deployment script at /tmp/rdc_deploy_command_*.sh on the target server.

2. Run the script manually and verify whether the application starts correctly within the script's execution context.

3. Check for environment variable differences between the manual execution environment and the script's execution environment.

4. Check for working directory differences between the two environments.

5. If the script uses relative paths, replace them with absolute paths.

Windows host deployment fails with a "deploy channel error"

Alibaba Cloud DevOps does not currently support adding Windows hosts. Try one of the following workarounds:

1. Use a Linux server as an intermediary. Write commands in your deployment script that run on the Linux server to interact with the Windows host.

2. Use an Alibaba Cloud DevOps component to upload the build artifact to your own OSS bucket. The Windows host can then download the artifact from OSS for deployment.

Host deployment fails with a 'not a valid identifier' error

+ export pause_strategy=FirstBatchPause
+ pause_strategy=FirstBatchPause
+ export rootMixActionCode=ENTER
+ rootMixActionCode=ENTER
+ export TIMESTAMP=1607427603743
+ TIMESTAMP=1607427603743
+ export appId=0
+ appId=0
+ export execute_user=root
+ execute_user=root
+ export ENGINE_PIPELINE_CREATOR_ALIYUN_PK=1218858987274713
+ ENGINE_PIPELINE_CREATOR_ALIYUN_PK=1218858987274713
+ export triggerMode=1
+ triggerMode=1
+ export abc=hello world
+ abc=hello
/tmp/rdc_deploy_command_1607427664248_globalParams.sh: line 16: export: 'world': not a valid identifier
[ERROR] The environment variable contains special characters. Please follow the documentation to troubleshoot.
https://thoughts.aliyun.com/sharespace/5e86a419546fd9001aee81f2/docs/5fbca20e550406001d388abc
------Print environment variables------
]
[2020-12-08 19:41:09]Execution result code:[
exit 0
]
DeployCommand execution completed

This error occurs because an environment variable contains special characters, such as spaces. To use these variables, you must configure your pipeline as follows:

1. In the host deployment task, select the Encode variables checkbox.

2. In your deployment script, decode any environment variable you need to use with Base64. For example, to use the PIPELINE_ID environment variable, add the following line at the beginning of your script: export PIPELINE_ID=$(echo $PIPELINE_ID | base64 -d).

   + export PIPELINE_NAME=6aKE5Y+RYmFzZTY05rWL6K+V5rWB5rC057q/
   + PIPELINE_NAME=6aKE5Y+RYmFzZTY05rWL6K+V5rWB5rC057q/
   + export BUILD_NUMBER=Ng==
   + BUILD_NUMBER=Ng==
   + export FLOW_ENGINE_ENCODE_GLOBAL_PARAM=dHJ1ZQ==
   + FLOW_ENGINE_ENCODE_GLOBAL_PARAM=dHJ1ZQ==
   + export DATETIME=MjAyMC0xMi0wOC0yMC0yMi01MA==
   + DATETIME=MjAyMC0xMi0wOC0yMC0yMi01MA==
   + export parentMixFlowInstId=MA==
   + parentMixFlowInstId=MA==
   + export GIT_BRANCH=null
   + GIT_BRANCH=null
   + export CI_COMMIT_SHA=MmZhNThkMTk1OWFjYjcwZDQzNDZjZDBlM2E2YzhhYzQyNmE2MTE4NQ==
   + CI_COMMIT_SHA=MmZhNThkMTk1OWFjYjcwZDQzNDZjZDBlM2E2YzhhYzQyNmE2MTE4NQ==
   + export machine_group_id=MTE5MDA=
   + machine_group_id=MTE5MDA=
   + export ENGINE_PIPELINE_INST_ID=MjExNzYxMw==
   + ENGINE_PIPELINE_INST_ID=MjExNzYxMw==
   ------Print environment variables------
   hello world
   ]
   [2020-12-08 20:23:46]Execution result code:[
   exit 0
   ]
   DeployCommand execution completed

Host deployment fails with a User.NoPermission error

The deployment details show a specific error code. The following information describes the error and how to troubleshoot it.

User.NoPermission means the user lacks the required API permissions. Verify that the service connection for the deployment group is configured correctly.

Pipeline status is stuck on "Deploying" after successful service start

A host deployment script that remains in the "Deploying" state typically has one of two causes: an incorrectly passed exit code or a child process that fails to exit.

For a sample script, see https://atomgit.com/flow-example/spring-boot/blob/master/deploy.sh.

To diagnose and resolve the issue:

1. Verify the exit code

   • After critical commands in your script, such as the service startup command, add echo $? to ensure the exit code is 0. If a non-zero value is returned, check the error handling logic for that command.

   • Explicitly declare exit 0 at the end of your script to avoid relying on the exit code of the last command.

2. Manage child processes

   • If you use nohup to start a background process, such as a Java service, ensure you use the standard format: nohup java -jar ${JAR_NAME} > ${JAVA_OUT} 2>&1 &. The ampersand (&) at the end runs the process in the background and prevents it from blocking the main script.

   • Check other commands that might create child processes, such as systemd or docker run -d, to ensure they detach from the process context correctly.

3. Implement a robust timeout mechanism

   • If your service takes a long time to start, add a polling check (such as an HTTP health check) to your script. The script should only exit after the service is ready, instead of relying on a fixed timeout.

Example of a corrected script:

# Start the service and capture the exit code
nohup java -jar app.jar > log.txt 2>&1 &
echo "Service started with exit code: $?"
# Explicitly exit the script with a success code
exit 0

If the issue persists after these optimizations, check the pipeline configuration parameters for the current stage, or check if the Flow Runner service status is abnormal.

Windows batch script (.bat) causes the pipeline to hang

On Windows hosts, calling a .bat script (for example, service-start.bat) may cause the pipeline to hang indefinitely. This happens because the batch process starts a child process in a blocking manner, causing PowerShell to wait for the child process to exit before returning control.

To resolve this issue:

1. Wrap your deployment commands in a script and configure it through Alibaba Cloud DevOps to start as a background process, ensuring it does not block the current console session.

2. To identify the specific blocking command, reduce the number of commands in your script and test each command individually.

3. Check whether the blocking command starts a child process that does not terminate. If so, modify the startup method to run the process in the background.

"Clean Workspace" task gets stuck during a host deployment

If you manually installed the Docker service, use the following commands to check its status and perform related operations.

# Check Docker status
systemctl status docker
### Key output information:
# Active: active (running): The service is running.
# Active: inactive (dead): The service is not running.
# Loaded: loaded (...): The service configuration has been loaded correctly.
# View detailed information about the Docker client and server, including images, containers, and volumes.
docker ps
# Restart the Docker service
sudo systemctl restart docker