Troubleshoot host deployments

Self-hosted instance is offline

Verify that the agent on your server is online. If it is offline, restart it. The basic agent management commands are:

• Check status:

  /home/staragent/bin/staragentctl status

• Restart:

  /home/staragent/bin/staragentctl restart

• Uninstall:

  /home/staragent/bin/staragentctl stop
  rm -rf /home/staragent
  rm /usr/sbin/staragent_sn

Host deployment fails without error logs

If you are using a non-ECS machine, check if the self-hosted instance was created from an image. If so, uninstall the agent, add the self-hosted instance again, and retry the deployment.

Cannot add a non-ECS self-hosted instance to a host group

For non-ECS self-hosted instances, we recommend using the hybrid cloud hosting mode. For more information, see https://help.aliyun.com/document_detail/201140.html. Click New host group. In the Self-hosted Instance · Add Host dialog box, set Add Method to Hybrid Cloud Hosting, to register a non-Alibaba Cloud machine as an Alibaba Cloud-managed instance. Then, select a service authorization and region. You can then view and add your non-ECS self-hosted instance from the Available Hosts list.

Errors during cluster creation or runner installation

The error message is Failed to install runner for instance i-2ze99uh20nyrykgdobvb! (3000005).

In Alibaba Cloud DevOps Flow, you can create a host group or build cluster by either selecting an ECS instance or installing an agent on a self-hosted instance. However, you cannot use both methods to add the same machine to the same Alibaba Cloud DevOps organization.

For example, if you have already added Host A as an ECS instance, you cannot add it again as a self-hosted instance.

• To switch the registration method, you must first uninstall the Flow Runner from the host. Use the following commands to uninstall the Flow Runner:

  # Stop the specified systemd service.
  systemctl stop runner-{version}-{tenant-name}.service
  # Delete the systemd configuration file for the service.
  rm -rf /etc/systemd/system/runner-{version}-{tenant-name}.service
  # Delete the runner configuration directory for the tenant.
  rm -rf /root/yunxiao/{tenant-name}/runner/config

• The deployment command runs successfully when executed manually on the host but fails in the Flow pipeline.

  • Add the relevant environment variables for the command. For example: /source /root/.bash_profile; source /etc/profile.

  • Use absolute paths in deployment scripts (for example, /home/admin/app/deploy.sh) instead of relative paths.

Flow Runner configuration conflicts from ECS image cloning

During a host deployment, if you clone Machine A to create Machine B, Machine B reuses the Flow Runner configuration from Machine A. This can cause channel conflicts, leading to the following issues:

• Incorrect deployment target: Tasks intended for Machine B are incorrectly dispatched to Machine A.

• Root cause: After the image was cloned, the unique identifier of the Flow Runner (such as its registration information) was not reset. As a result, the scheduling system cannot distinguish between Machine A and Machine B.

• Key impact: This conflict disrupts automated workflows and can lead to environment contamination or service overlap.

• Solution:

  Note

  If you clone a machine with a configured Flow Runner, you must reinstall the Flow Runner on the new machine (Machine B) to ensure that Machine A and Machine B have unique channel IDs.

  # Uninstall the Flow Runner on Machine B
  # Check the Flow Runner service name. The name is typically in the format runner-{version}-{tenant-name}.service
  ls -al /etc/systemd/system | grep runner
  systemctl stop runner-{version}-{tenant-name}.service
  rm -rf /etc/systemd/system/runner-{version}-{tenant-name}.service
  rm -rf /root/yunxiao/{tenant-name}/runner/config
  # Delete the runner-related directory and its files on Machine B
  rm -rf /root/yunxiao/ 
  # Verify that the Flow Runner has been completely removed
  ps -ef | grep runner

Troubleshoot Flow Runner issues

ECS instance shows a "deploy channel error" or appears offline

• On the ECS instance, verify that Cloud Assistant is running correctly. If not, restart it. For more information, see View execution results and troubleshoot common issues.

• If Cloud Assistant is working properly, check if the ECS instance disk is full. If so, free up disk space.

Host deployment fails to download the latest artifact

Check the logs for messages indicating a full disk. If the server's disk is full, free up space.

Roll back multiple tasks

Rollbacks are performed on a per-task basis. If you have configured three tasks, you must roll back each one individually by selecting the task from the drop-down menu in the deployment history.

Deployment fails and the service does not start

Run the deployment commands directly on the server to debug your deployment script and ensure it is free of errors.

1. Alibaba Cloud DevOps executes the commands you configure in the deployment settings. Copy these commands and run them manually on the server. If the failure persists, debug the deployment script. For example, you can create a new .sh file on the server, paste the commands from your deployment script into it, and then execute the file to debug.

2. If the script runs successfully on the server but fails to start the service when run by Alibaba Cloud DevOps, check for relative paths in your script. Replace them with absolute paths and try again.

Windows host deployment fails with a "deploy channel error"

Alibaba Cloud DevOps does not currently support adding Windows hosts. Try one of the following workarounds:

1. Use a Linux server as an intermediary. Write commands in your deployment script that run on the Linux server to interact with the Windows host.

2. Use an Alibaba Cloud DevOps component to upload the build artifact to your own OSS bucket. The Windows host can then download the artifact from OSS for deployment.

Host deployment fails with a 'not a valid identifier' error

+ export pause_strategy=FirstBatchPause
+ pause_strategy=FirstBatchPause
+ export rootMixActionCode=ENTER
+ rootMixActionCode=ENTER
+ export TIMESTAMP=1607427603743
+ TIMESTAMP=1607427603743
+ export appId=0
+ appId=0
+ export execute_user=root
+ execute_user=root
+ export ENGINE_PIPELINE_CREATOR_ALIYUN_PK=1218858987274713
+ ENGINE_PIPELINE_CREATOR_ALIYUN_PK=1218858987274713
+ export triggerMode=1
+ triggerMode=1
+ export abc=hello world
+ abc=hello
/tmp/rdc_deploy_command_1607427664248_globalParams.sh: line 16: export: 'world': not a valid identifier
[ERROR] The environment variable contains special characters. Please follow the documentation to troubleshoot.
https://thoughts.aliyun.com/sharespace/5e86a419546fd9001aee81f2/docs/5fbca20e550406001d388abc
------Print environment variables------
]
[2020-12-08 19:41:09]Execution result code:[
exit 0
]
DeployCommand execution completed

This error occurs because an environment variable contains special characters, such as spaces. To use these variables, you must configure your pipeline as follows:

1. In the host deployment task, select the Encode variables checkbox.

2. In your deployment script, decode any environment variable you need to use with Base64. For example, to use the PIPELINE_ID environment variable, add the following line at the beginning of your script: export PIPELINE_ID=$(echo $PIPELINE_ID | base64 -d).

   + export PIPELINE_NAME=6aKE5Y+RYmFzZTY05rWL6K+V5rWB5rC057q/
   + PIPELINE_NAME=6aKE5Y+RYmFzZTY05rWL6K+V5rWB5rC057q/
   + export BUILD_NUMBER=Ng==
   + BUILD_NUMBER=Ng==
   + export FLOW_ENGINE_ENCODE_GLOBAL_PARAM=dHJ1ZQ==
   + FLOW_ENGINE_ENCODE_GLOBAL_PARAM=dHJ1ZQ==
   + export DATETIME=MjAyMC0xMi0wOC0yMC0yMi01MA==
   + DATETIME=MjAyMC0xMi0wOC0yMC0yMi01MA==
   + export parentMixFlowInstId=MA==
   + parentMixFlowInstId=MA==
   + export GIT_BRANCH=null
   + GIT_BRANCH=null
   + export CI_COMMIT_SHA=MmZhNThkMTk1OWFjYjcwZDQzNDZjZDBlM2E2YzhhYzQyNmE2MTE4NQ==
   + CI_COMMIT_SHA=MmZhNThkMTk1OWFjYjcwZDQzNDZjZDBlM2E2YzhhYzQyNmE2MTE4NQ==
   + export machine_group_id=MTE5MDA=
   + machine_group_id=MTE5MDA=
   + export ENGINE_PIPELINE_INST_ID=MjExNzYxMw==
   + ENGINE_PIPELINE_INST_ID=MjExNzYxMw==
   ------Print environment variables------
   hello world
   ]
   [2020-12-08 20:23:46]Execution result code:[
   exit 0
   ]
   DeployCommand execution completed

Host deployment fails with a User.NoPermission error

The deployment details show a specific error code. The following information describes the error and how to troubleshoot it.

User.NoPermission means the user lacks the required API permissions. Verify that the service connection for the deployment group is configured correctly.

Pipeline status is stuck on "Deploying" after successful service start

A host deployment script that remains in the "Deploying" state typically has one of two causes: an incorrectly passed exit code or a child process that fails to exit.

For a sample script, see https://atomgit.com/flow-example/spring-boot/blob/master/deploy.sh.

To diagnose and resolve the issue:

1. Verify the exit code

   • After critical commands in your script, such as the service startup command, add echo $? to ensure the exit code is 0. If a non-zero value is returned, check the error handling logic for that command.

   • Explicitly declare exit 0 at the end of your script to avoid relying on the exit code of the last command.

2. Manage child processes

   • If you use nohup to start a background process, such as a Java service, ensure you use the standard format: nohup java -jar ${JAR_NAME} > ${JAVA_OUT} 2>&1 &. The ampersand (&) at the end runs the process in the background and prevents it from blocking the main script.

   • Check other commands that might create child processes, such as systemd or docker run -d, to ensure they detach from the process context correctly.

3. Implement a robust timeout mechanism

   • If your service takes a long time to start, add a polling check (such as an HTTP health check) to your script. The script should only exit after the service is ready, instead of relying on a fixed timeout.

Example of a corrected script:

# Start the service and capture the exit code
nohup java -jar app.jar > log.txt 2>&1 &
echo "Service started with exit code: $?"
# Explicitly exit the script with a success code
exit 0

If the issue persists after these optimizations, check the pipeline configuration parameters for the current stage, or check if the Flow Runner service status is abnormal.

"Clean Workspace" task gets stuck during a host deployment

If you manually installed the Docker service, use the following commands to check its status and perform related operations.

# Check Docker status
systemctl status docker
### Key output information:
# Active: active (running): The service is running.
# Active: inactive (dead): The service is not running.
# Loaded: loaded (...): The service configuration has been loaded correctly.
# View detailed information about the Docker client and server, including images, containers, and volumes.
docker ps
# Restart the Docker service
sudo systemctl restart docker