Persist workflow pod logs with Simple Log Service (SLS) to query them after pods are deleted.
How it works
When you create a workflow cluster, the system creates an SLS project named k8s-log-<clusterid>. All pod logs are collected into a Logstore named workflow-logstore.
Once enabled, you can view logs from the Argo CLI, Argo Server, or Argo UI — regardless of whether the pod is still running.
Limitations
-
Compared with
oss-artifact-repository, SLS provides automatic log lifecycle management with configurable retention days and powerful query capabilities. To collect logs by using SLS, remove thearchiveLogs: truesetting. -
After log collection is enabled, you can view workflow logs before the workflow is deleted, even if the pod is deleted.
-
If a workflow is persisted to a database, you can download its logs as a ZIP file or view them in the SLS console even after the workflow is deleted. See Persist workflows.
Enable SLS
If the k8s-log-<clusterid> project was not created automatically, create it manually. See Create an SLS project.
Choose one of the following methods:
| Method | Best for |
|---|---|
| Alibaba Cloud Argo CLI | Quick setup with a single interactive command |
| AliyunLogConfig | GitOps workflows or declarative configuration management |
Enable SLS with the Alibaba Cloud Argo CLI
The Alibaba Cloud Argo CLI extends the open-source Argo CLI with log collection support for deleted pods.
Prerequisites
-
Alibaba Cloud Argo CLI 3.4.12 is installed.
-
If you use a RAM user, the AliyunLogReadOnlyAccess policy is attached to the RAM user. See Create a RAM user and authorize the RAM user to access SLS.
Steps
Configure SLS. When prompted, enter the log retention days (default: 7).
argo config sls
Please input log retention days. Default is 7 days.
10
Expected output:
Start to config SLS for your cluster.
Created AliyunLogConfig CR workflow-sls-config in default namespace.
Created SLS logstore workflow-logstore in SLS project k8s-log-<clusterid>, log retention days is 10 days
The output confirms that SLS is configured. All workflow logs are collected to the workflow-logstore Logstore in the k8s-log-<clusterid> project.
Enable SLS with AliyunLogConfig
Apply the following YAML to create an AliyunLogConfig custom resource. The SLS controller automatically creates the k8s-log-<clusterid> project and workflow-logstore Logstore.
cat << EOF | kubectl apply -f -
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
name: workflow-sls-config
namespace: default
spec:
lifeCycle: 5 # Log retention period in days
logstore: workflow-logstore
logtailConfig:
configName: workflow-sls-config
inputType: plugin # Use the plugin-based input method
inputDetail:
plugin:
inputs:
- type: service_docker_stdout # Collect container stdout and stderr
detail:
Stdout: true
Stderr: true
EOF
Verify log collection
After enabling SLS, confirm that logs are being collected:
-
Log on to the SLS console.
-
Find the
k8s-log-<clusterid>project. -
Click workflow-logstore and check that log entries appear.
If no logs appear, check whether the AliyunLogConfig custom resource was created:
kubectl get aliyunlogconfigs -n default
The output should include workflow-sls-config with no error status. If missing or errored, re-run the enable steps or verify RAM user permissions.
Access logs with the Argo CLI
The Argo CLI provides two ways to view workflow logs. The direct method (--sls flag) is recommended because it does not require Argo Server.
You can run kubectl logs <pod-name> to view logs of an existing pod, but this command returns up to 2,000 rows by default. To view more logs or logs for a deleted pod, use one of the following methods.
Use the Argo CLI directly (recommended)
Prerequisites
-
Alibaba Cloud Argo CLI 3.4.12 is installed.
-
If you use a RAM user, the AliyunLogReadOnlyAccess policy is attached to the RAM user.
Steps
-
Grant the Argo CLI access to your cluster:
argo config init # Follow the prompts to enter your AccessKey ID and AccessKey secret. -
Query logs:
# View logs for a specific pod argo logs <workflow-name> <pod-name> --sls # Download all logs for a workflow as a ZIP file argo logs <workflow-name> --sls
Use Argo Server
To access SLS through Argo Server, set the following environment variables.
To query SLS logs via Argo Server, you must specify<pod-name>. To access a different namespace, obtain theKUBE_TOKENfor that namespace.
-
Set the required environment variables:
export ARGO_SERVER=argo.<cluster id>.<region>.alicontainer.com:2746 export KUBE_TOKEN=$(kubectl create token default -n default --duration 24h) export ARGO_TOKEN="Bearer $KUBE_TOKEN" export ARGO_INSECURE_SKIP_VERIFY=true -
Query logs for a specific pod:
argo logs <workflow-name> <pod-name>
View logs for workflows persisted to databases
If a workflow is persisted to a database, you can download its logs after the workflow is deleted.
-
Grant the Argo CLI access to your cluster:
argo config init # Follow the prompts to enter your AccessKey ID and AccessKey secret. -
Get the archived workflow UID:
argo archive listExpected output:
NAMESPACE NAME STATUS AGE DURATION PRIORITY MESSAGE P/R/C PARAMETERS UID default hello-world-l6c2r Succeeded 3d 1m 0 0/0/0 179eaef0-fde3-496f-946d-549e8f****** -
Download workflow logs with the UID:
argo archive logs 179eaef0-fde3-496f-946d-549e8f******The logs are saved as
<workflow-name>.zipin the current directory.
View logs in the Argo UI
The Argo UI displays workflow pod logs directly. If a pod is deleted, SLS logs are shown instead of live pod output.
Disable SLS
-
Delete the
AliyunLogConfigcustom resource:kubectl delete aliyunlogconfigs.log.alibabacloud.com workflow-sls-config -n default -
Log on to the SLS console and delete the
workflow-logstoreLogstore.
Next steps
-
Persist workflows — Store workflow state in a database to retain logs and results after deletion.
-
Alibaba Cloud Argo CLI — Download and configure the CLI for full log access.