Install KubeRay in ACK

Prerequisites

Ensure the following:

• An ACK Managed Cluster Pro running Kubernetes v1.24 or later with at least one node of 8 vCPUs and 32 GB of memory. For GPU workloads, use GPU-accelerated instance types.

• kubectl installed and connected to the cluster.

Create a cluster or upgrade an existing one if needed.

Install KubeRay Operator

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. Click the name of your cluster.

3. On the cluster details page, choose Operations > Add-ons > Manage Applications.

4. Find and install Kuberay-Operator.

   

Verify: After installation, confirm the operator pod is running:

kubectl get pods -A | grep kuberay

The operator pod should show Running status. If it stays in Pending, run kubectl describe pod <pod-name> to check for scheduling errors such as insufficient node resources.

Enable log collection for KubeRay Operator

Collect control-plane logs from the KubeRay Operator itself.

1. On the cluster details page, choose Operations > Log Center > Control Plane Component Logs.

2. Click Enable Component Log Collection.

3. Select kuberay-operator.

Enable log collection for Ray clusters

Forward Ray pod logs to SLS via an AliyunLogConfig custom resource. Logtail watches the Ray log directory on each pod and tags entries by cluster name and node type.

1. Apply this manifest to create an AliyunLogConfig object in kube-system:

   Sample code

   cat <<EOF | kubectl apply -f -
   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     name: rayclusters
     namespace: kube-system
   spec:
      # The name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates one.
     logstore: rayclusters
     # Configure Logtail.
     logtailConfig:
       # The type of data source. If you want to collect text logs, you must set the value to file.
       inputType: file
       # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name.
       configName: rayclusters
       inputDetail:
         # Configure Logtail to collect text logs in simple mode.
         logType: common_reg_log
         # The path of the log file.
         logPath: /tmp/ray/session_*-*-*_*/logs
         # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log.
         filePattern: "*.*"
         # If you want to collect container text logs, you must set dockerFile to true.
         dockerFile: true
         # The filter conditions of containers.
         advanced:
           k8s:
             IncludeK8sLabel:
               ray.io/is-ray-node: "yes"
             ExternalK8sLabelTag:
               ray.io/cluster: "_raycluster_name_"
               ray.io/node-type : "_node_type_"
   EOF

   Key parameters:

   Parameter	Description
   logPath	Log directory on each Ray pod. Matches all sessions under /tmp/ray/. Customize if your cluster uses a different path.
   advanced.k8s.ExternalK8sLabelTag	Adds _raycluster_name_ and _node_type_ tags to each log entry for filtering in SLS.

   All AliyunLogConfig parameters are documented in Use CRDs to collect container logs in DaemonSet mode.

2. View logs in the ACK console:

   1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

   2. Click the name of your cluster.

   3. Choose Cluster Information > Basic Information > Cluster Resources.

   4. Click the Log Service Project link to open the SLS project.

3. Select the rayclusters Logstore. Filter by _raycluster_name_ to isolate logs from a specific Ray cluster.

   

Enable monitoring for Ray clusters

Collect Prometheus metrics from Ray pods with a PodMonitor for workers and a ServiceMonitor for the head node. Workers need a PodMonitor because they are standalone pods; the head node exposes metrics through a stable Service endpoint.

1. Create a PodMonitor to collect metrics from Ray worker pods:

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     annotations:
       arms.prometheus.io/discovery: 'true'
       arms.prometheus.io/resource: arms
     name: ray-workers-monitor
     namespace: arms-prom
     labels:
       # `release: $HELM_RELEASE`: Prometheus can only detect PodMonitor with this label.
       release: prometheus
       #ray.io/cluster: raycluster-kuberay # $RAY_CLUSTER_NAME: "kubectl get rayclusters.ray.io"
   spec:
     namespaceSelector:
       any: true
     jobLabel: ray-workers
     # Only select Kubernetes Pods with "matchLabels".
     selector:
       matchLabels:
         ray.io/node-type: worker
     # A list of endpoints allowed as part of this PodMonitor.
     podMetricsEndpoints:
     - port: metrics
       relabelings:
       - action: replace
         regex: (.+)
         replacement: $1
         separator: ;
         sourceLabels:
           - __meta_kubernetes_pod_label_ray_io_cluster
         targetLabel: ray_io_cluster

2. Create a ServiceMonitor to collect metrics from the Ray head node:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     annotations:
       arms.prometheus.io/discovery: 'true'
       arms.prometheus.io/resource: arms
     name: ray-head-monitor
     namespace: arms-prom
     labels:
       # `release: $HELM_RELEASE`: Prometheus can only detect ServiceMonitor with this label.
       release: prometheus
   spec:
     namespaceSelector:
       any: true
     jobLabel: ray-head
     # Only select Kubernetes Services with "matchLabels".
     selector:
       matchLabels:
         ray.io/node-type: head
     # A list of endpoints allowed as part of this ServiceMonitor.
     endpoints:
       - port: metrics
         path: /metrics
     targetLabels:
     - ray.io/cluster

3. View dashboards in Application Real-Time Monitoring Service (ARMS):

   1. Log on to the ARMS console. In the left-side navigation pane, click Integration Center.

   2. Search for Ray and select it.

   3. In the Ray panel, select your cluster and click OK.

   4. In the left-side navigation pane, click Integration Management, then click the target environment name.

   5. On the Component Management tab, find Dashboards under Addon Type, then click Ray Cluster.

   6. Set Namespace, RayClusterName, and SessionName to filter monitoring data for a specific task.

Verify: In the ARMS console, both ray-workers-monitor and ray-head-monitor should appear as active scrape targets. If a target is missing, verify the release: prometheus label and that the namespace selector matches your Ray cluster namespace.

Next steps

• Deploy a RayCluster custom resource to run distributed Ray workloads.

• Use the _raycluster_name_ tag in SLS to correlate logs across Ray clusters.

• Set up ARMS alerting rules based on Ray cluster metrics from Prometheus.