Cloud-native AI Suite O&M guide

Deploy the suite

1. Log in to the ACK console and click Clusters in the left navigation pane.

2. Click the cluster name, then choose Applications > Cloud-native AI Suite in the left pane.

3. Click Deploy.

4. Select the components to install, then click Deploy Cloud-native AI Suite at the bottom of the page. The system checks environment dependencies before deploying the selected components. After installation, the Components list shows component names and versions. From this list you can Deploy, Upgrade, or Uninstall individual components.

5. After installing ack-ai-dashboard and ack-ai-dev-console, links to AI Dashboard and AI Developer Console appear in the upper-left corner of the Cloud-native AI Suite page.

   

Configure AI Dashboard

Grant RAM permissions to the worker role

Before AI Dashboard can access cluster data, attach a custom RAM policy to the cluster's worker role.

1. Create a custom policy.

   1. Log in to the RAM console and choose Permissions > Policies in the left navigation pane.

   2. Click Create Policy, select the JSON tab, and add the following policy:

      {
          "Version": "1",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "cs:*",
                      "log:GetProject",
                      "log:GetLogStore",
                      "log:GetConfig",
                      "log:GetMachineGroup",
                      "log:GetAppliedMachineGroups",
                      "log:GetAppliedConfigs",
                      "log:GetIndex",
                      "log:GetSavedSearch",
                      "log:GetDashboard",
                      "log:GetJob",
                      "ecs:DescribeInstances",
                      "ecs:DescribeSpotPriceHistory",
                      "ecs:DescribePrice",
                      "eci:DescribeContainerGroups",
                      "eci:DescribeContainerGroupPrice",
                      "log:GetLogStoreLogs",
                      "ims:CreateApplication",
                      "ims:UpdateApplication",
                      "ims:GetApplication",
                      "ims:ListApplications",
                      "ims:DeleteApplication",
                      "ims:CreateAppSecret",
                      "ims:GetAppSecret",
                      "ims:ListAppSecretIds",
                      "ims:ListUsers"
                  ],
                  "Resource": "*"
              }
          ]
      }

   3. Name the policy using the format k8sWorkerRolePolicy-{ClusterID} and click OK.

2. Attach the policy to the cluster's worker role.

   1. In the RAM console, choose Identities > Roles and search for the role in the format KubernetesWorkerRole-{ClusterID}.

   2. Click Grant Permission for the role.

   3. In the Select Policy section, click Custom Policy, search for the policy you created (k8sWorkerRolePolicy-{ClusterID}), select it, and click OK.

Complete the AI Dashboard setup

1. On the Cloud-native AI Suite page, select Sample Console in the Interaction Mode section. The Note dialog box appears.

   • If Authorized is shown, skip to step 3.

   • If Unauthorized is shown in red and OK is dimmed, complete the RAM permissions above, then click Authorization Check. After authorization succeeds, Authorized is displayed.

2. Set the Console Data Storage parameter. Select Pre-installed MySQL for testing or ApsaraDB RDS for production. For details, see Install and configure AI Dashboard and AI Developer Console.

3. Click Deploy Cloud-native AI Suite. AI Dashboard is ready when its status changes to Ready.

(Optional) Create and accelerate a dataset

Algorithm developers can mount datasets from Object Storage Service (OSS) as persistent volumes (PVs) and accelerate access through AI Dashboard.

Create a PV and PVC

1. Create a namespace:

   kubectl create ns demo-ns

2. Create a file named fashion-mnist.yaml with the following content:

   Placeholder	Description	Example
   fashion-mnist	OSS bucket name	my-dataset-bucket
   oss-cn-beijing.aliyuncs.com	OSS endpoint for the bucket's region	oss-cn-hangzhou.aliyuncs.com
   AKID	AccessKey ID	LTAI5tXxx
   AKSECRET	AccessKey secret	xXxXxXx

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: fashion-demo-pv
   spec:
     accessModes:
     - ReadWriteMany
     capacity:
       storage: 10Gi
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeAttributes:
         bucket: fashion-mnist
         otherOpts: "-o max_stat_cache_size=0 -o allow_other"
         url: oss-cn-beijing.aliyuncs.com
         akId: "AKID"
         akSecret: "AKSECRET"
       volumeHandle: fashion-demo-pv
     persistentVolumeReclaimPolicy: Retain
     storageClassName: oss
     volumeMode: Filesystem
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: fashion-demo-pvc
     namespace: demo-ns
   spec:
     accessModes:
     - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: fashion-demo-pv
     storageClassName: oss
     volumeMode: Filesystem
     volumeName: fashion-demo-pv

   Replace the following placeholders:

3. Apply the manifest:

   kubectl create -f fashion-mnist.yaml

4. Verify the PV and PVC are bound:

   kubectl get pv fashion-demo-pv
   kubectl get pvc fashion-demo-pvc -n demo-ns

   Expected output for the PV:

   NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   AGE
   fashion-demo-pv   10Gi       RWX            Retain           Bound    demo-ns/fashion-demo-pvc     oss            8h

   Expected output for the PVC:

   NAME               STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   fashion-demo-pvc   Bound    fashion-demo-pv   10Gi       RWX            oss            8h

Accelerate the dataset

1. Log in to AI Dashboard as an administrator.

2. Choose Dataset > Dataset List in the left navigation pane.

3. Find the dataset (fashion-demo-pvc) and click Accelerate in the Operator column.

   

Cluster dashboard

After logging in, AI Dashboard opens the cluster dashboard by default. It shows cluster-wide GPU health and allocation.

GPU Utilization shows whether the GPU executed any work during the sample window — it does not indicate how efficiently the hardware was used. A node showing 100% GPU Utilization may be running lightweight kernels rather than heavy parallel workloads. Pair this metric with GPU Memory (Used/Total) to get a fuller picture of actual compute load.

Node dashboard

Click Nodes in the upper-right corner of the Cluster page to open the node dashboard. It breaks down GPU metrics per node and per GPU device.

Training job dashboard

Click TrainingJobs in the upper-right corner of the Nodes page to open the training job dashboard. It shows resource consumption and GPU efficiency per job.

Resource quota dashboard

Click Quota in the upper-right corner of the Training Jobs page to open the quota dashboard. It displays quota consumption by resource type (CPU, memory, nvidia.com/gpu, aliyun.com/gpu-mem, aliyun.com/gpu).

Set up a quota tree

1. Create namespaces for each team. If namespaces already exist, make sure they contain no running pods before associating them with a quota node.

   kubectl create ns namespace1
   kubectl create ns namespace2
   kubectl create ns namespace3
   kubectl create ns namespace4

2. In AI Dashboard, create quota nodes and associate each leaf node with a namespace. Set Min (guaranteed minimum) and Max (maximum when cluster has idle capacity) for each node.

Create users and user groups

A user can belong to multiple user groups. A user group can contain multiple users. Associate users with user groups to grant access to the resources allocated to that group.

1. Create a user. See Generate the kubeconfig file and logon token for a new user.

2. Create a user group. See Add a user group.

Capacity scheduling example

This example shows how the scheduler shares and reclaims CPU resources across four namespaces. The quota tree has this structure:

Quota configuration:

Walk-through:

Without elastic quotas, each leaf namespace can only use its Min (10 cores, so 2 pods at 5 cores/pod). With elastic quotas enabled and 40 cores available in the cluster, namespaces can borrow idle capacity up to their Max.

Step 1: Deploy five pods to namespace1, each requesting 5 CPU cores (25 cores total).

With root.a.1 Max set to 20 cores, 4 pods run (20 cores). The fifth pod stays Pending.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx1
  namespace: namespace1
  labels:
    app: nginx1
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx1
  template:
    metadata:
      name: nginx1
      labels:
        app: nginx1
    spec:
      containers:
      - name: nginx1
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Step 2: Deploy five pods to namespace2, each requesting 5 CPU cores.

With 20 cores remaining (40 total minus 20 used by namespace1), 4 pods run. The fifth stays Pending. Both namespace1 and namespace2 together now consume all 40 root cores.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx2
  namespace: namespace2
  labels:
    app: nginx2
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx2
  template:
    metadata:
      name: nginx2
      labels:
        app: nginx2
    spec:
      containers:
      - name: nginx2
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Step 3: Deploy five pods to namespace3, each requesting 5 CPU cores.

The cluster has no idle capacity. The scheduler reclaims 10 cores from root.a to guarantee root.b.1's minimum. It reclaims 5 cores from root.a.1 (reducing namespace1 from 4 running pods to 3) and 5 cores from root.a.2 (reducing namespace2 from 4 running pods to 3). With 10 reclaimed cores, 2 pods run in namespace3.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx3
  namespace: namespace3
  labels:
    app: nginx3
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx3
  template:
    metadata:
      name: nginx3
      labels:
        app: nginx3
    spec:
      containers:
      - name: nginx3
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Step 4: Deploy five pods to namespace4, each requesting 5 CPU cores.

The scheduler reclaims another 10 cores from root.a to guarantee root.b.2's minimum: 5 cores from root.a.1 and 5 from root.a.2. After reclamation, namespace1 and namespace2 each have 2 running pods (10 cores), and namespace4 gets 2 running pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx4
  namespace: namespace4
  labels:
    app: nginx4
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx4
  template:
    metadata:
      name: nginx4
      labels:
        app: nginx4
    spec:
      containers:
      - name: nginx4
        image: nginx
        resources:
          limits:
            cpu: 5
          requests:
            cpu: 5

Result:

Each team's minimum guarantee was honored. Teams that borrowed idle capacity had resources reclaimed when other teams needed their guaranteed minimums back.