Configure node self-healing notifications

SMS, email, and in-console notifications

1. Log in to the PAI console.

2. Click in the upper-right corner to open Message Center.

3. In the left-side navigation pane, choose Message receiving management > Basic receiving management.

4. In the Message type column, find Product O&M notifications, and select In-console notification, Email, or SMS.

5. The default notification recipient is the Alibaba Cloud primary account. You can also click Modify in the Actions column to configure more contacts.

6. When an abnormal node is detected, the system notifies you of the affected node name, resource quota, and jobs running on the node after the configuration is complete.

Chatbot notifications

1. Log in to the PAI console.

2. Click in the upper-right corner to open Message Center.

3. In the left-side navigation pane, choose Message receiving management > Bot receiving management.

4. Add a chatbot. Skip this step if you have already added one.

   1. In the upper-right corner of the Bot receiving management page, click Bot management.

   2. On the Bot management page, follow the platform links at the top to get the chatbot webhook information, and then click Add chatbot to complete the configuration.

   3. Click Test in the Actions column to test chatbot connectivity. A dialog showing Test successful indicates that the connection is normal.

5. On the Bot receiving management page, find Product O&M notifications in the Message type column, and click Modify in the Actions column.

6. Switch to the Message receiving chatbot tab, select the added chatbot, and click Save.

Custom notification rules

You can configure different chatbots and notification rules based on your business requirements. For example, if one team cares only about PAI messages and another team needs all Alibaba Cloud product O&M notifications, configure two chatbots:

• Chatbot test1: receives all messages. No notification rule is required.

• Chatbot test2: receives only messages that contain the PAI keyword. To configure this rule:

1. In the left-side navigation pane of the Message Center page, choose Message receiving management > Bot receiving management.

2. Find Product O&M notifications in the Message type column and click Modify in the Actions column.

3. Switch to the Message receiving chatbot tab. In the Notification rule column of the target recipient, click Edit. On the Configure custom notification rule page, set an allowlist keyword, such as PAI. After this configuration, you receive only PAI-related notifications.

   You can also select multiple recipients and click Batch edit rules to configure rules in batches.

4. After the configuration is complete, click OK, and then click Save on the Modify message receiving configuration page. After the configuration succeeds, Enabled allowlist is displayed in the Notification rule column.

Manual migration

For DSW instances on an abnormal node, if you have the instance open in the browser, a dialog prompts you to save the environment and close the instance as soon as possible to ensure node self-healing.

Automatic instance migration

Use Automatic Instance Migration (from Abnormal Node) to migrate DSW instances automatically when a node becomes abnormal.

1. Log in to the PAI console.

2. In the left-side navigation pane, click Workspaces, find the workspace in the list, and click its name to open it.

3. On the Workspace Details page, choose Workspace configuration Scheduling Settings.

4. In the DSW section, turn on the Enable Automatic Instance Migration from Abnormal Node switch.

   After this feature is enabled, when a node in the underlying Lingjun infrastructure becomes abnormal, the system automatically closes and restarts the instance to support node self-healing and keep resources available. The restart process saves the environment image, but running processes cannot be recovered.

For DSW instances on an abnormal node, if you have the instance open in the browser, a dialog prompts you to save the environment and close the instance. The dialog also displays the remaining time before automatic restart so that node self-healing can proceed.