Create alerts for exceeding scale-out thresholds using events

Background information

E-HPC integrates with EventBridge to deliver events related to E-HPC cluster operations to the default event bus. This integration enables flexible, custom delivery of E-HPC events using Function Compute. For example, you can create an alert policy for scale-out thresholds based on E-HPC node scale-out events. Then, you can send alert notifications through DingTalk using a custom event bus. The following figure illustrates this scenario:

Preparations

Before you start, complete the following preparations:

• Activate EventBridge. For more information, see Activate EventBridge and grant permissions.

• Activate Function Compute. For more information, see Activate Function Compute.

Step 1: Create a DingTalk chatbot

After you create a DingTalk chatbot, a unique webhook URL is generated. You can use this webhook URL to connect to other services, such as EventBridge, to receive notifications.

1. Log on to the DingTalk Developer Platform and apply for developer permissions.

2. Create and publish an internal enterprise chatbot. For more information, see Create and install an internal enterprise chatbot.

3. Open the DingTalk group where you want to receive event notifications and add the chatbot.

   1. In the upper-right corner of the DingTalk group, click the icon to open the Group Settings panel. Then, click Robot.

   2. In the Robot Management panel, click Add Robot. In the dialog box that opens, click Add Robot.

   3. Find the newly created enterprise chatbot, click its icon, and follow the on-screen instructions to add the chatbot.

4. Copy and save the chatbot's webhook URL.

   In the Robot Management panel, click the newly added chatbot to obtain the webhook URL.

Step 2: Create a custom event bus

Create a custom event bus to receive alert events delivered by Function Compute and push alert information to DingTalk.

1. Go to the Create Custom Event Bus panel.

   1. Log on to the EventBridge console.

   2. In the navigation pane on the left, click Event Buses.

   3. In the top navigation bar, select a region.

   4. In the Custom Event Bus section in the upper-right corner, click Quick Create.

2. On the Bus configuration page, enter a name and description, and click Next.

3. On the Event Source configuration page, enter an event source name, select Custom Application as the event provider, and then click Next.

4. On the Rule configuration page, enter a rule name, and click Next.

5. On the Target configuration page, complete the configurations, and then click Create.

   When you configure the target, note the following parameters:

   • Service Type: Select DingTalk.

   • Address: Enter the Webhook URL that you saved in Step 1.

   • Key: Enter the signing key that you saved in Step 1.

   • Push Content: Configure the content as shown in the following example. For more information, see Event content transform.

     • Variable

       {
         "content": "$.data.content"
       }

     • Template

       {
         "msgtype": "text", 
         "text": {
             "content": "${content}"
         }
       }

Step 3: Create a function

Create a function in Function Compute to receive E-HPC cluster scale-out events from the default event bus. This function checks whether the number of scaled-out nodes exceeds a configured threshold. If the threshold is exceeded, the function delivers an alert event to the custom event bus.

1. Create a Function Compute service.

   For more information, see Create a service. When you create the service, configure the Service Role in the advanced options. This role must have permissions to access EventBridge, such as the AliyunFcDefaultRole permission.

2. Create a function.

   For more information, see Create a function. When you create the function, in the Function Code section, set the runtime environment to Python 3.6 and keep the default values for other parameters.

3. Write the function code.

   1. On the function details page, click the Function Code tab.

   2. In the Terminal in the lower-right corner, run the following commands to install the EventBridge Python SDK.

      pip install alibabacloud_eventbridge -t .
      pip install alibabacloud_tea_console -t .

   3. In the Resource Manager on the left, open `index.py` and write the function code. After you finish, click Deploy Code.

      By default, `index.py` contains the sample code you selected when creating the function. Replace it with the code for delivering events when the scale-out threshold is exceeded.

      The following is a code example. Replace the parameters in the code with your actual configurations.

      Parameter	Example Value	Description
      MAX_COUNT	5	The scale-out threshold. An alert event is delivered if the number of scaled-out nodes exceeds this value.
      config.endpoint	eventbridge.cn-hangzhou.aliyuncs.com	The endpoint of the event bus.
      event.source	my.event	The name of the event source for the custom event bus.
      aliyuneventbusname	test	The name of the custom event bus.

      # -*- coding: utf-8 -*-
      import logging
      import json
      
      from alibabacloud_eventbridge.client import Client as EventBridgeClient
      from alibabacloud_eventbridge import models as event_bridge_models
      from alibabacloud_tea_console.client import Client as ConsoleClient
      from alibabacloud_tea_util.client import Client as UtilClient
      
      MAX_COUNT = 5
      
      def create_client(creds):
          """
          Create client and initialize public request parameters
          """
          config = event_bridge_models.Config()
          config.access_key_id = creds.access_key_id
          config.access_key_secret = creds.access_key_secret
          config.security_token = creds.security_token
          config.endpoint = "eventbridge.cn-hangzhou.aliyuncs.com"
          return EventBridgeClient(config)
      
      def put_events(client, content):
          """
          PutEvents: Deliver a custom bus event
          """
          event = event_bridge_models.CloudEvent()
          data =  json.dumps(content)
      
          event.datacontenttype = "application/json"
          event.data = UtilClient.to_bytes(data)
          event.id = "a5074581-7e74-4e4c-868f-47e7afdf****"
          # The event source of the custom event bus
          event.source = "my.event"
          event.specversion = "1.0"
          event.type = "ui:Created:PostObject"
          event.time = "2022-08-30T03:48:37.492Z"
          event.subject = "my:subject"
          # The name of the custom event bus
          event.extensions = {
              "aliyuneventbusname": "test"
          }
          try:
              resp = client.put_events([event])
              ConsoleClient.log("--------------------Publish event to the aliyun EventBus--------------------")
              ConsoleClient.log(UtilClient.to_jsonstring(resp.to_map()))
          except Exception as error:
              ConsoleClient.log(error)
      
      def handler(event, context):
        logger = logging.getLogger()
        evt = json.loads(event)
        actual_count = evt['data']['Count']
        # If the actual number of scaled-out nodes exceeds the preset threshold, deliver an alert event to the custom event bus.
        if actual_count > MAX_COUNT:
          creds = context.credentials
          client = create_client(creds)
          content = { 'content': 'Your cluster %s (%s) scaled out %d nodes at %s in a single operation, which exceeds your set threshold of %d. The request ID for this scale-out operation is %s.' % 
              (evt['data']['ClusterName'], evt['data']['ClusterId'], evt['data']['CreateTime'], actual_count, MAX_COUNT, evt['data']['RequestId'])}
          put_events(client, content)
          return 'call putevent!'
        else:
          return 'do nothing!'

Step 4: Configure a default event bus trigger

Create a rule for the default event bus to deliver E-HPC cluster scale-out events to Function Compute.

1. On the Event Buses page of the EventBridge console, click default.

2. In the navigation pane on the left, click Event Rules, and then click Create Rule.

3. On the Create Rule panel, complete the rule configuration.

   1. On the Configure Basic Information page, enter a name and description, and click Next.

   2. On the Configure Event Pattern page, configure the event pattern as follows, and then click Next.

      The following are the parameter descriptions. Keep the default values for parameters that are not mentioned.

      • Event Source Type: Select Alibaba Cloud Service Event Source.

      • Event Source: Select acs.ehpc.

      • Event Type: Select the ehpc:NodeOperation:NodeCreate event.

   3. On the Configure Event Target page, configure the event target as follows, and then click Create.

      The following are the parameter descriptions. Keep the default values for parameters that are not mentioned.

      • Service Type: Select Function Compute.

      • Service: Select the Function Compute service that you created in Step 3.

      • Function: Select the function that you created in Step 3.

Verify the results

In this example, an alert notification is sent through DingTalk when an E-HPC cluster is scaled out by more than five nodes. Follow these steps to verify the result.

1. Manually scale out the nodes of a cluster. For more information, see Scale out a cluster.

2. Check the message notifications in the DingTalk group.