CDP DingTalk alerting support-Cloudera Data Platform(CDP)-阿里云帮助中心

This topic describes how to configure CDP alerts for DingTalk.

CDP supports multiple alerting methods. By default, it supports email and Simple Network Management Protocol (SNMP) alerts. For more information about how to configure these two alert patterns, see the official CDP documentation.

In addition to these two alert patterns, CDP also supports custom alert scripts. When the health status of a cluster triggers an alert threshold, CDP calls a custom alert script to send an alert.

For example:

The path specified for the alerting script is: /opt/cloudera/alert/alert.py

Then when an alert is triggered, CDP calls: /opt/cloudera/alert/alert.py error.log

The name of the file that contains the alert information is passed as the first parameter to the alert script.

In the alert script, you can process the alert content, extract important information, and send it to your DingTalk robot using an HTTP request. The robot then forwards this information to your DingTalk group. This process enables DingTalk alerting.

The preceding example explains the basic principle of DingTalk alerting. For detailed instructions, see the following sections.

1. Create an alerting robot in your O&M group

In the upper-right corner of your DingTalk group, navigate to Settings -> Group Assistant -> Add Robot -> Custom -> Add. Then, enter the robot information, click Done, and save the robot's webhook.

Note

When you enter the robot information, navigate to Security Settings and set a Custom Keyword. Only messages that contain this keyword are forwarded. In this example, the keyword is "CDP alert". You can customize this keyword.
The Webhook URL is a DingTalk URL that contains an access token: https://oapi.dingtalk.com/robot/send?access_token=xxx
Save the access token. You will need it for the script.

2. Write the alert script

The preceding section explains the principle of alerting. The alert script acts as a relay to forward alert information from CDP to the DingTalk robot.

An example script is provided below. In this script, you must modify the following two constants based on your DingTalk robot settings:

access_token: Set this to the value of the access_token query parameter from your robot's webhook.
robot_keyword: Set this to the custom keyword that you configured when you created the robot.

After you test the alert script, place it in the /opt/cloudera/alert folder.

Create the alert folder: cd /opt/cloudera; mkdir alert; chown cloudera-scm:cloudera-scm alert
Create the alert.py file: cd /opt/cloudera/alert; touch alert.py; chown cloudera-scm:cloudera-scm alert.py; chmod +x alert.py
Copy the tested script into the alert.py file.

#!/usr/bin/python3
import sys
import urllib.request
import json

MAX_RETRIES = 3
# Modify this based on the webhook of your DingTalk group robot.
access_token = "xxx"
# Keyword: This must be the same as the custom keyword for your DingTalk robot. Otherwise, the message is not forwarded and error code 300 is returned.
robot_keyword = "CDP alert"


def sent_ding_talk(token: str, alert_content, retry_times: int = 1) -> bool:
    if retry_times > MAX_RETRIES:
        return False
    webhook = "https://oapi.dingtalk.com/robot/send?access_token={}".format(token)
    headers = {'Content-Type': 'application/json'}
    msg = {
        'msgtype': 'text',
        'text': {
            "content": alert_content
        }
    }
    post_data = json.dumps(msg).encode('utf-8')
    req = urllib.request.Request(url=webhook, data=post_data, headers=headers, method='POST')
    response = json.load(urllib.request.urlopen(req))
    if 'errmsg' in response and response['errmsg'] != 'ok':
        sent_ding_talk(token, alert_content, retry_times + 1)
    else:
        return True


def test_send_ding_talk():
    assert sent_ding_talk(access_token, robot_keyword)


if __name__ == '__main__':
    with open(sys.argv[1], 'r') as f:
        errors = json.load(f)
        for error in errors:
            # The alert content must include the custom keyword from the robot's security settings. Otherwise, the robot does not forward the alert content.
            alert_content = robot_keyword + ", "
            if error['header']['type'] == 'alert':
                # For more alert information, see the alert JSON format:
                # https://docs.cloudera.com/cloudera-manager/7.4.2/monitoring-and-diagnostics/topics/cm-alerts-script.html
                # Select the fields that are important or of interest.
                alert_content += "occur time: " + error['body']['alert']['timestamp']['iso8601']
                alert_content += ", alert summary: " + error['body']['alert']['attributes']['ALERT_SUMMARY'][0]
                alert_content += ", alert detail: " + error['body']['alert']['content']
                sent_ding_talk(access_token, alert_content)

3. Configure settings on the CM interface

Navigate to Administration -> Alerts -> Custom Alert Script.

Set the alert script path to /opt/cloudera/alert/alert.py. After the configuration is complete, click Save Changes in the lower-right corner.
Click Restart this Alert Publisher in the upper-right corner.
After the restart, click Send Test Alert in the upper-right corner.
If the configuration is successful, you will receive the alert information in your DingTalk group.