Configure monitoring and alerts for fully managed services - ACK deployment solution

Step 1: Configure Managed Service for Prometheus for ACK

By default, Prometheus does not support metric forwarding. You must apply additional configuration in Managed Service for Prometheus to forward application monitoring metrics to specific users.

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "arms:GetPrometheusApiToken"
      ],
      "Resource": "*"
    }
  ]
}

Step 2: Configure the ROS template

1. Create a namespace named after the service instance ID and add tenant labels to it.

   Note

   The keys for the tenant labels are tenant_userid, tenant_clusterid, tenant_token, and tenant_cloudproductcode. The ROS template uses placeholders for these values. The namespace itself is named using the {{ serviceInstanceId }} placeholder, which is replaced with the ID of the created service instance.

   ClusterNameSpaceApplication:
       Type: ALIYUN::CS::ClusterApplication
       Properties:
         YamlContent:
           Fn::Sub:
             - |
               apiVersion: v1
               kind: Namespace
               metadata:
                 name: '${Name}'
                 labels:
                   tenant_userid: '{{ aliUid }}'
                   tenant_clusterid: '{{ tenantClusterId }}'
                   tenant_token: '{{ tenantToken }}'
                   tenant_cloudproductcode: '{{ tenantCloudProductCode }}'
             - Name: '{{ serviceInstanceId }}'
         ClusterId:
           Fn::If:
             - Condition: CreateACKCondition
             - Ref: ManagedKubernetesCluster
             - Ref: ClusterId

   Important

   Deploy the service provider application in the namespace that you create in the preceding step to ensure that application monitoring metrics are routed to the correct tenant.

2. (Optional) In the single-tenant fully managed service scenario, use the AckArmsConfig module in the ROS template to configure Managed Service for Prometheus for each newly created ACK cluster.

3. (Optional) If you want to collect custom metrics, expose them over an HTTP port or by using an exporter. The Service that exposes metrics must also be named {{ serviceInstanceId }}, and you must define a ServiceMonitor for service discovery. The following example shows the Service and ServiceMonitor used to collect MySQL metrics with mysqld-exporter.

   apiVersion: v1
   kind: Service
   metadata:
     name: {{ serviceInstanceId }}
     labels:
       io.mysql.service: {{ serviceInstanceId }}
   spec:
     selector:
       app: mysql
     ports:
       - protocol: TCP
         port: 3306
         targetPort: 3306
         name: mysql
       - protocol: TCP
         port: 9104
         targetPort: 9104
         name: mysql-exporter
     type: LoadBalancer

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: prometheus-service-monitor
     annotations:
       arms.prometheus.io/discovery: 'true'
     labels:
       prometheus-service-monitor: prometheus-service-monitor
   spec:
     selector:
       matchLabels:
         io.mysql.service: {{ serviceInstanceId }}
     namespaceSelector:
       matchNames:
         - {{ serviceInstanceId }}
     endpoints:
       - port: mysql-exporter
         scheme: http
         path: /metrics
         interval: 10s
         scrapeTimeout: 10s

Step 3: Configure the Grafana dashboard

Each service instance supports only one dashboard. If you have multiple dashboards, combine them into a single dashboard. The dashboard must meet the following requirement:

Create a dashboard variable named Namespace and use it as a global filter for application monitoring metrics. Create the dashboard in Managed Service for Grafana and obtain the dashboard link.

Step 4: Configure product identifier and dashboard link

To use Managed Service for Prometheus with a multi-tenant fully managed service from Compute Nest, you must configure the product identifier and monitoring dashboard information.

1. Product identifier: This uniquely identifies the service in the monitoring system. For testing, use the shared product identifier cn-mariadb. This identifier is available for testing in the China (Hangzhou) and China (Hong Kong) regions.

   Note

   For production use, your service requires a dedicated product identifier. To obtain one, join the technical support group mentioned at the beginning of this topic for assistance.

2. Monitoring dashboard settings include a title and a Grafana dashboard link. These settings have a one-to-one mapping with a product identifier.

   Note

   This configuration currently requires manual setup. Please join the technical support group for assistance.

3. The following is the configuration for a sample service that uses the cn-mariadb product identifier. When using cn-mariadb, you can use the default monitoring dashboard configuration. The default monitoring dashboard title is MySQL Exporter Quickstart and Dashboard.

Step 5: Configure alerts

1. Go to the ARMS console for Prometheus Monitoring and create a Prometheus alert rule template. For more information, see Create a Prometheus alert rule template.

2. After creating the template, obtain the TemplateId from the network requests in the console as described below.

   Important

   The corresponding network request appears only after you apply a search filter.

   On the alert rule template page, use the static threshold drop-down list at the top to filter and search. A alert.json?action=AlertTemplateAction request then appears in the Network panel of your browser's DevTools. Click the request and view the Response tab. In the returned JSON, find the id field (for example, d9a9fb3b-47a9-4338-84f5-93240e03b4ee). This value is the required TemplateId.

3. In the ROS template, define and apply the alert rule template to the ACK cluster by using the ALIYUN::ARMS::ApplyAlertRuleTemplate resource. This creates the corresponding alert rules in the cluster. The following example shows a ROS template snippet.

   ROSTemplateFormatVersion: '2015-09-01'
   Description:
     en: ApplyAlertRule
     zh-cn: ApplyAlertRule
   Parameters:
     ClusterIds:
       AssociationPropertyMetadata:
         Parameter:
           Required: true
           Type: String
           Description:
             en: The ID of the Prometheus Instance.
       Description:
         en: The IDs list of Prometheus Instances.
       Default: Null
       MinLength: 1
       Required: false
       MaxLength: 100
       AssociationProperty: List[Parameter]
       Type: Json
     TemplateIds:
       AssociationPropertyMetadata:
         Parameter:
           Required: true
           Type: String
           Description:
             en: The ID of the Prometheus alert rule template.
       Description:
         en: The IDs list of Prometheus alert rule templates.
       Default: Null
       MinLength: 1
       Required: false
       MaxLength: 100
       AssociationProperty: List[Parameter]
       Type: Json
   Resources:
     ApplyAlertRuleTemplate:
       Type: ALIYUN::ARMS::ApplyAlertRuleTemplate
       Properties:
         ClusterIds:
           Ref: ClusterIds
         TemplateIds:
           Ref: TemplateIds

Step 6: View monitoring data

After the service instance is deployed, both the tenant and the service provider can view the monitoring dashboard on the service instance details page.

• Tenant view:

• Service provider view: