How it works

General monitoring logic

• Data source definition: builds a basic metadata model, which is the foundation of the monitoring system. For example, applications, deployed instances, data center information, and unitized information are defined. In the SOFAStack scenario, this part of information must be automatically fetched from PaaS.

• Data collection: collects monitoring data by using the Agent installed on the machine. The data can be logs or metrics that you obtain through various methods.

• Data cleansing: parses the collected unstructured data into structured data that the monitoring system can process. The parsing capability depends on the rule that you preset in the monitoring system, such as a log parsing rule and a Prometheus protocol rule.

  Assume that the following sample business log is obtained. The RMS service can calculate the number of transactions created for each commodity every minute.

    2012-11-11 11:11:11,2950211004,clothes,transaction,created,7,Y,
    2012-11-11 11:11:12,2950211005,clothes,transaction,created,8,Y,
    2012-11-11 11:11:13,2950211006,foods,transaction,created,9,Y,
    2012-11-11 11:11:14,2950211007,home design,transaction,created,11,Y,
    2012-11-11 11:11:15,2950211008,foods,transaction,created,7,N,
    ...

  Note

  The log is in the format of log printing time,order id,product name,service name,method name,time consumed,result.

• Data statistics: collects statistics of the structured data from the massive amounts of clusters faced by the monitoring system. Common statistical methods include calculating the number of rows, the sum, the mean value, the largest value, and the smallest value.

• Alert configuration: defines various alert rules for statistical data, such as an absolute value, a year-over-year comparison rule, a period-over-period comparison rule, and a sum for the last N minutes. RMS allows you to set multiple rules and logical operations at a time.

How it works

RMS deploys Agents on all applications and hosts to collect various metrics, logs, traces, and other data of the applications and hosts. Then, RMS stores the monitoring data in CeresDB and performs data analysis and calculation in Pontus Spark to display the results and send alert notifications in different product scenarios.

How RMS works

Collection layer

DaemonSet Agent, a log collection Agent, also known as the Pontus Agent

• The DaemonSet Agent collects all data, including raw log data and metric data.

• The DaemonSet Agent provides plug-in capabilities and unified local scheduling. It pulls up plug-ins, parses data, and then performs aggregate calculations. Then, it stores the results in the local cache and waits for a Vessel to pull the data.

• The DaemonSet Agent provides one-time tasks to execute for performance analysis.

• It communicates with the Register upon startup. After communication, the Agent retrieves all the configurations required for running, including the runtime collection configurations and Vessel connection configurations.

Register, a registry center

• The Register maintains network heartbeats with Agents and collects the status and version information of all Agents.

• The Register generates and delivers configurations of Agents and Vessels, that is, manages all collected configurations.

• The Register manages the connections and sessions between Agents and Vessels. It detects loads of the Vessel cluster and notifies an Agent of the address of a Vessel that the Agent can connect to.

Vessel, a traffic scrubbing component

• Abstractly, a Vessel is an independently deployed Agent that obtains logs or other raw data by using the remote input method.

• One Vessel instance can host one or more Agents, that is, collect data from one or more Agents.

• A Vessel can convert unstructured data into structured data based on parsing rules and return structured data. This process is called data cleansing.

• All upper-layer components, including components at the computing layer and the tracing layer, depend on Vessels to obtain data.

Computing layer

DimServer, the dimension data service, also known as the metadata service

• The DimServer reads and writes dimension data. The business semantics are not inherent at the current layer but are injected from the upper product layer.

• Both the computing and collection layers need dimension data, that is, metadata. They implement data communication based on the Java Client provided by the DimServer.

• To adapt to all dimension data structures, the DimServer converts dimension data into tables. That is, the DimServer uses multiple wide tables to process all possible metadata table structures. This type of wide table is called a dimension table.

Pontus Console, the console of the monitoring data platform

• The Pontus Console manages internal O&M, configures the system, and provides configuration APIs for products.

• The Pontus Console provides standard data service APIs for users to perform operations on dimension tables and metric tables.

Scheduler, a global task scheduler

• The Scheduler is independent of middleware. It provides global task scheduling to monitor the internal system, achieving a functional self-closed loop.

• The Scheduler regularly generates computing tasks based on configuration information and then distributes and monitors the computing tasks.

• The Scheduler can be deployed across multiple data centers and remain highly available based on databases.

Pontus Spark, a distributed computing engine

• The Pontus Spark is a powerful Spark computing cluster that receives computing configurations from the product layer and performs offline computing and statistical collection on data.

• The Pontus Spark obtains a Vessel address from the Register and obtains required monitoring data from the Vessel.

Pontus Alarm, a monitoring alert component, also known as the Alarm

• The Pontus Alarm receives the alerts exported after the monitoring data is calculated, and then pushes the alerts outward by using SMS messages, emails, and DingTalk.

• In output modes of the SOFAStack, alert channels are provided by the Gotone service.

DST, the distributed tracing service, also known as the Monitor Gateway

• Distributed Service Tracing (DST) processes distributed traces.

• DST collects trace information in three methods: active log collection, Log Service (SLS)-based log collection, and tracing log reporting.

Data layer

The data layer is borne by the time series database CeresDB. CeresDB is a time series database developed by Ant Group. The time series engine is a distributed database that stores and manages time series data. It provides various features for time series, including high-performance read and write, pre-processing, and visual query.

Product layer

The product layer, where the monitorprod application resides, hosts all the pages, user interaction, and configuration logic of RMS. This layer is the portal open to monitoring users.

The product layer is connected to the user permission system to read the identity information of users. In addition, the product layer is connected to PaaS metadata to synchronize metadata to the DimServer.