DocsICP FilingConsole
官方文档
首页

How it works

更新时间：
复制 MD 格式
产品详情
我的收藏

The data transformation feature in Simple Log Service uses consumer groups to consume and process log data. You can orchestrate more than 200 built-in functions to create transformation rules. This document explains how data is scheduled during transformation and how the rules engine works.

Scheduling basics

The data transformation feature of Simple Log Service uses a consumer group to consume log data from the source Logstore as a stream. It processes each log using a transformation rule and then outputs the result.

  • Scheduling mechanism

    For each transformation rule, the scheduler starts one or more running instances. Each instance acts as a consumer for one or more shards in the source Logstore. The scheduler determines the number of parallel instances based on their memory and CPU usage. The maximum number of instances is equal to the number of shards in the source Logstore.

  • Running instance

    A running instance reads source logs from its assigned shards according to your configuration. It processes the data using the transformation rule and outputs the result to the destination Logstore. Transformation rules can use external resources to enrich log data. The instance uses the consumer group mechanism to save its consumption position in a shard. This ensures that if the instance stops unexpectedly, it can resume consumption from the last saved breakpoint.

  • Task stop

    • If you do not configure an end time for the task, the running instances continue to run by default, and the data transformation job does not stop.

    • If you configure an end time for the task, the running instances process all logs received up to that time. Then, the instances automatically exit and the data transformation job stops.

    • If a job stops for any reason and is restarted, it resumes consumption from the last saved position by default.

Rules engine basics: Basic operations

You can use the built-in functions provided by the SLS DSL to write transformation rules. Each function represents a transformation step, and the rules engine executes the functions in sequence. 

e_set("log_type", "access_log")
e_drop_fields("__action")
e_if(e_search("ret: pass"), e_set("result", "pass"))
e_if(e_search("ret: unknown"), DROP)

The following figure shows the logic.

  • Basic logic

    Each event function in a rule is executed in sequence. Each function processes and modifies an event before passing the modified event to the next function.

    For example, e_set("log_type", "access_log") adds a field named log_type with the value access_log to each event. The next function then receives the updated event, which now includes the log_type field.

  • Conditional logic

    Steps can have conditions. If an event does not meet a condition, it skips that step's operation.

    For example, the function e_if(e_search("ret: pass"), e_set("result", "pass")) first checks if the ret field contains pass. If the condition is not met, no action is taken. If the condition is met, the function sets the value of the result field to pass.

  • Stop processing

    If a step returns zero events, the event is considered deleted.

    For example, e_if(e_search("ret: unknown"), DROP) discards events where the value of the ret field is unknown. After an event is discarded, no subsequent operations are performed on it, and processing begins on the next event.

Rules engine basics: Output, copy, and split

The rules engine also supports copying, outputting, and splitting events. 

e_coutput("archive_Logstore") )
e_split("log_type")
e_if(e_search("log_type: alert"), e_output("alert_Logstore") )
e_set("result", "pass")

Assume the source log to be processed is as follows: 

log_type: access,alert
content: admin login to database.

The following figure shows the logic.规则引擎原理逻辑图

  • Output event

    An output operation can be considered a special type of stop processing. For example, in Step 3, if an event has a log_type field with the value alert, the e_output("alert_Logstore") function is called. This function outputs the event to the specified destination and then deletes the event. Subsequent operations are not performed on this event.

  • Copy and output event

    The e_coutput function copies the current event and outputs the copy. The original event continues to the next processing step. For example, in Step 1, all received events are copied and then output to the archive_Logstore destination.

  • Split and parallel processing

    In Step 2, if the log_type field contains the values access and alert, e_split("log_type") splits the event into two separate events based on the values in the log_type field. The two new events are identical except for the log_type field, which will have the value access in one event and alert in the other. Each new event then continues through the subsequent steps independently.

    After the split, each resulting event proceeds through the subsequent steps independently.

Previous：TermsNext：Limits