Processing plugins parse raw logs into structured data during collection.
Background
SLS provides two types of processing plugins: native and extended.
Native plugins: High-performance and suitable for most scenarios. Recommended as the default choice.
Extended plugins: Cover more use cases but consume additional resources. Use when native plugins cannot handle your log format.
Limitations
Performance limitations
Extended plugins consume more resources (primarily CPU). Adjust collector parameters as needed. Configuration management.
If the raw data rate exceeds 5 MB/s, avoid complex plugin combinations. Use an extended plugin for initial processing anddata transformation for more complex tasks.
Log collection limitations
Extended plugins process text logs in line mode. File-level metadata such as
__tag__:__path__and__topic__, is stored in each log.Extended plugins affect tag-related features:
Contextual query and LiveTail become unavailable unless you add an aggregators configuration.
The
__topic__field is renamed to__log_topic__. With an aggregators configuration, logs contain both the__topic__and__log_topic__fields. To remove the__log_topic__field, use the Field processing plugins plugin.Fields like
__tag__:__path__lose native field indexes. You must create a field index for them.
Plugin combination limitations
For Logtail versions earlier than 2.0:
You cannot combine native and extended plugins.
Native plugins support text log collection only, with these requirements:
The first plugin must be a Data Parsing plugin: Regex, Delimiter, JSON, NGINX, Apache, or IIS mode.
After the first plugin, you can add only one Time Parsing plugin, one Data Filtering plugin, and multiple Data Masking plugins.
In Logtail 2.0, place extended plugins after all native plugins.
Native plugin parameter combination limits
For native parsing plugins (Regex, JSON, Delimiter, NGINX, Apache, and IIS modes) in Logtail versions earlier than 2.0, only specific parameter combinations are valid. Other combinations may produce unpredictable results.
Upload only successfully parsed logs:
Clear the Keep original fields on parsing failure and Keep original fields on parsing success checkboxes.
Upload parsed logs on success and raw logs on failure:
Select the Keep original fields on parsing failure checkbox, clear the Keep original fields on parsing success checkbox, and set Renamed original field to
__raw__.On successful parsing, upload both the parsed log and the original log field. On parsing failure, upload the raw log.
For example, after successfully parsing a log containing
"content": "{"request_method":"GET", "request_time":"200"}", a new field is added to the parsed log. The name of this field is the value of Renamed original field (or the original field name if left empty), and the value of this new field is the original content:{"request_method":"GET", "request_time":"200"}.Select the Keep original fields on parsing failure and Keep original fields on parsing success checkboxes. Turn on the Advanced Parameters switch and enter
{"CopingRawLog":true}. For Logtail versions earlier than 2.0, only some parameter combinations are effective.
Processing plugin list
Native plugins
Plugin name | Description |
Data Parsing (Regex Mode) | |
Data Parsing (JSON Mode) | |
Data Parsing (Delimiter Mode) | |
Data Parsing (NGINX Mode) | |
Data Parsing (Apache Mode) | |
Data Parsing (IIS Mode) | |
Time Parsing | |
Data Filtering | |
Data Masking |
Extended plugins
Feature | Description |
Extract fields | |
Add fields | |
Drop fields | |
Rename fields | |
Encapsulate fields | Encapsulates fields into a JSON object. Field processing plugins. |
Expand JSON fields | |
Filter logs | Filters logs by matching field values against a regex. Data filtering plugin. |
Filters logs by matching field names against a regex. Data filtering plugin. | |
Extract log time | Parses the time field from raw logs and sets the log time. Time-processing plugins. |
Convert IP addresses | Converts IP addresses to geographic locations (country, province, city, coordinates). Data transformation plugins. |
Mask sensitive data | Replaces sensitive log data with a specified string or MD5 hash. Data masking and encryption plugins. |
Map field values | |
Encrypt fields | |
Encode and decode data | |
Convert logs to metrics | Converts logs to SLS metrics. Data transformation plugins. |
Convert logs to traces | Converts logs to SLS traces. Data transformation plugins. |
Add a plugin
Modify a configuration
Log in to the Simple Log Service (SLS) console.
In the Project list, click your project.
On the Log Storage > Logstores tab, click the > icon next to your Logstore, and then navigate to Data Collection > Logtail Configurations.
In the Logtail Configuration list, find your Logtail configuration and click Manage Logtail Configuration in the Actions column.
At the top of the page, click Edit. In the Processor Configurations section at the bottom of the page, add a plugin, and then click Save.
Create a configuration
Log in to the Simple Log Service (SLS) console.
On the right side of the page, click the Quick Data Import card.
In the Import Data dialog box, click a card and follow the wizard. In the Logtail Configuration step, add a plugin. Collect text logs from a host.
NotePlugin configuration is the same for new and existing Logtail configurations.