Data transformation plug-ins-Simple Log Service(SLS)-阿里云帮助中心

Data transformation plugins

Log Service provides the following data transformation plugins.

Parameter	Type	Description
Convert IP address	extension	Converts an IP address to a geographic location (country, region, city, latitude, and longitude).
Log to metric	extension	Converts logs to Log Service metrics.
Log to trace	extension	Converts logs to Log Service trace data.

Entry point

To use a Logtail plugin for log processing, add it when you create or modify a Logtail configuration. For more information, see Overview.

IP address conversion plugin

The processor_geoip plugin converts IP addresses in logs into geolocation data, such as country, province, city, longitude, and latitude.

Configuration

Note

The Logtail installation package does not include a GeoIP database, so you must manually download one to the server where Logtail is installed and configure it. We recommend that you download a database that is accurate to the City level. For more information, see MaxMind GeoLite2.
Ensure that the database is in the MMDB format.
Form configuration is available for text logs and container standard output.
JSON configuration is not available for text logs.

Form configuration

Parameters

Set the Processor Type to Convert IP Addresses.

Parameter	Description
Original Field	The source field containing the IP address to convert.
GeoIP Database Path	The absolute path to the GeoIP database. For example, `/user/data/GeoLite2-City_20180102/GeoLite2-City.mmdb`.
Retain Original Field	Keeps the source field in the processed log.
Report Original Field Missing Error	Reports an error if the source field is missing from the raw log.
Report Parsing Failure Error	Reports an error if the IP address is invalid or not found in the database.
Language	The language for location names. Default: zh-CN. Important Ensure that your GeoIP database supports the specified language.

Example
- Raw log
```
"source_ip" : "**.**.**.**"
```
- Logtail plugin configuration: Configure the processor with the following settings: Set Processor Type to Convert IP Address, specify source_ip for Source Field, and /user/data/GeoLite2-City_20180102/GeoLite2-City.mmdb for GeoIP Database Path. In the advanced settings, select Keep Source Field, Report Error on Missing Source Field, and Report Error on Parsing Failure. Set Language to zh-CN, and then click Confirm.
- Processing result
```
"source_ip" : "**.**.**.**"
"source_ip_province_" : "Zhejiang"
"source_ip_city_" : "China (Hangzhou)"
"source_ip_province_code_" : "ZJ"
"source_ip_country_code_" : "CN"
"source_ip_longitude_" : "120.********"
"source_ip_latitude_" : "30.********"
```

JSON configuration

Parameters

Set type to processor_geoip. The detail parameter is described in the following table.

Parameter	Type	Required	Description
SourceKey	String	Yes	The source field that contains the IP address to convert.
DBPath	String	Yes	The full path to the GeoIP database. For example, /user/data/GeoLite2-City_20180102/GeoLite2-City.mmdb.
NoKeyError	Boolean	No	Reports an error if the field specified by `SourceKey` is missing from the raw log. `true`: Report an error. `false` (default): Do not report an error.
NoMatchError	Boolean	No	Reports an error if the IP address is invalid or not found in the database. `true` (default): Report an error. `false`: Do not report an error.
KeepSource	Boolean	No	Keeps the source field in the processed log. `true` (default): Keep the source field. `false`: Do not keep the source field.
Language	String	No	The language for location names. Default: zh-CN. Important Ensure that your GeoIP database supports the specified language.

Example

Raw log
```
"source_ip" : "**.**.**.**"
```

Logtail plugin configuration

{
   "type": "processor_geoip",
    "detail": {
         "SourceKey": "source_ip",
         "NoKeyError": true,
         "NoMatchError": true,
         "KeepSource": true,
         "DBPath" : "/user/data/GeoLite2-City_20180102/GeoLite2-City.mmdb"
    }
}

Processing result

"source_ip" : "**.**.**.**"
"source_ip_city_" : "China (Hangzhou)"
"source_ip_province_code_" : "ZJ"
"source_ip_country_code_" : "CN"
"source_ip_longitude_" : "120.********"
"source_ip_latitude_" : "30.********"

Log to Metric processor

Use the processor_log_to_sls_metric plugin to convert logs into SLS metrics.

Configuration

Important

The processor_log_to_sls_metric plugin is supported in Logtail 1.7.1 and later versions.

Console

Set the Processor Type to Log to Metric. The following table describes the parameters.

Parameters

Parameter	Description
MetricTimeKey	The time field in the log to map to the `__time_nano__` field in the time series data. By default, the `__time__` field value is used. Ensure that the specified field contains a valid Unix timestamp. The supported units are seconds (10 digits), milliseconds (13 digits), microseconds (16 digits), and nanoseconds (19 digits).
MetricLabelKeys	The list of keys for the `__labels__` field. Keys must follow the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`. Values cannot contain a vertical bar (\|) or `#$#`. For more information, see Time series data (Metric). You cannot add the `__labels__` field to the MetricLabelKeys parameter. If the `__labels__` field exists in the source field, its value will be appended to the new `__labels__` field.
MetricValues	A map of log fields to metric names and values. In each key-value pair, the key specifies the log field for the metric name (`__name__`), and the value specifies the log field for the metric value (`__value__`). The metric name corresponds to the `__name__` field and must follow the regular expression `^[a-zA-Z_:][a-zA-Z0-9_:]*$`. The metric value corresponds to the `__value__` field, which must be of the Double type. For more information, see Time series data (metrics).
CustomMetricLabels	For the custom `__labels__` field, the Key must follow the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`, and the Value cannot contain a vertical bar (\|) or `#$#`. For more information, see Time Series Data (Metric).
IgnoreError	Whether to log an error when no logs match for conversion. Default: `false` (an error is logged).

Configuration example

Raw log

{
    "script_md5":"8c6aebe9****c27f",
    "remote_addr":"123.XX.XX.123",
    "remote_user":"-",
    "time_local":"15/Aug/2023:12:03:20",
    "method":"GET",
    "url":"/www.a.com",
    "protocol":"HTTP/1.1",
    "status":"404",
    "body_bytes_sent":"4146",
    "http_referer":"-",
    "http_user_agent":"curl/7.74.0",
    "http_x_forwarded_for":"-",
    "__labels__":"field1#$#field1|field2#$#field2|tag1#$#tag1|tag2#$#tag2",
    "body_bytes_sent_name":"body_bytes_sent",
    "status_name":"status",
    "httptime":"1692101000",
    "__time__":"1692101000"
}

Logtail plugin configuration

In the processor configuration pop-up window, set the Processor Type to Log to Metric. Set MetricTimeKey to httptime. Add url and method to MetricLabelKeys. In MetricValues, add two mappings: body_bytes_sent_name to body_bytes_sent, and status_name to status. In CustomMetricLabels, add a label with the key nginx and the value test. Click Confirm.

Processing result

First time series data record

{
    "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com",
    "__name__":"body_bytes_sent",
    "__value__":"4146",
    "__time_nano__":"1692101000000000000",
    "__time__":"1692101000"
}

Second time series data record

{
    "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com",
    "__name__":"status",
    "__value__":"404",
    "__time_nano__":"1692101000000000000",
    "__time__":"1692101000"
}

JSON

Set type to processor_log_to_sls_metric. The following table describes the parameters in the detail object.

Parameters

Parameter	Type	Required	Description
MetricTimeKey	String	No	The time field in the log to map to the `__time_nano__` field in the time series data. By default, the `__time__` field value is used. Ensure that the specified field contains a valid Unix timestamp. The supported units are seconds (10 digits), milliseconds (13 digits), microseconds (16 digits), and nanoseconds (19 digits).
MetricLabelKeys	Array of strings	Yes	The list of keys for the `__labels__` field. Keys must conform to the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`. Values cannot contain vertical bars (\|) or `#$#`. For more information, see Time series data (metrics). You cannot add the `__labels__` field to the MetricLabelKeys parameter. If the `__labels__` field exists in the source field, its value is appended to the new `__labels__` field.
MetricValues	Map	Yes	A map of log fields to metric names and values. In each key-value pair, the key specifies the log field for the metric name (`__name__`), and the value specifies the log field for the metric value (`__value__`). The metric name corresponds to the `__name__` field and must follow the regular expression `^[a-zA-Z_:][a-zA-Z0-9_:]*$`. The metric value corresponds to the `__value__` field and must be of the Double type. For more information, see Time series data (metrics).
CustomMetricLabels	Map	No	For the custom `__labels__` field, the Key must follow the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`, and the Value cannot contain a vertical bar (\|) or `#$#`. For more information, see Time series data (Metric).
IgnoreError	Boolean	No	Whether to log an error when no logs match. Default: `false` (an error is logged).

Configuration example

Raw log

{
    "script_md5":"8c6aebe9****c27f",
    "remote_addr":"123.XX.XX.123",
    "remote_user":"-",
    "time_local":"15/Aug/2023:12:03:20",
    "method":"GET",
    "url":"/www.a.com",
    "protocol":"HTTP/1.1",
    "status":"404",
    "body_bytes_sent":"4146",
    "http_referer":"-",
    "http_user_agent":"curl/7.74.0",
    "http_x_forwarded_for":"-",
    "__labels__":"field1#$#field1|field2#$#field2|tag1#$#tag1|tag2#$#tag2",
    "body_bytes_sent_name":"body_bytes_sent",
    "status_name":"status",
    "httptime":"1692101000",
    "__time__":"1692101000"
}

Logtail plugin configuration

{
    "processors":[
        {
            "detail":{
                "CustomMetricLabels":{
                    "nginx":"test"
                },
                "IgnoreError":false,
                "MetricLabelKeys":[
                    "url",
                    "method"
                ],
                "MetricTimeKey":"httptime",
                "MetricValues":{
                    "body_bytes_sent_name":"body_bytes_sent",
                    "status_name":"status"
                }
            },
            "type":"processor_log_to_sls_metric"
        }
    ]
}

Processing result

First time series data record

{
    "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com",
    "__name__":"body_bytes_sent",
    "__value__":"4146",
    "__time_nano__":"1692101000000000000",
    "__time__":"1692101000"
}

Second time series data record

{
    "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com",
    "__name__":"status",
    "__value__":"404",
    "__time_nano__":"1692101000000000000",
    "__time__":"1692101000"
}

Log-to-trace processor

Use the processor_otel_trace plugin to convert logs into SLS traces.

Configuration

Important

The processor_otel_trace processor is supported in Logtail 1.7.1 and later.

Form configuration

Set Processor Type to Log to Trace. The following table describes the parameters.

Parameters

Parameter	Description
SourceKey	The name of the source field.
Format	The output format. Valid values: protobuf json protojson
NoKeyError	If `true`, reports an error if the source field is missing from the log. The default is `false`.
TraceIDNeedDecode	If `true`, decodes the `TraceID` from Base64. The default is `false`. If you set `Format` to `protojson` and the `TraceID` is Base64-encoded, set this parameter to `true` to prevent conversion failures.
SpanIDNeedDecode	If `true`, decodes the `SpanID` from Base64. The default is `false`. If you set `Format` to `protojson` and the `SpanID` is Base64-encoded, set this parameter to `true` to prevent conversion failures.
ParentSpanIDNeedDecode	If `true`, decodes the `ParentSpanID` from Base64. The default is `false`. If you set `Format` to `protojson` and the `ParentSpanID` is Base64-encoded, set this parameter to `true` to prevent conversion failures.

Configuration example

Raw log

INFO OtlpJsonLoggingSpanExporter - {"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"io.opentelemetry.spring-webmvc-3.1","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"eecaa7e1b63c9f61","parentSpanId":"fa85fcee7fe0e0e8","name":"HelloWorldController.sayHello","kind":1,"startTimeUnixNano":"1696659118615748056","endTimeUnixNano":"1696659128701715998","attributes":[{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}}],"events":[],"links":[],"status":{}}]},{"scope":{"name":"io.opentelemetry.tomcat-7.0","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"fa85fcee7fe0e0e8","name":"GET /hello","kind":2,"startTimeUnixNano":"1696659118479360000","endTimeUnixNano":"1696659128703706165","attributes":[{"key":"http.status_code","value":{"intValue":"200"}},{"key":"user_agent.original","value":{"stringValue":"curl/7.84.0"}},{"key":"net.host.name","value":{"stringValue":"localhost"}},{"key":"http.response_content_length","value":{"intValue":"12"}},{"key":"http.target","value":{"stringValue":"/hello"}},{"key":"net.sock.peer.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.host.port","value":{"intValue":"18080"}},{"key":"net.sock.peer.port","value":{"intValue":"53649"}},{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}},{"key":"http.route","value":{"stringValue":"/hello"}},{"key":"net.sock.host.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.protocol.name","value":{"stringValue":"http"}},{"key":"net.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"http.method","value":{"stringValue":"GET"}}],"events":[],"links":[],"status":{}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.19.0"}

Logtail processor configuration

Set the processor type to log to trace, set SourceKey to content, set Format to protojson, and select NoKeyError. Leave TraceIDNeedDecode, SpanIDNeedDecode, and ParentSpanIDNeedDecode unselected, and then click Confirm.

Processing result

The following code shows a sample processing result.

1  10-07 14:12:08
attribute: {"http.method":"GET","http.response_content_length":"","http.route":"/hello","http.scheme":"http","http.status_code":"","http.target":"/hello","net.host.name":"localhost","net.host.port":"","net.protocol.name":"http","net.protocol.version":"1.1","net.sock.host.addr":"127.0.0.1","net.sock.peer.addr":"127.0.0.1","net.sock.peer.port":"","thread.id":"","thread.name":"http-nio-18080-exec-1","user_agent.original":"curl/7.84.0"}
duration: 10224346
end: 1696659128703706
host: B-7xxx xxx.local
kind: server
links: []
logs: []
name: GET /hello
otlp.name: io.opentelemetry.tomcat-7.0
otlp.version: 1.26.0-alpha
parentSpanID:
resource: {"host.arch":"x86_64","os.description":"Mac OS X 13.0.1","os.type":"darwin","process.command_args":"","process.executable.path":"/Library/Java/JavaVirtualMachines/jdk-20.jdk/Contents/Home/bin/java","process.pid":"","process.runtime.description":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 20.0.1+9-29","process.runtime.name":"Java(TM) SE Runtime Environment","process.runtime.version":"20.0.1+9-29","telemetry.auto.version":"1.26.0","telemetry.sdk.language":"java","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.26.0"}
service: unknown_service:java
spanID: 7dexxx xxxb47bc
start: 1696659118479360
statusCode: UNSET
statusMessage:
traceID: 6bb690xxx xxxff79
traceState:

JSON configuration

Set type to processor_otel_trace. The following table describes the parameters in the detail object.

Parameters

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the source field.
Format	String	Yes	The output format. Valid values are `protobuf`, `json`, and `protojson`.
NoKeyError	Boolean	No	If `true`, reports an error if the source field is missing from the log. The default is `false`.
TraceIDNeedDecode	Boolean	No	If `true`, decodes the `TraceID` from Base64. The default is `false`. If you set `Format` to `protojson` and the `TraceID` is Base64-encoded, set this parameter to `true` to prevent conversion failures.
SpanIDNeedDecode	Boolean	No	If `true`, decodes the `SpanID` from Base64. The default is `false`. If you set `Format` to `protojson` and the `SpanID` is Base64-encoded, set this parameter to `true` to prevent conversion failures.
ParentSpanIDNeedDecode	Boolean	No	If `true`, decodes the `ParentSpanID` from Base64. The default is `false`. If you set `Format` to `protojson` and the `ParentSpanID` is Base64-encoded, set this parameter to `true` to prevent conversion failures.

Configuration example

Raw log

INFO OtlpJsonLoggingSpanExporter - {"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"io.opentelemetry.spring-webmvc-3.1","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"eecaa7e1b63c9f61","parentSpanId":"fa85fcee7fe0e0e8","name":"HelloWorldController.sayHello","kind":1,"startTimeUnixNano":"1696659118615748056","endTimeUnixNano":"1696659128701715998","attributes":[{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}}],"events":[],"links":[],"status":{}}]},{"scope":{"name":"io.opentelemetry.tomcat-7.0","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"fa85fcee7fe0e0e8","name":"GET /hello","kind":2,"startTimeUnixNano":"1696659118479360000","endTimeUnixNano":"1696659128703706165","attributes":[{"key":"http.status_code","value":{"intValue":"200"}},{"key":"user_agent.original","value":{"stringValue":"curl/7.84.0"}},{"key":"net.host.name","value":{"stringValue":"localhost"}},{"key":"http.response_content_length","value":{"intValue":"12"}},{"key":"http.target","value":{"stringValue":"/hello"}},{"key":"net.sock.peer.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.host.port","value":{"intValue":"18080"}},{"key":"net.sock.peer.port","value":{"intValue":"53649"}},{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}},{"key":"http.route","value":{"stringValue":"/hello"}},{"key":"net.sock.host.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.protocol.name","value":{"stringValue":"http"}},{"key":"net.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"http.method","value":{"stringValue":"GET"}}],"events":[],"links":[],"status":{}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.19.0"}

Logtail processor configuration

{
  "processors": [
    {
      "type": "processor_split_log_regex",
      "detail": {
        "PreserveOthers": true,
        "SplitKey": "content",
        "SplitRegex": "INFO\\sOtlpJsonLoggingSpanExporter\\s.+"
      }
    },
    {
      "type": "processor_regex",
      "detail": {
        "SourceKey": "content",
        "Regex": "INFO\\sOtlpJsonLoggingSpanExporter\\s-(\\s.+)",
        "Keys": [
          "trace_data"
        ]
      }
    },
    {
      "detail": {
        "SourceKey": "trace_data",
        "Format": "protojson",
        "NoKeyError": true
      },
      "type": "processor_otel_trace"
    }
  ]
}

Processing result

The following code shows a sample processing result.

1  10-07 14:12:08
attribute: {"http.method":"GET","http.response_content_length":"","http.route":"/hello","http.scheme":"http","http.status_code":"","http.target":"/hello","net.host.name":"localhost","net.host.port":"","net.protocol.name":"http","net.protocol.version":"1.1","net.sock.host.addr":"127.0.0.1","net.sock.peer.addr":"127.0.0.1","net.sock.peer.port":"","thread.id":"","thread.name":"http-nio-18080-exec-1","user_agent.original":"curl/7.84.0"}
duration: 10224346
end: 1696659128703706
host: B-7xxx xxx.local
kind: server
links: []
logs: []
name: GET /hello
otlp.name: io.opentelemetry.tomcat-7.0
otlp.version: 1.26.0-alpha
parentSpanID:
resource: {"host.arch":"x86_64","os.description":"Mac OS X 13.0.1","os.type":"darwin","process.command_args":"","process.executable.path":"/Library/Java/JavaVirtualMachines/jdk-20.jdk/Contents/Home/bin/java","process.pid":"","process.runtime.description":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 20.0.1+9-29","process.runtime.name":"Java(TM) SE Runtime Environment","process.runtime.version":"20.0.1+9-29","telemetry.auto.version":"1.26.0","telemetry.sdk.language":"java","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.26.0"}
service: unknown_service:java
spanID: 7dexxx xxxb47bc
start: 1696659118479360
statusCode: UNSET
statusMessage:
traceID: 6bb690xxx xxxff79
traceState: