how to use spl statements to extract string key-value pairs-Simple Log Service(SLS)-阿里云帮助中心

You can use SPL statements to extract dynamic key-value pairs from strings, with options to transform keys and values during extraction.

Keyword extraction

Use the parse-kv instruction to extract keywords and values from delimiter-separated key-value pairs. The following example extracts from a log in the format k1: q=asd&a=1&b=2&1=3.

Raw log
```
k1: q=asd&a=1&b=2&__1__=3
```
SPL statement
```
* | parse-kv k1,'&','='
```

Result

k1: q=asd&a=1&b=2&__1__=3
q: asd
a: 1
b: 2
__1__: 3

Value extraction

For logs with clear key-value identifiers such as a=b or a="cxxx", use the parse-kv instruction to extract values.

Raw log

content1:  k1="helloworld",the change world, k2="good"

To exclude non-key-value content such as the change world, use the following SPL statement:
```
* | parse-kv content1,',','='
```

Result

content1:  k1="helloworld",the change world, k2="good"
k1: "helloworld"
k2: "good"

Keyword transformation

Add a prefix to extracted keys by using the prefix="" parameter with the parse-kv instruction.

Raw log
```
k1: q=asd&a=1&b=2
```

SPL statement

* | parse-kv -prefix='start_' k1,'&','='

Result

k1: q=asd&a=1&b=2
start_a: 1
start_b: 2
start_q: asd

Value transformation

If a value contains quotation marks, such as k1:"v1"abc", the parse-kv instruction correctly extracts the quoted value.

Raw log

"""
The \ here is a regular character, not an escape character
"""
content2:  k1:"v1\"abc", k2:"v2", k3:"v3"

SPL statement
```
* | parse-kv content2,',', ':'
```

Result

content2: k1:"v1\"abc", k2:"v2", k3: "v3"
k1: "v1\"abc"
k2: "v2"
k3: "v3"

Customer use cases

The following example extracts URL components from a website access log and expands the query parameters into separate fields.

Requirements
- Requirement 1: Parse the log to extract fields such as proto, domain, and param.
- Requirement 2: Expand the key-value pairs in param.

Raw log

__source__:  10.43.xx.xx
__tag__:__client_ip__:  12.120.xx.xx
__tag__:__receive_time__:  1563517113
__topic__:
request:  https://yz.m.sm.cn/video/getlist/s?ver=3.2.3&app_type=supplier&os=Android8.1.0

SPL solution

* | parse-regexp request, '([^:]+)://([^/]+)(.+)' as uri_proto, uri_domain, uri_param | parse-regexp uri_param, '([^?]*)\?(.*)' as uri_path,uri_query | parse-kv uri_query,'&','='

Detailed rules and results

Use the parse-kv instruction to parse the request field.

Use unnamed capture groups

* | parse-regexp request, '([^:]+)://([^/]+)(.+)' as uri_proto, uri_domain, uri_param

Result

uri_proto:  https
uri_domain:  yz.m.sm.cn
uri_param:  /video/getlist/s?ver=3.2.3&app_type=supplier&os=Android8.1.0

Use the parse-regexp instruction to parse the uri_param field.

Use unnamed capture groups

* | parse-regexp uri_param, '([^?]*)\?(.*)' as uri_path,uri_query

Result

uri_path:  /video/getlist/s
uri_query:  ver=3.2.3&app_type=supplier&os=Android8.1.0

Extract fields from uri_param.

Use unnamed capture groups
```
* | parse-kv uri_query,'&','='
```

Result

ver:  3.2.3
app_type:  supplier
os:  Android8.1.0

Preview of the processed log

__source__:  10.43.xx.xx
__tag__:__client_ip__:  12.120.xx.xx
__tag__:__receive_time__:  1563517113
__topic__:
request:  https://yz.m.sm.cn/video/getlist/s?ver=3.2.3&app_type=supplier&os=Android8.1.0
uri_domain:  yz.m.sm.cn
uri_path:  /video/getlist/s
uri_proto:  https
uri_query:  ver=3.2.3&app_type=supplier&os=Android8.1.0
app_type:  supplier
os:  Android8.1.0
ver:  3.2.3

To parse only the request parameters without extracting the full URL structure, use the parse-kv instruction directly on the request field. For example:

* | parse-kv uri_query,'&','='

Processed log:

__source__:  10.43.xx.xx
__tag__:__client_ip__:  12.120.xx.xx
__tag__:__receive_time__:  1563517113
__topic__:
request:  https://yz.m.sm.cn/video/getlist/s?ver=3.2.3&app_type=supplier&os=Android8.1.0
app_type:  supplier
os:  Android8.1.0
ver:  3.2.3

上一篇: Parse Java error logs 下一篇: Parse fields from complex logs