Basic SPL syntax

SPL overview

SPL extracts structured information, manipulates fields, and filters raw data. SLS chains multiple pipeline levels: the first applies index filter conditions, and subsequent levels run SPL instructions to produce the final output. SQL users can compare approaches in Comparison of use cases for SPL and SQL.

How it works

SPL integrates with Logtail data collection, ingest processor, rule-based data consumption, data transformation (new version), and Scan-based query.

Note

For more information about the SPL functions in each use case, see General reference.

SPL-SPL Doc - ch.jpeg

Limitations

Category	Limit	Logtail data collection	Write Processor	Real-time consumption	Data Transformation (New)	Scan query
SPL complexity	Number of script pipeline levels	16	16	16	16	16
SPL complexity	Script length	64 KB	64 KB	10 KB	10 KB	64 KB
SPL runtime	Runtime memory size Important Resolve issues using Fault handling.	50 MB	1 GB	1 GB	1 GB	2 GB
SPL runtime	Runtime timeout Important Resolve issues using Fault handling.	1 second	5 seconds	5 seconds	5 seconds	2 seconds

SPL syntax

SPL statements

SPL statements chain multi-level data processing steps, connected by pipelines (|) and terminated by semicolons (;).

Syntax

<data-source> | <spl-expr> | <spl-expr> ;

Parameters

Parameter	Description	Example
data-source	An SLS Logstore or an SPL-defined dataset. For more information about SPL data sources in different scenarios, see General reference.	`* \| project status, body;`
\|	The SPL pipeline operator. Passes the output of the preceding instruction as input to the next.
spl-expr	An SPL instruction expression. For more information, see SPL syntax.

Data source

Data source definitions vary by feature type in SLS.

Feature type	Input from Logstore index filtering	Case-sensitive field names	Full-text field __line__	Best practices
Logtail collection	Not supported. Use `` to represent all raw Logtail data as input. Example: ` \| parse-json content`	Sensitive	Not supported	Use SPL to collect text logs
Ingest Processor	Not supported. Use `*` to represent all raw written data as input.	Sensitive	Not supported	Use an Ingest Processor to process cloud product logs
Real-time consumption	Not supported. Use `` to represent all raw Logstore data as input. Example: ` \| where msg like '%wrong%'`	Sensitive	Not supported	Use an SDK to consume logs based on a Consum Processor (SPL) Use a consumer group to consume logs based on a Consum Processor (SPL) Use Flink SQL to filter rows and project columns based on SPL Use Flink SQL to perform weakly structured analysis based on SPL
Data transformation (new version)	Not supported. Use `*` to represent all raw Logstore data as input.	Sensitive	Not supported	Data transformation best practices
Scan and query	Supported. Index filtering runs first, then SPL processes the filtered results. Example: `error and msg:wrong \| project level, msg`	Not sensitive	Supported	Scan logs

SPL instruction expressions

For more information about the instructions supported by SPL, see SPL instructions and functions.

Instruction expression syntax

cmd -option=<option> -option ... <expression>, ... as <output>, ...

Parameters

Parameter	Description
cmd	The instruction name.
option	Optional. Runtime parameters for the instruction. Each name starts with a hyphen (`-`). Order is not fixed. Two types: Key-value parameter `-option=<option>`: Specified as a key-value pair. Switch parameter -option: Disabled by default. Add to enable.
expression	Required. The processing logic applied to the data source. No parameter name needed, but order must match the instruction definition. Expression types: SPL expression A field name or wildcard pattern, such as content. Enclose in double quotes ("") if the name contains characters other than [a-zA-Z_]. Example: `"__tag__:__path__"`. A field assignment expression. `level='INFO'` assigns a value directly. `msg=regex_extract(content, '\S+')` assigns a regex extraction result. SQL expression A data calculation expression. Example: `cast(status as int)>=500`.
output	Output fields from the processing result. Example: `\| parse-csv content as a, b, c`.

Syntax symbols

SPL uses the following syntax symbols.

Symbol	Description	Example
*	Placeholder for Logstore data used as SPL input.	Filter and classify access logs by status code and output the results. SPL statement -- Define the SPL processing result as the named dataset 'src' to use as input for subsequent SPL expressions. .let src = * \| where status=cast(status as BIGINT); -- Use the named dataset 'src' as input. Filter for data where the status field is 5xx. Define the result as the dataset 'err'. Do not output the result. .let err = $src \| where status >= 500 \| extend msg='ERR'; -- Use the named dataset 'src' as input. Filter for data where the status field is 2xx. Define the result as the dataset 'ok'. Do not output the result. .let ok = $src \| where status >= 200 and status < 300 \| extend msg='OK'; -- Output the named datasets 'err' and 'ok'. $err; $ok; Input data `# Entry 1 status: '200' body: 'this is a test' # Entry 2 status: '500' body: 'internal error' # Entry 3 status: '404' body: 'not found'` Output `# Entry 1: Dataset is 'err' status: '500' body: 'internal error' msg: 'ERR' # Entry 2: Dataset is 'ok' status: '200' body: 'this is a test' msg: 'OK'`
.	Indicates an SPL keyword when used as the first character of a statement.
\|	Introduces an SPL instruction expression. Syntax: `\| <spl-cmd> ...`.
;	Terminates an SPL statement. Optional for single statements or the last statement in a multi-statement block.
'...'	Encloses string constants.
"..."	Encloses field names and field name patterns.
--	Single-line comment.
/.../	Multi-line comment.
$	References a named dataset. Syntax: `$<dataset-name>`.

SPL data types

SPL supports the following log field data types.

Type category	Type name	Type description
Basic numeric types	BOOLEAN	Boolean.
	TINYINT	8-bit integer.
	SMALLINT	16-bit integer.
	INTEGER	32-bit integer.
	BIGINT	64-bit integer.
	HUGEINT	128-bit integer.
	REAL	32-bit floating-point.
	DOUBLE	64-bit floating-point.
	TIMESTAMP	UNIX timestamp with nanosecond precision.
	DATE	Date in YYYY-MM-DD format.
	VARCHAR	Variable-length character string.
	VARBINARY	Variable-length binary.
Structured numeric types	ARRAY	Array. Access elements with `[]`. Index starts from 1. For example, `* \| extend a = ARRAY[0, 1, 2] \| extend b = a[1]`.
Structured numeric types	MAP	Dictionary. Keys must be basic numeric types; values can be any type. Access elements with `[]`. For example, `* \| extend a = MAP(ARRAY['foo', 'bar'], ARRAY[0, 1]) \| extend b = a['foo']`.
JSON type	JSON	JSON.

For more information about data type conversions during SPL data processing, see General reference.