Process data during ingestion (ingest processor)

An ingest processor processes log data before it is written to a Logstore. Use an ingest processor to filter data, extract or add fields, and mask sensitive information.

How it works

Log data collected through Logtail, the Simple Log Service SDK, or web tracking passes through the ingest processor before being written to a Logstore. Processing runs within Simple Log Service and consumes no client-side resources.
The ingest processor, log query and analysis, and data transformation all support SPL syntax.

Use cases

Field extraction: Extract fields from raw logs using regular expressions, key-value pairs, or JSON parsing.
Field extension: Add new fields to raw logs.
Field deletion: Remove specific fields from raw logs.
Data masking: Mask sensitive information in raw logs.
Data filtering: Discard unwanted log entries.

Limitations

Item	Limit
Ingest processors per project	Up to 100 per project.
SPL statement length	Up to 4,000 characters per SPL statement.
Supported SPL commands	Row-level commands only. Aggregation and conditional logic are not supported.
Ingest processors per Logstore	One ingest processor per Logstore. All data written to an associated Logstore passes through the ingest processor.
Time for changes to take effect	Configuration changes, deletions, or Logstore association updates take effect within one minute.

Billing

Ingest processors are billed based on the computing resources consumed during data processing. The billing unit is OCU.

The Observability Capacity Unit (OCU) is a new billing unit introduced by Alibaba Cloud-native Observability. OCU usage is automatically calculated based on hourly resource consumption. On the China site, the public-cloud price is CNY 0.15 per OCU.

Billing for the compute-optimized features of SLS is transitioning to an OCU-based model that charges based on actual compute resource consumption. In CPU scenarios, one OCU is equivalent to approximately 0.5 CPU cores, 2 GB of memory, and 3,000 IOPS. SLS calculates separate OCU values based on the consumption of CPU cores, memory, and IOPS. The highest of these three values determines the final OCU count for billing.

Suppose your computing job consumes 1 CPU core, 2 GB of memory, and 3,000 IOPS. The job consumes 2 OCUs. During a one-hour metering period, the average OCU consumption can be estimated as follows: Processing 1 GB of data with an ingest processor consumes approximately 1/3 of an OCU; processing 1 GB of data with data transformation (new version) consumes approximately 1/3 of an OCU; and processing 1 GB of data with rule-based consumption consumes approximately 0.3 OCUs.

For OCU pricing information, see Billable items of the pay-by-ingested-data mode and Billable items of the pay-by-feature billing model.