Parsing security vendor logs into CTDR fields-阿里云帮助中心

Cloud Threat Detection and Response (CTDR) allows you to ingest logs from security vendors into the Security Center console for unified risk management. A parsing function is a specific rule used to standardize the fields from collected security vendor logs, which are stored in a dedicated CTDR Logstore. This document describes the parsing functions in CTDR to help you configure log parsing rules.

Types of parsing functions

Type	Description
direct mapping	Maps the value of an original field directly to a target field without any changes. For example, the value of Field A is mapped directly to Field B.
constant mapping	Assigns a fixed, constant value to a target field. This is useful when no corresponding original field exists. For example, the string "abc" is assigned to Field B.
dictionary mapping	Maps values from a specific log field to standard CTDR field values by using a predefined dictionary. For example, the standard CTDR field `alert_level` accepts the values 1 (Informational), 2 (Low), 3 (Medium), 4 (High), and 5 (Critical). To normalize a vendor's log, you can create a dictionary to map its alert level values to the CTDR standard. The mapping might look like this: `other`, `none`, `unknown`: 1 (Informational) `low`: 2 (Low) `medium`: 3 (Medium) `high`: 4 (High) After you configure the dictionary mapping, CTDR standardizes the alert levels from the security vendor logs.
function mapping	Processes a value from an original field by using a function and then maps the result to a target field.

Supported functions

Function	Syntax	Description	Parameters	Example
JSON_VALUE	`JSON_VALUE(content, path)`	Extracts data from a JSON object. Supports nested paths.	`content` (VARCHAR): The JSON object to parse, represented as a string. The string cannot contain the characters $, [], *, or .. `path` (VARCHAR): The path expression for parsing the JSON object.	JSON_VALUE(json, '$.passenger_name')
LOWER	`LOWER(A)`	Converts a string to lowercase.	`A` (VARCHAR): The string to convert.	LOWER(var1)
UPPER	`UPPER(A)`	Converts a string to uppercase.	`A` (VARCHAR): The string to convert.	UPPER(var1)
SUBSTRING	`SUBSTRING(a, start)` `SUBSTRING(a, start, len)`	Extracts a substring starting at the specified `start` position. If `len` is provided, it extracts a substring of that length. If `len` is not specified, it extracts to the end of the string. The `start` position is 1-based. A `start` value of 0 is treated as 1. If `start` is negative, the position is calculated from the end of the string.	`a` (VARCHAR): The source string. `len` (INT): The length of the substring to extract. `start` (INT): The starting position for the extraction.	SUBSTRING(str, 2) SUBSTRING(str, -2, 1)
KEYVALUE	`KEYVALUE( str, split1, split2, key_name)`	Parses a string `str` of key-value pairs by using two delimiters: `split1` separates the key-value pairs. `split2` separates the key from the value within each pair. The function retrieves the value corresponding to the specified `key_name`. Returns `NULL` if the `key_name` is not found or an error occurs.	`str` (VARCHAR): The string to parse. `split1` (VARCHAR): The delimiter for key-value pairs. If this parameter is null, whitespace is used. If the string contains multiple characters, each character acts as a delimiter. `split2` (VARCHAR): The delimiter for keys and values. When split2 is `null`, whitespace is used as the delimiter for keys and values. If the string contains multiple characters, each character acts as a delimiter. `key_name` (VARCHAR): The name of the key whose value you want to retrieve.	KEYVALUE(str, ';', ':', 'key1')
REGEXP_EXTRACT	`REGEXP_EXTRACT( str, pattern, index)`	Extracts a substring from `str` that matches the `index`-th capturing group of a regular expression `pattern`. The index is 1-based. Returns `null` if an input parameter is empty or the regular expression is invalid.	`str` (VARCHAR): The string to parse. `pattern` (VARCHAR): The regular expression. `index` (INT): Specifies which matching group to return. Note This function uses Java's regular expression engine. Therefore, you must write patterns according to Java syntax. For example, if you want to match a digit, in SQL, you should write `'\d'` instead of `\d`. This ensures the expression is correctly compiled. For more information, see the Java Pattern documentation.	REGEXP_EXTRACT(str1, 'foo(.*?)(bar)', 1)
DATE_FORMAT	`DATE_FORMAT(TIMESTAMP time, VARCHAR to_format)`	Formats a `TIMESTAMP` value as a string based on a specified format string. Returns `null` if any parameter is `null` or a parsing error occurs. The first parameter is the source string. The second parameter, from_format, is optional. It specifies the format of the source string. The default value is yyyy-MM-dd HH:mm:ss. The third parameter is the format for the returned date. The function returns the formatted date as a string. If any parameter is `null` or a parsing error occurs, the function returns `null`.	date: A date of the VARCHAR type. The default date format is yyyy-MM-dd HH:mm:ss. `time` (TIMESTAMP): The timestamp value to format. from_format: Specifies the input date format. `to_format` (VARCHAR): Specifies the output date format, for example, 'yyyy-MM-dd HH:mm:ss'.	DATE_FORMAT(datetime1, 'yyMMdd')
FROM_UNIXTIME	FROM_UNIXTIME(BIGINT unixtime, VARCHAR format)	Converts a Unix timestamp to a formatted date string. The function accepts two parameters: `unixtime`: A long integer timestamp in seconds. `format`: An optional string that specifies the date format. The default is `yyyy-MM-dd HH:mm:ss`. The function returns a VARCHAR string that represents the date and time. Returns `null` if `unixtime` or `format` is `null`, or if a formatting error occurs.	`unixtime` (BIGINT): The Unix timestamp in seconds. The function accepts only standard numerical Unix timestamps. An exception is thrown for non-integer or incorrectly formatted data. `format` (VARCHAR): Specifies the output date format.	FROM_UNIXTIME(unixtime1, 'MMdd-yyyy')
UNIX_TIMESTAMP	UNIX_TIMESTAMP(VARCHAR date, VARCHAR format)	Converts a date string to a Unix timestamp in seconds. It accepts two optional parameters: The first parameter is the date and time string to convert. The second parameter specifies the format of the date and time string, which defaults to yyyy-MM-dd HH:mm:ss. The function returns a long integer timestamp in seconds. If called with no arguments, this function returns the current timestamp, which is equivalent to the `NOW()` function. Returns `null` if a parameter is `null` or a parsing error occurs.	`timestamp` (TIMESTAMP): `date` (VARCHAR): The date string to convert. The default format is yyyy-MM-dd HH:mm:ss. `format` (VARCHAR): Defines the format of the date parameter.	UNIX_TIMESTAMP('2024-11-19', 'yyyy-MM-dd')
SPLIT_INDEX	SPLIT_INDEX(str, sep, index)	Splits the string `str` into segments by using `sep` as the delimiter and returns the segment at the specified `index`. If the segment at the specified `index` does not exist, this function returns `null`. The `index` is zero-based. If any parameter is `null`, this function returns `null`.	`str` (VARCHAR): The string to be split. `sep` (VARCHAR): The character or symbol used as a delimiter. `index` (INT): The position of the substring to return. 0 represents the first element.	SPLIT_INDEX('a,b,c', ',', 0)
CONCAT	CONCAT(string1, string2, ...)	Concatenates two or more `VARCHAR` strings into a single string. If any argument is `null`, it is ignored during concatenation. All arguments must be `VARCHAR` strings; other data types, such as `INT` and `BIGINT`, are not supported.	`var1` (VARCHAR): A string value. `var2` (VARCHAR): A string value.	CONCAT(var1, '-', var2)
REGEXP_REPLACE	REGEXP_REPLACE( str, pattern, replacement)	Replaces all substrings in `str` that match the regular expression `pattern` with the `replacement` string and returns the result. Returns `null` if an input parameter is empty or the regular expression is invalid.	`str` (VARCHAR): The string to search and modify. `pattern` (VARCHAR): The regular expression defining the substrings to be replaced. `replacement` (VARCHAR): The string to use as a replacement. If `replacement` is empty, the matched substrings are deleted. Note This function uses Java's regular expression engine. Therefore, you must write patterns according to Java syntax. For example, if you want to match a digit, in SQL, you should write `'\d'` instead of `\d`. This ensures the expression is correctly compiled. For more information, see the Java Pattern documentation.	REGEXP_REPLACE(str1, '(\d+)', '')

Manage mapping dictionaries

Add a mapping dictionary

Log on to the Security Center console.
In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner, select the region where your assets are located: Chinese Mainland or Outside Chinese Mainland.
On the Service Integration page, click Log Parsing Rule in the upper-right corner.
On the Log Parsing Rule page, click Manage Mapping Dictionary in the upper-right corner.
In the Manage Mapping Dictionary panel, click Add Dictionary.

In the Add Mapping Dictionary panel, configure the parameters and click OK.

Parameter	Description
Dictionary Name	Enter a name for the dictionary.
Target Standard Field	Select the standard target field to which the dictionary applies.
Remarks	Enter remarks for the dictionary.
Configure Mappings	Set the mapping between original field values and target field values. You can click Add Mapping to add multiple mappings.

Edit a mapping dictionary

Editing a mapping dictionary does not affect existing rules that use it. Changes take effect only after you modify and save a rule, which causes it to fetch the latest dictionary version.

Log on to the Security Center console.
In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner, select the region where your assets are located: Chinese Mainland or Outside Chinese Mainland.
On the Service Integration page, click Log Parsing Rule in the upper-right corner.
On the Log Parsing Rule page, click Manage Mapping Dictionary in the upper-right corner.
In the Actions column of the target dictionary, click Edit.
In the Edit Mapping Dictionary panel, update the dictionary content and click OK.

Delete a mapping dictionary

Deleting a mapping dictionary does not affect existing rules that already use it.

Log on to the Security Center console.
In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner, select the region where your assets are located: Chinese Mainland or Outside Chinese Mainland.
On the Service Integration page, click Log Parsing Rule in the upper-right corner.
On the Log Parsing Rule page, click Manage Mapping Dictionary in the upper-right corner.
In the Actions column of the target dictionary, click Delete.
In the Delete Confirmation dialog box, click OK.