Semantic enrichment
This topic describes how to use the semantic enrichment feature in SLS to extract key information from logs.
Semantic enrichment overview
Logs are categorized as structured or unstructured. Different analysis methods are used for each type.
Structured logs: Have a clearly defined schema and data type. You can use a SQL compute engine to perform statistical operations on them.
Unstructured logs: Are typically long-form text, such as large language model (LLM) chat histories or agent interaction logs. Extracting valuable information from them often requires manual intervention.
The semantic enrichment feature in SLS uses the natural language processing capabilities of an LLM to extract key and valid information from unstructured logs.
Supported regions
Semantic enrichment is available only in the Chinese mainland.
Large language model (LLM) functions
ai_gen() function
Syntax
ai_gen(prompt varchar)or
ai_gen(prompt varchar, model varchar)Parameters
Parameter | Description |
prompt | The data provided to the large language model. |
model | The model to use. The following model is available:
|
Return value
row(
result varchar
error_msg varchar
)Field | Description |
result | The content generated by the LLM. |
error_msg | The error message. A null value indicates success. |
Best practices
LLMs have a limited context window, so you cannot input too much data. To make the best use of an LLM's context capability, provide high-quality context without duplicate content. The best practice is as follows:
First, perform clustering on the large volume of data. For more information about clustering functions, see the documentation on vector indexes.
For the clustering results:
If you want to call the LLM for each cluster separately, randomly select 10 to 20 samples from each cluster and assemble a separate prompt for each cluster.
If you want to call the LLM for all clusters at once, randomly select one or two samples from each cluster and merge them to form a single, large prompt.
Assemble the instruction and the data.
Call the `ai_gen` function.
Perform post-processing on the LLM output.
Example
* | set session enable_remote_functions=true;
with t1 as (select array[input,'' ] as x, "input__embedding__" as embd from log where input <> 'null' and input__embedding__ is not null limit 5000),
t2 as (select cluster(array_agg(embd),'kmeans','{
"n_clusters": "10",
"max_iter": "100",
"tol": "0.0001",
"metric":"cosine",
"init_mode": "kmeans++"
}') as cluster_res, array_agg(x) as xs from t1),
t3 as (select label,x from t2,unnest(cluster_res.assignments,xs) as t(label,x) ),
t4 as (select cast(label as bigint) as "cluster_id",array_join(slice(array_distinct(array_agg((x[1]))),1,5),concat(chr(10),'----',chr(10))) as "content" ,count(1) as pv from t3 group by label),
t5 as (select array_join(array_agg(content),concat(chr(10),'----',chr(10))) as content from t4),
xt1 as (
select
"content" as trans, -- Data to be processed
'{{query}}' as targets, -- Placeholder in the evaluation template
'{"model":"<QWEN_MODEL>","input":{"messages":[{"role":"system","content":"<SYSTEM_PROMPT>"},{"role":"user","content":"<USER_PROMPT>"}]}}' as body_template, -- Body template for accessing Model Studio
cast('```{{query}}```. Categorize the task scenarios in the text above, such as NL2SQL scenarios.' as varchar) as eval_prompt -- Evaluation template that contains instructions and placeholders
FROM t4
),
xt1_1 as (
select
body_template,
eval_prompt,
replace(eval_prompt, targets, trans) as eval_content,
trans
FROM xt1
),
xt2 as (
select
body_template,
eval_content,
trans
FROM xt1_1
),
xt3 as (
select
trans as oldeval,
body_template,
replace(
replace(
replace(
replace(
replace(
replace(replace(eval_content, chr(92), '\\'), '"', '\"'),
chr(8),
'\b'
),
chr(12),
'\f'
),
chr(10),
'\n'
),
chr(13),
'\r'
),
chr(9),
'\t'
) as eval,
trans
FROM xt2
),
xt4 as (
select
replace(
replace(
replace(body_template, '<QWEN_MODEL>', 'qwen-max'),
'<SYSTEM_PROMPT>',
'You are a helpful assistant.'
),
'<USER_PROMPT>',
eval
) as body,
oldeval,
trans
FROM xt3
),
xt5 as (
select
ai_gen(
body
) as response,
body,
oldeval,
trans
FROM xt4
),
xt6 as (
select
oldeval,
body,
response.result as res,
trans
FROM xt5
),
xt7 as (select
trans as "original_text",
res as "summary_information"
FROM xt6)
select * from xt7Example breakdown:
Read the embedding column.
select array[input,'' ] as x, "input__embedding__" as embdPerform clustering on the embeddings.
select cluster(array_agg(embd),'kmeans','{ "n_clusters": "10", "max_iter": "100", "tol": "0.0001", "metric":"cosine", "init_mode": "kmeans++" }') as cluster_res, array_agg(x) as xs from t1Assemble the prompt.
select "content" as trans, -- Data to be processed '{{query}}' as targets, -- Placeholder in the evaluation template '{"model":"<QWEN_MODEL>","input":{"messages":[{"role":"system","content":"<SYSTEM_PROMPT>"},{"role":"user","content":"<USER_PROMPT>"}]}}' as body_template, -- Body template for accessing Model Studio cast('```{{query}}```. Categorize the task scenarios in the text above, such as NL2SQL scenarios.' as varchar) as eval_prompt -- Evaluation template that contains instructions and placeholders FROM t4Escape the prompt content. This step is critical for escaping invisible characters.
select trans as oldeval, body_template, replace( replace( replace( replace( replace( replace(replace(eval_content, chr(92), '\\'), '"', '\"'), chr(8), '\b' ), chr(12), '\f' ), chr(10), '\n' ), chr(13), '\r' ), chr(9), '\t' ) as eval, trans FROM xt2Call the LLM.
select ai_gen( body ) as response, body, oldeval, trans FROM xt4Retrieve the LLM processing result.
select oldeval, body, response.result as res, trans FROM xt5Extract the LLM processing result.
select trans as "original_text", res as "summary_information" FROM xt6