AI observability

Running attributes

Plugin execution stage: default stage. Plugin execution priority: 200.

Configuration description

By default, this plugin handles requests that conform to the OpenAI protocol. Without any special configuration, the plugin provides the following basic observable values:

• Metrics: Provides metrics such as input tokens, output tokens, response time (RT) for the first token in streaming requests, and total request RT. You can observe these metrics at the gateway, route, service, and model dimensions.

• Logs: Provides fields such as input_token, output_token, model, llm_service_duration, and llm_first_token_duration.

You can also extend the observable values through configuration:

Attribute configuration:

The following list describes the valid values for value_source:

• fixed_value: A static value.

• request_header: Fetches the attribute value from the HTTP request header. Set `value` to the header key.

• request_body: Fetches the attribute value from the request body. Set `value` to a gjson JSONPath expression.

• response_header: Fetches the attribute value from the HTTP response header. Set `value` to the header key.

• response_body: Fetches the attribute value from the response body. Set `value` to a gjson JSONPath expression.

• response_streaming_body: Fetches the attribute value from the streaming response body. Set `value` to a gjson JSONPath expression.

When value_source is response_streaming_body, you must configure the rule parameter to specify how to fetch the value from the streaming body. The valid values are as follows:

• first: Fetches the value from the first valid chunk.

• replace: Fetches the value from the last valid chunk.

• append: Concatenates the values from multiple valid chunks. This can be used to obtain the full answer content.

Configuration examples

To record statistics in the gateway access log, modify the `log_format` by adding a new field. For example:

'{"ai_log":"%FILTER_STATE(wasm.ai_log:PLAIN)%"}'

route_upstream_model_metric_input_token{ai_route="llm",ai_cluster="outbound|443||qwen.dns",ai_model="qwen-turbo"} 10
route_upstream_model_metric_llm_duration_count{ai_route="llm",ai_cluster="outbound|443||qwen.dns",ai_model="qwen-turbo"} 1
route_upstream_model_metric_llm_first_token_duration{ai_route="llm",ai_cluster="outbound|443||qwen.dns",ai_model="qwen-turbo"} 309
route_upstream_model_metric_llm_service_duration{ai_route="llm",ai_cluster="outbound|443||qwen.dns",ai_model="qwen-turbo"} 1955
route_upstream_model_metric_output_token{ai_route="llm",ai_cluster="outbound|443||qwen.dns",ai_model="qwen-turbo"} 69

{
  "ai_log":"{\"model\":\"qwen-turbo\",\"input_token\":\"10\",\"output_token\":\"69\",\"llm_first_token_duration\":\"309\",\"llm_service_duration\":\"1955\"}"
}

attributes:
  - key: model
    value_source: response_body
    value: usage.models.0.model_id
    apply_to_log: true
    apply_to_span: false
  - key: input_token
    value_source: response_body
    value: usage.models.0.input_tokens
    apply_to_log: true
    apply_to_span: false
  - key: output_token
    value_source: response_body
    value: usage.models.0.output_tokens
    apply_to_log: true
    apply_to_span: false

route_upstream_model_metric_input_token{ai_route="bailian",ai_cluster="qwen",ai_model="qwen-max"} 343
route_upstream_model_metric_output_token{ai_route="bailian",ai_cluster="qwen",ai_model="qwen-max"} 153
route_upstream_model_metric_llm_service_duration{ai_route="bailian",ai_cluster="qwen",ai_model="qwen-max"} 3725
route_upstream_model_metric_llm_duration_count{ai_route="bailian",ai_cluster="qwen",ai_model="qwen-max"} 1

{
  "ai_log": "{\"model\":\"qwen-max\",\"input_token\":\"343\",\"output_token\":\"153\",\"llm_service_duration\":\"19110\"}"  
}

attributes:
  - key: consumer # Record the consumer for authentication
    value_source: request_header
    value: x-mse-consumer
    apply_to_log: true

attributes:
  - key: question # Record the question
    value_source: request_body
    value: messages.@reverse.0.content
    apply_to_log: true
  - key: answer   # Extract the Large Language Model (LLM) answer from a streaming response
    value_source: response_streaming_body
    value: choices.0.delta.content
    rule: append
    apply_to_log: true
  - key: answer   # Extract the LLM answer from a non-streaming response
    value_source: response_body
    value: choices.0.message.content
    apply_to_log: true

Advanced

You can use Alibaba Cloud Simple Log Service (SLS) data transformation to extract and process AI-related fields. For example, if the raw log is as follows:

ai_log:{"question":"Calculate 2 to the power of 3 using Python","answer":"You can use Python's power operator `**` to calculate the power of a number. To calculate 2 to the power of 3, use the following code:\n\n```python\nresult = 2 ** 3\nprint(result)\n```\n\nRunning this code returns 8.","model":"qwen-max","input_token":"16","output_token":"76","llm_service_duration":"5913"}

Use the following data transformation script to extract `question` and `answer`:

e_regex("ai_log", grok("%{EXTRACTJSON}"))
e_set("question", json_select(v("json"), "question", default="-"))
e_set("answer", json_select(v("json"), "answer", default="-"))

After extraction, the `question` and `answer` fields are added in SLS. For example:

ai_log:{"question":"Calculate 2 to the power of 3 using Python","answer":"You can use Python's power operator `**` to calculate the power of a number. To calculate 2 to the power of 3, use the following code:\n\n```python\nresult = 2 ** 3\nprint(result)\n```\n\nRunning this code returns 8.","model":"qwen-max","input_token":"16","output_token":"76","llm_service_duration":"5913"}

question:Calculate 2 to the power of 3 using Python

answer:You can use Python's power operator `**` to calculate the power of a number. To calculate 2 to the power of 3, use the following code:

result = 2 ** 3
print(result)

Running this code returns 8.