Using observability settings-Alibaba Cloud Service Mesh-阿里云帮助中心

Prerequisites

An ASM instance of version 1.17.2.35 or later is required. For more information, see Create an ASM instance or Upgrade an ASM instance.

Scopes

Type	Description
Global	A global configuration supports settings for logs, metrics, and tracing. Only one global configuration exists and it cannot be deleted. Tracing settings are supported only at the global scope.
Namespace	Allows you to create a dedicated observability configuration for a namespace. Each namespace can have only one namespace-level observability configuration.
Custom	A custom configuration uses a label selector to define its scope. Each workload can be selected by at most one custom configuration.

Procedure

Global

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Global tab. Configure logs, metrics, and tracing as required, and then click Submission.

Click the links in the following table to view detailed descriptions of the configurations.

Configuration section	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Filter logs
Metric settings	Enable or disable metric generation Metric dimensions
Tracing settings	Sampling percentage Custom tags

Namespace

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Namespace tab and click Create. Select the target Namespace, configure logs and metrics as required, and then click Create.

Click the links in the following table to view detailed descriptions of the configurations.

Configuration section	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Filter logs
Metric settings	Enable or disable metric generation Metric dimensions

Custom

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Custom tab, select the target Namespace, and click Create. Enter a Name and Label selector, configure logs and metrics as required, and then click Create.

Click the links in the following table to view detailed descriptions of the configurations.

Configuration section	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Filter logs
Metric settings	Enable or disable metric generation Metric dimensions

Log settings

Log settings include enabling or disabling access log output, setting the log output format, customizing the log format, and filtering logs.

Access log output

In the Log Settings section, turn the Enable Log Output switch on or off as required.
- If you enable this option, sidecars or gateways on the data plane output access logs to the container's standard output (stdout).
- If you disable this option, sidecars or gateways on the data plane stop outputting logs to the container's stdout.

View logs in the stdout of the data plane sidecar container.

The following examples show how to view access logs by using kubectl.

Run the following command to view sidecar logs:

kubectl logs httpbin-5c5944c58c-w**** -c istio-proxy --tail 1

Sample output

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.29:80",
    "downstream_remote_address":"221.220.XXX.XXX:0",
    "duration":"80",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"default",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"inbound|80||",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"127.0.X.X:55879",
    "upstream_response_time":"79",
    "upstream_service_time":"79",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

Run the following command to view ingress gateway logs:

kubectl -n istio-system logs istio-ingressgateway-6cff9b6b58-r**** --tail 1

Sample output

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.63:80",
    "downstream_remote_address":"221.220.XXX.XXX:64284",
    "duration":"81",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"-",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"httpbin",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"192.168.0.63:36140",
    "upstream_response_time":"81",
    "upstream_service_time":"81",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

(Optional) View access logs in the Container Service for Kubernetes console.

If you use an Alibaba Cloud Container Service for Kubernetes (ACK) cluster, you can also view access logs in the ACK console.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Pods.
3. On the Pods page, click the name of the target pod, and then click the Logs tab at the bottom of the page to view the access logs.

Log output format

Note

This feature requires an ASM instance of version 1.20.6.36 or later. For more information about how to upgrade an instance, see Upgrade an ASM instance.

In the Log Settings section, set Log Output Format to JSON or TEXT as required.

If you select JSON, access logs are output to the container's stdout as JSON strings.
If you select TEXT, access logs are output to the container's stdout as plain text strings.

Custom log format

In the Log Settings section, select fields, modify custom field information, or click the icon at the bottom of the log field list to add a new log field.

You can customize the log format only when the Enable Log Output switch is on. In the Log Format section, the default selected log fields are mandatory and cannot be deselected. Log field values can be retrieved from request headers, response headers, or built-in Envoy values.

For example, to log the accept-encoding request header, set accessLogFormat key to accept-encoding, Type to Request Properties, and accessLogFormat value to Accept-Encoding. The format template for a request header is %REQ(HEADER_NAME)% (for example, %REQ(:AUTHORITY)%), for a response header is %RESP(HEADER_NAME)% (for example, %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%), and for a built-in Envoy attribute is %ATTRIBUTE_NAME% (for example, %REQUEST_DURATION%).

Run the following command to view the logs of data plane components in the service mesh.

kubectl logs httpbin-5c5944c58c-w**** -c istio-proxy --tail 1|grep accept-encoding --color=auto

Sample output

{
    "bytes_received":"0",
    "bytes_sent":"9593",
    "downstream_local_address":"192.168.0.29:80",
    "downstream_remote_address":"69.164.XXX.XX:0",
    "duration":"2",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/",
    "protocol":"HTTP/1.1",
    "request_id":"29939dc9-62be-4ddf-acf6-32cb098d****",
    "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"default",
    "start_time":"2023-06-30T04:18:19.734Z",
    "trace_id":"-",
    "upstream_cluster":"inbound|80||",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"127.0.X.X:34723",
    "upstream_service_time":"2",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 zgrab/0.x",
    "x_forwarded_for":"69.164.XXX.XX",
    "authority_for":"47.110.XX.XXX",
    "upstream_response_time":"2",
    "accept-encoding":"gzip"
}

The access log now shows the value of the Accept-Encoding header that you added in step 1.

Log filtering

Under the Log Settings section, select Enable Log Filter to activate log filtering, and then enter a filtering expression in the text box. Access logs are generated only for requests that match the expression.

For example, to log only requests that meet the condition Response Http Status >= 400, use the expression response.code >= 400. For more information, see CEL expressions and common fields.

CEL expressions and common fields

Log filtering expressions are standard Common Expression Language (CEL) expressions. The following table lists common fields for CEL expressions. For more information, see the official CEL and Envoy documentation.

Attribute	Type	Description
request.path	string	The request path.
request.url_path	string	The request path without the query string.
request.host	string	The host portion of the URL.
request.method	string	The request method.
request.headers	map<string, string>	All request headers, indexed by their lowercase names.
request.useragent	string	The value of the User-Agent header.
request.time	timestamp	The time when the first byte of the request is received.
request.id	string	The request ID.
request.protocol	string	The request protocol. Valid values: `HTTP/1.0`, `HTTP/1.1`, `HTTP/2`, or `HTTP/3`.
request.query	string	The query string of the request URL.
response.code	int	The HTTP status code of the response.
response.code_details	string	Details about the response code.
response.grpc_status	int	The gRPC status code in the response.
response.headers	map<string, string>	All response headers, indexed by their lowercase names.
response.size	int	The size of the response body in bytes.
response.total_size	int	The total size of the response message in bytes, including the body and headers.

Metric settings

Metric settings include enabling or disabling metric generation and configuring metric dimensions.

Metric generation

Metrics are categorized as client-side metrics and server-side metrics.

Client-side metrics: Metrics generated when a sidecar acts as a client to initiate requests. Gateway metrics are also categorized as client-side metrics.
Server-side metrics: Metrics generated when a sidecar acts as a server to receive requests.

In the Metric Settings section, in the Client-Side Metrics or Server-Side Metrics column, select or clear the Enabled checkbox for a metric as required.
- If a metric is enabled, the data plane sidecar or gateway exposes this metric through the /stats/prometheus path on port 15020.
- If a metric is disabled, it is not exposed on the specified port and path.

Run the following command to view the metrics exposed by a sidecar or gateway.

You can use kubectl to run a curl command inside the sidecar or gateway container to access the /stats/prometheus path on local port 15020 and view the exported metrics.

kubectl exec httpbin-5c5944c58c-w**** -c istio-proxy -- curl 127.0.0.1:15020/stats/prometheus|head -n 10

Sample output:

# TYPE istio_agent_cert_expiry_seconds gauge
istio_agent_cert_expiry_seconds{resource_name="default"} 46725.287654548
# HELP istio_agent_endpoint_no_pod Endpoints without an associated pod.
# TYPE istio_agent_endpoint_no_pod gauge
istio_agent_endpoint_no_pod 0
# HELP istio_agent_go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE istio_agent_go_gc_duration_seconds summary
istio_agent_go_gc_duration_seconds{quantile="0"} 5.0149e-05
istio_agent_go_gc_duration_seconds{quantile="0.25"} 9.8807e-05
......

Metric dimensions

Metric dimensions provide rich contextual information. You can use these dimensions to filter target metrics in Prometheus. For example, you can use the source_app dimension to filter metrics for requests from a specific client application.

Edit default dimensions

Follow these steps to edit the default dimensions:

In the Metric Settings section, in the Client-Side Metrics or Server-Side Metrics column, click Edit dimension for an enabled metric.
In the Customize CLIENT dimension configuration or Customize SERVER dimension configuration dialog box, select or clear the checkboxes for the dimensions to be exported, and then click Confirm.

For example, if no dimensions are disabled, you can run a curl command inside the sidecar or gateway container to access the /stats/prometheus path on local port 15020 and view the exported metrics.

kubectl exec httpbin-5c5944c58c-w**** -c istio-proxy -- curl 127.0.0.1:15020/stats/prometheus

Taking the istio_request_bytes_sum metric (which corresponds to the REQUEST_SIZE metric in the console) as an example, you can see that it includes all dimensions by default.

istio_request_bytes_sum{reporter="destination",source_workload="istio-ingressgateway",source_canonical_service="unknown",source_canonical_revision="latest",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway",source_app="istio-ingressgateway",source_version="unknown",source_cluster="c479fc4abd2734bfaaa54e9e36fb26c01",destination_workload="httpbin",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/httpbin",destination_app="httpbin",destination_version="v1",destination_service="httpbin.default.svc.cluster.local",destination_canonical_service="httpbin",destination_canonical_revision="v1",destination_service_name="httpbin",destination_service_namespace="default",destination_cluster="c479fc4abd2734bfaaa54e9e36fb26c01",request_protocol="http",response_code="200",grpc_response_status="",response_flags="-",connection_security_policy="mutual_tls"} 18000

If you modify the default server-side REQUEST_SIZE metric to retain only the response_code dimension and then access the /stats/prometheus path, you will see that the metric now includes only the response_code dimension.

istio_request_bytes_sum{response_code="200"} 16550

Add custom dimensions

Follow these steps to add custom dimensions:

In the Metric Settings section, in the Client-Side Metrics or Server-Side Metrics column, click Edit dimension for an enabled metric.
In the Customize CLIENT dimension configuration or Customize SERVER dimension configuration dialog box, under the Custom Dimensions option, edit the dimension name and value, and then click Confirm.

For example, if you edit the server-side REQUEST_SIZE metric and add a custom dimension with the name request_path and value request.path, the exported metric will include the custom request_path dimension when you access the /stats/prometheus path.

istio_request_bytes_sum{response_code="200",request_path="/spec.json"} 5800

Important

You can reduce the memory consumption of Envoy and Prometheus by removing unneeded default dimensions. However, since most dimensions are typically useful, the Metric Settings section only displays the dimensions that you have explicitly removed.

Tracing settings

Tracing settings include sampling percentage and custom tags. To build complete call chains, tracing requires consistent reporting configurations across all services. Inconsistent reporting endpoints or sampling rates can lead to incomplete traces. For this reason, ASM versions earlier than 1.24.6.83 do not allow namespace-level or workload-level tracing configurations. Starting from version 1.24.6.83, ASM supports modifying Telemetry resources by using the Kubernetes API to enable namespace-level and workload-level tracing configurations. For more information about configuring Telemetry resources, see Telemetry CRD.

Sampling percentage

You can customize the sampling percentage for tracing, which determines the percentage of requests for which traces are generated. A value of 0 disables tracing.

Custom tags

You can customize the tags attached to reported tracing spans. In the Tracing Analysis Settings section, click Add Custom Tags, and then configure the Name, Type, and Value.

Valid types include Fixed Value, Request Header, and Environment Variable. The following table describes each type and provides examples.

Type	Description	Example configuration
Fixed Value	The tag value is fixed to the string that you specify.	Name: env Type: Fixed Value Value: prod
Request Header	The tag uses the value of a specified request header. If the header does not exist in the request, the default value is used. For example, you can get the tag value from the `User-Agent` header and use `unknown` as the default value if the header is not present.	Name: useragent Type: Request Header Value: Header name: User-Agent Default value: unknown
Environment Variable	The tag uses the value of a specified environment variable from the workload. If the environment variable does not exist in the workload, the default value is used. For example, you can get the tag value from the `ENV` environment variable and use `unknown` as the default value if the variable does not exist.	Name: env Type: Environment Variable Value: Environment Variable Name: ENV Default value: unknown