Enable log shipping for an MSE cloud-native gateway-Microservices Engine(MSE)-阿里云帮助中心

Log shipping integrates MSE cloud-native gateways with Alibaba Cloud Simple Log Service (SLS), allowing you to analyze gateway access logs to understand client behavior, map user geographic distribution, and troubleshoot issues. This topic describes how to enable log shipping for an MSE cloud-native gateway.

Prerequisites

You have created an MSE cloud-native gateway instance. For more information, see Create a cloud-native gateway.
Simple Log Service must be activated. If not, log on to the Simple Log Service console with your Alibaba Cloud account to activate it. For more information, see What is Simple Log Service?.

Note

The log shipping feature itself is free of charge, but standard Simple Log Service (SLS) charges apply based on usage. For more information about SLS billing, see pay-as-you-go.

Enable log shipping

Log on to the MSE console. In the top navigation bar, select a region.
In the left-side navigation pane, choose Cloud-native Gateway > Gateways. On the Gateways page, click the ID of the gateway.
In the left-side navigation pane, click Parameter Settings. In the Observability Parameters section, click the edit icon next to Log Shipping. In the Log Shipping Settings dialog box, turn on the Enable Log Shipping (Ship Gateway Access Logs to Log Service) switch.
Note
- After enabling log shipping, Simple Log Service creates a default SLS project. You can also select an existing SLS project.
- After enabling Tracing Analysis, you can view gateway monitoring data on the Tracing Analysis console. For more information, see Enable Tracing Analysis for a gateway.
After enabling log shipping, you can click the link next to Project in the Observability Parameters section. This action redirects you to the gateway's Logstore. For more information, see Quick start for log query and analysis.

Troubleshooting: no content in plug-in logs

If log shipping is enabled but no content appears when you query plug-in logs, troubleshoot in the following order:

Widen the query time range: Make sure the query time window covers the time when the requests occurred. Simple Log Service (SLS) log shipping has a latency of seconds to minutes, so expand the time window as needed.
Verify that requests invoke the plug-in: Look for the corresponding request records in the gateway access logs. If no such records exist, the requests never reached the plug-in. Check the route configuration.
Check the log output logic in the plug-in code: Verify that business requests actually reach the log output statements. Even when log shipping works correctly, if conditional branches in the plug-in code prevent requests from reaching the log output statements, the plug-in logs contain no content.

Log shipping fields

Field name	Type	Description
__time__	long	The time the log entry was generated.
cluster_id	string	The ID of the gateway instance.
consumer	string	The consumer of the request. This field is populated only when consumer authentication is enabled.
custom_log	json	Stores user-defined log fields. This can be used with custom plug-ins.
authority	string	The Host header value in the request.
bytes_received	long	The size of the request body in bytes, excluding headers.
bytes_sent	long	The size of the response body in bytes, excluding headers.
downstream_local_address	string	The address of the gateway pod.
downstream_remote_address	string	The address of the client connected to the gateway.
downstream_transport_failure_reason	string	The reason for a downstream transport failure.
duration	long	Total request processing time in milliseconds, measured from the first byte received from the client to the last byte sent in the response.
method	string	The HTTP method.
path	string	The path in the HTTP request.
protocol	string	The HTTP protocol version.
request_id	string	The gateway generates an ID for each request and includes it in the `x-request-id` header. The backend service can log this ID for troubleshooting.
requested_server_name	string	The server name used for the SSL connection.
request_duration	long	The time in milliseconds from when the gateway receives the first byte to the last byte of the downstream request.
response_code	long	The HTTP response status code.
response_code_details	string	Additional details about the response code. For example, `via_upstream` indicates that the response code was returned by the backend service, and `route_not_found` indicates that no matching route was found for the request.
response_flags	string	The reason for a response failure.
response_tx_duration	long	The time in milliseconds from when the gateway receives the first byte from the upstream service to when it sends the last byte to the downstream client.
route_name	string	The name of the route.
start_time	string	The start time of the request in UTC format.
trace_id	string	The Trace ID.
upstream_cluster	string	The upstream cluster.
upstream_host	string	The IP address of the upstream service.
upstream_local_address	string	The local address used to connect to the upstream service.
upstream_protocol	string	The protocol used to request the backend service.
upstream_service_time	long	The time in milliseconds for the upstream service to process the request. This duration includes network latency and the service's own processing time.
upstream_transport_failure_reason	string	The reason for an upstream connection failure.
user_agent	string	The User-Agent header in the HTTP request.
x_forwarded_for	string	The value of the `x-forwarded-for` HTTP header. This header typically indicates the real IP address of the client.
ext_authz_status_code	long	The response code from the custom authorization service.
ext_authz_duration	long	The response time of the custom authorization service.

Add custom fields to access logs

To record a custom field, such as Referer, in access logs, use the log format adjustment feature to configure a custom log format.

On the gateway details page, click Parameter Settings in the left-side navigation pane.
In the Observability Parameters section, find Log Format Adjustment under Log Shipping.
Click Default Format (Click to Customize). In the dialog box that appears, click Add.
Set Field Type to Request Header, and enter the field value, such as Referer.
Click OK to save the configuration. The status changes to Custom Format (Click to Edit).

After the configuration takes effect, requests that include the Referer request header are recorded, and the request_headers field in the access log contains the Referer value.

Log format adjustment supports the following field types: request header, response header, request dynamic metadata, and response dynamic metadata.

Response flags

The reason for a failed request is primarily determined by the value of Response_Flag in the logs. The following describes the different values of Response_Flag.

Note

In this context, downstream refers to the client, and upstream refers to the backend service.

UH: No healthy upstream hosts are available in the cluster.
UF: The upstream connection failed.
NR: No configured route matched the request. When a request matches no route (response_flags=NR and response_code_details=route_not_found), response_code is 404, and both route_name and upstream_cluster are displayed as -. The request is rejected at the gateway layer and never reaches the backend service. Therefore, empty values for upstream_cluster and all upstream_* fields are expected behavior and do not indicate a problem with the upstream cluster configuration. To troubleshoot this issue, check whether a route has been configured for the request path.
URX: The request was rejected because the upstream retry limit (HTTP) or maximum connection attempts (TCP) was reached.
NC: The upstream cluster was not found.
DT: The request or connection exceeded max_connection_duration or max_downstream_connection_duration.
DC: The downstream connection was terminated.
LH: The local service failed a health check request.
UT: The upstream request timed out.
LR: The connection was reset locally.
UR: The upstream connection was reset remotely.
UC: The upstream connection was terminated.
DI: The request processing was delayed for a period specified by fault injection.
FI: The request was aborted with a response code specified by fault injection.
RL: The request was locally limited by the HTTP rate limiting filter, excluding 429 responses.
UAEX: The request was rejected by an external authorization service.
RLSE: The request was rejected due to an error in the rate limiting service.
IH: The request was rejected because a strictly-checked header contained an invalid value.
SI: The stream idle timeout was reached.
DPE: An HTTP protocol error occurred in the downstream request.
UPE: An HTTP protocol error occurred in the upstream response.
UMSDR: The upstream request reached its maximum duration.
OM: The overload manager terminated the request.
DF: The request was terminated due to a DNS resolution failure.