The Alibaba Cloud Image for retrieval-augmented generation (RAG) services includes a built-in automatic instrumentation feature and an ARMS collector component based on the OpenTelemetry standard. You can use these features to trace data from large language model (LLM) services. This lets you visually debug and analyze key information, such as inputs and outputs for different operation types, token consumption, error counts, and response times. This capability improves system observability and debugging efficiency. This topic describes how to configure tracing analysis for an RAG service in Elastic Algorithm Service (EAS).
Obtain an OpenTelemetry endpoint and an authentication token
Activate ARMS and follow the procedure below to obtain an OpenTelemetry endpoint and an authentication token.
Log on to the ARMS console. In the navigation pane on the left, click Integration Center. In the Server-side Application area, click the OpenTelemetry card.
On the Start Integration tab of the OpenTelemetry configuration panel, configure the following parameters and then copy the Endpoint and Authentication Token.
Select Region: Select the region where you want to report data. This must be the same region where your RAG service is deployed.
Connection Method: If your RAG service is in the same region as the selected integration region, select Alibaba Cloud Private Network. Otherwise, select Public Network.
Report Method: Only gRPC is supported. Select gRPC.
Enable and configure tracing analysis
Enable Tracing Analysis
You can enable the tracing analysis feature when you deploy or update an RAG service on the Elastic Algorithm Service (EAS) page:
Deploy an RAG service: The key configurations are described below. For more information about other parameters, see Deploy a custom RAG service.
Parameter
Description
Basic Information
Tags
Click
and add the following tag:Tag Key: EnableTrace
Tag Value: true
Environment Context
Image Configuration
Select .
NoteOnly pai-rag:0.3.4 and later versions are supported. Because versions are iterated quickly, select the latest image version during deployment.
Service Registration
Virtual Private Cloud (VPC)
A virtual private cloud (VPC) must be configured. By default, EAS services cannot access the public network. If you select the public network connection method when you obtain an OpenTelemetry endpoint and an authentication token, select a VPC that can access the public network. This ensures that EAS can access the public network. For more information, see Access public or private resources from EAS.
VSwitch
Security Group Name
Update the RAG service: If you deployed a RAG service using a custom method, you can also go to the Elastic Algorithm Service (EAS) page and click Update in the Actions column for the target service. On the Update Service page, modify the parameters to your desired configuration.
Configure tracing analysis
On the Elastic Algorithm Service (EAS) page, in the Service Method column for the target RAG service, click View Web App.
On the OpenTelemetry Tracing Analysis tab of System Settings, configure the following parameters. Select Enable OpenTelemetry Tracing Analysis, and then click Save OpenTelemetry Information.
Application Name (EAS Service Name): Set this to the RAG service name.
OpenTelemetry gRPC Endpoint: Set this to the endpoint that you obtained previously.
OpenTelemetry Authentication Token: Set this to the authentication token that you obtained previously.
View call traces
Invoke the service
Configure the RAG service
On the WebUI of the RAG service, configure the settings required for service inference. For example, to use an LLM service for a knowledge base Q&A, complete the following configuration steps:
Configure LLM service connection information
Deploy an LLM service and obtain its endpoint and token. For more information, see Deploy an LLM service.
On the WebUI page, navigate to the Model and Storage Configuration tab on the System Settings page. Configure the LLM service connection information and click Save Model Configuration.
URL: Set this to the LLM service endpoint.
Key: Set this to the LLM service token.
Model Name: Set this to the model name.
Upload knowledge base files
On the WebUI page, navigate to the Knowledge Base page and upload the knowledge base files. For more information, see RAG knowledge base management.
Invoke the RAG service
On the Dialogue tab of the WebUI page, send a service request.
View link information
On the Elastic Algorithm Service (EAS) page, click the name of the RAG service. Then, click the Tracing Analysis tab to view trace information.
In the Operation column for the target trace, click View Trace to open the Trace Details page. On this page, you can view the service's input and output data, and the related log information.
FAQ
What do I do if I cannot find any trace information?
This issue can be caused by the following reasons. Review the following potential causes to resolve the issue:
Inconsistent application name:
The configured application name is different from the RAG service name, which causes log records to be associated with another service. Verify that the configured application name is correct. For more information, see Configure tracing analysis.
Inconsistent regions:
The OpenTelemetry gRPC endpoint uses a private network address, but the selected integration region is different from the RAG service's region. Ensure that the integration region and the RAG service region are the same.
Public network access is not enabled:
The OpenTelemetry gRPC endpoint uses a public network address, but the RAG service does not have public network access enabled. By default, EAS services cannot access the public network. To resolve this, connect the RAG service to a VPC that has public network access enabled. For more information, see Enable tracing analysis.
Incorrect authentication token configuration:
If the OpenTelemetry authentication token is configured incorrectly, the following error message may appear in the logs.
[2025-07-03 15:24:43] 2025-07-03 15:24:43.188 | 1056 | ERROR | logging:callHandlers:1706 - Failed to export traces to tracing-analysis-dc-hz-internal.aliyuncs.com:8090, error code: StatusCode.PERMISSION_DENIEDVerify that the configured token matches the one you obtained. For more information, see Configure tracing analysis.