App Performance Analytics-Alibaba Cloud Model Studio(Model Studio)-阿里云帮助中心

The multi-node architecture within an Alibaba Cloud Model Studio application presents several development challenges, such as:

Tracking internal call chains within an application
Viewing model response latency
Viewing the model's thought process

You can use App Performance Analytics to view the end-to-end processing flow of your Alibaba Cloud Model Studio applications within a workspace. This includes processes such as vector generation, vector retrieval, and Large Language Model (LLM) calls. You can also view metrics such as latency and token count, which are updated every minute.

App Performance Analytics does not currently provide an API.

Examples

Track internal call processes in an application	View model response latency	View the model's thought process

Supported applications

Agent applications, workflow applications, and code-based applications.

App Performance Analytics does not currently support agent applications created using the assistant API.

Get started

Prerequisites

When you use App Performance Analytics for the first time, click App Performance Analytics Settings in the upper-right corner of the page and follow the instructions to complete these steps:

Grant permissions to the service-linked role for OpenTelemetry.
Activate the OpenTelemetry service.
Initialize the LogStore for OpenTelemetry.

Use your Alibaba Cloud account for these operations. The service typically becomes effective within a minute after activation, but there might be delays during peak hours.

To use a RAM user for activation, the root account must configure the required permissions for the RAM user.

Usage

1. Select an application to monitor

Go to App Performance Analytics and click Select observed application > Add. If an application you created does not appear in the list, it might be because:

The application is not published.
Click Manage Applications, find the application you want to publish in the list, and then click Manage > Publish.
The application does not belong to the current workspace.

2. Start monitoring

After you add an application, it appears in the App Performance Analytics list. All prompts for the application, along with their corresponding data and metrics, are automatically tracked and synchronized to App Performance Analytics every minute.
After you click Disable Observation, the application's tracking data is no longer synchronized. If you add the application again, only new data is synchronized.
Applications and Average Application Latency can help you optimize your application's operational efficiency and cost.

In the Actions column, click View Details to see a summary of all prompts for the application over a selected period (up to 30 days). The summary includes the prompt content, Output, Latency, Call Time, and Token Number (for an explanation, see the Appendix).
Interactive experience: A CHAIN node in the list represents a complete internal call trace for the application and can be expanded. For more information, see Supported node types for App Performance Analytics.
The status. Valid values: Normal and Error.
You can search by Request ID, Trace ID, or Span ID, and filter by a specified time range.
To retrieve an ID: Click the name of a specific node in the Name column, such as AgentApp. In the Node Details panel that appears, click View IDs to see the Request ID, Trace ID, and Span ID.
Table Header Settings: Click the table header edit button in the upper-right corner of the list to customize the displayed columns.

Click the Name of a specified node to view information such as Details, Raw Data, and Annotation Records. If the node has child nodes, you can expand it.

3. Export data

On the application details page, in the upper-right corner of the Trace list tab, click Export Data. You can export the data that matches the current filter criteria in JSONL or Excel format.

4. View monitoring statistics

On the application details page, click the Monitoring Statistics tab to view the application's performance monitoring data, including the following:

Call Count: A trend chart of the application's call count.
- Failure Count and Failure Rate: Statistics on failed application calls.
- Total Token Count: A trend chart of the total, input, and output token counts.
- Average Token Count per Request: The average input and output token count for each request.
- Average Time to First Token: The response time for the first token in streaming call scenarios.
- Average Call Duration: The average latency of application calls.

You can view data by time range (up to 30 days) and aggregation granularity (by minute, by hour, or by day). You can zoom in on, download, and copy each chart.

Add to evaluation set

With App Performance Analytics, you can add Span data directly to an evaluation set for future application evaluations. This feature lets you use real online call data as evaluation samples to build evaluation sets that more closely match your actual business scenarios.

In the App Performance Analytics list, click View Details to the right of the target application to go to the Span list page. Then, click Batch Operations and select the checkboxes of the Span data you want to add to the evaluation set. You can select multiple items.
Click the Add to evaluation set button to go to the configuration page:
- Select Target Evaluation Set and Target Evaluation Set Name: You can select an existing evaluation set or create a new one.
- Import Method: Supports both Append Data and Full Overwrite.
- Field Mapping: Map fields from the Span data to the evaluation set fields. When you create a new evaluation set, the `input` and `output` fields are provided by default. You can add more custom fields. When you select an existing evaluation set, its existing table header information is automatically populated.
After you complete the configuration, click Start Import. A message appears indicating that the data has been added to the evaluation set.
You can select from the full parameters of the Span for field mapping. Each evaluation set supports a maximum of 50 field mappings.

Data annotation

App Performance Analytics supports tagging Span data for subsequent data analysis, filtering, and evaluation. The tags are shared and managed centrally using the tag management feature in application evaluation.

Add annotations

In the App Performance Analytics list, click View Details for the target application to go to the Span list page. On the Span list page, click the name of a specific node, such as AgentApp, in the Name column to open its details page. Click the Data Annotation button to add custom tags to the current node.
In the sidebar that appears, select the tags you want to add. To create a new tag, click New Tag to go to the tag management page.
Annotate based on the tag type:
- Boolean: Select Yes or No.
- Categorization: Select one or more options from the drop-down list.
- Number: Enter a number in the input box.
- Text: Enter text in the input box.

The annotations are saved automatically and immediately.

View annotations

After you save the data annotations, you can view the annotated content in the Tag column on the Span list page. The annotation results for multiple tags are displayed on separate lines in the same area.

View data

App Performance Analytics supports multi-dimensional filtering of Span data to help you quickly locate target data.

Span filter mode

Root Span: Displays only the root node, which is the entry Span for each call (default mode).
All Spans: Displays all Spans in a flat list.
Model Span: Displays only Spans that include model calls.

Filter: Add filter conditions

Click Filter, and then click Add Filter Condition. You can add multiple conditions.
After you add the conditions, click Application to filter for the specified Span data.

Filter Type	Filter Field	Supported Conditions
Status	Normal, Error (can be subdivided by error type)	is, is not
Span Name	Manually enter a Span name	contains, does not contain
Input	Manually enter a keyword	contains, does not contain
Output	Manually enter a keyword	contains, does not contain
Latency	Manually enter a value (in milliseconds)	equals, is greater than, is less than, is greater than or equal to, is less than or equal to
Total Token Count	Manually enter a value	equals, is greater than, is less than, is greater than or equal to, is less than or equal to
Input Tokens	Manually enter a value	equals, is greater than, is less than, is greater than or equal to, is less than or equal to
Output Tokens	Manually enter a value	equals, is greater than, is less than, is greater than or equal to, is less than or equal to
Tag	User-added tags	Based on tag type: Categorization (contains/does not contain/equals/does not equal), Boolean (equals/does not equal), Number (numeric comparison), Text (contains/does not contain/equals/does not equal)

Billing

The App Performance Analytics feature itself is free of charge.
Data generated by App Performance Analytics is stored in the OpenTelemetry service, which incurs separate charges. For more information about OpenTelemetry billing, see Billing.

Appendix

Glossary

Term	Description
Node	In application observability, a Node is a tracked unit of operation. Each Node has properties such as Name and Type, and records detailed information about the operation, including its start and end times. In addition, Node can form nested relationships.

Supported node types

Note: The following nodes are displayed only when they are triggered or called.

Agent Application

Node Type	Description
CHAIN	Chain nodes connect LLM nodes with other types of nodes to handle complex tasks. Chain nodes can contain other types of nodes, such as Retriever and LLM. When Chain is the root vertex, the possible values for Name are: AgentApp (agent application), WorkflowApp (workflow application).
AGENT	An Agent node represents a call to an agent.
RETRIEVER	A Retriever node performs retrieval operations. KnowledgeRetriever indicates a search within a knowledge base. Retrieval from long-term memory is not currently observable. When a Retriever is a child node of KnowledgeRetriever, its Name can be one of two types: TextRetriever: Indicates that a text retrieval was triggered. Text similarity is calculated using an improved BM25 algorithm. By default, 100 text segments are returned. This number cannot be adjusted. VectorRetriever: Indicates that a vector retrieval was triggered. By default, 100 text segments are returned. This number cannot be adjusted.
REWRITER	A Rewriter node automatically adjusts the original input prompt based on the session context to improve knowledge retrieval results.
EMBEDDING	An Embedding node converts an input prompt into a numerical vector. Token Number refers to the number of tokens vectorized by the embedding model in this instance.
RERANKER	A Reranker node calculates a similarity score for each input text segment and sorts them in descending order.
LLM	An LLM node represents a call to an LLM, such as Qwen-plus, for inference or text generation. Token Number is the sum of the model's input tokens and output tokens. The Latency (Call Duration) of an LLM node includes the time taken to output the reply.
TOOL	A Tool node represents a call to a plugin. Both official and custom plugins are supported. For more information, see Plugin overview. For example, calling a calculator or Quark Search.
GUARDRAIL	A Guardrail node represents a call to Alibaba Cloud Content Moderation. It is used for real-time monitoring, detection, and blocking of various types of non-compliant content, such as gambling and pornography. ManualIntervention indicates that an intervention rule you set for the agent application was triggered. SystemIntervention indicates that a system intervention rule was triggered.

Workflow Application

Node Type	Description
CHAIN	Chain nodes connect LLM nodes with other types of nodes to handle complex tasks. Chain nodes can contain other types of nodes, such as Retriever and LLM. When Chain is the root vertex, the possible values for Name are: AgentApp (agent application), WorkflowApp (workflow application).
START	Represents the start node.
RETRIEVER	A Retriever node performs retrieval operations. KnowledgeRetriever indicates a search within a knowledge base. Retrieval from long-term memory is not currently observable. When a Retriever is a child node of KnowledgeRetriever, its Name can be one of two types: TextRetriever: Indicates that a text retrieval was triggered. Text similarity is calculated using an improved BM25 algorithm. By default, 100 text segments are returned. This number cannot be adjusted. VectorRetriever: Indicates that a vector retrieval was triggered. By default, 100 text segments are returned. This number cannot be adjusted.
REWRITER	A Rewriter node automatically adjusts the original input prompt based on the session context to improve knowledge retrieval results.
EMBEDDING	An Embedding node converts an input prompt into a numerical vector. Token Number refers to the number of tokens vectorized by the embedding model in this instance.
RERANKER	A Reranker node calculates a similarity score for each input text segment and sorts them in descending order.
LLM	Represents the LLM node. Token Number is the sum of the model's input tokens and output tokens. The Latency (Call Duration) of an LLM node includes the time taken to output the reply.
API	Represents the API node.
CLASSIFIER	Represents the intent classification node.
TEXT_CONVERTER	Represents the text conversion node.
SCRIPT	Represents the script conversion node.
CONDITION	Represents the condition judgement node.
FUNCTION_COMPUTE	Represents the Function Compute node.
APP_FLOW	Represents the AppFlow node.
GUARDRAIL	A Guardrail node represents a call to Alibaba Cloud Content Moderation. It is used for real-time monitoring, detection, and blocking of various types of non-compliant content, such as gambling and pornography. ManualIntervention indicates that an intervention rule you set for the agent application was triggered. SystemIntervention indicates that a system intervention rule was triggered.
END	Represents the end node.

High-Code Application

Node Type	Description
CHAIN	FullCodeApp indicates that a created code-based application was triggered. Tracking its internal call chain is not currently supported.

FAQ

How do I configure permissions for a RAM user to activate App Performance Analytics?

Grant the AliyunBailianFullAccess global management (Alibaba Cloud Model Studio) permission to the RAM user.
Grant the App Performance Analytics - Operations (or Administrator) page permission to the RAM user, allowing them to perform write operations on the App Performance Analytics page.
Create a system policy to create a service-linked role and grant it to the RAM user.
1. Log on to the RAM console. In the navigation pane on the left, choose Permission Management > Access Policies. Then, click Create Policy.
2. On the Script Editor tab, enter the corresponding content from the following script into the Effect, Action, Resource, and Condition fields. Then, click OK.
```
{
    "Version": "1",
    "Statement": [
        {
            "Action": "ram:CreateServiceLinkedRole",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}
```
3. Enter CreateServiceLinkedRole as the policy name and click OK.
4. In the navigation pane on the left, choose Identity Management > Users. Find the RAM user to whom you want to grant permissions and click Add Permission in the Actions column.
5. From the Access Policy list, select the policy you just created (CreateServiceLinkedRole), and then click OK. The RAM user now has the permission to create service-linked roles.
After configuring all the permissions, return to the App Performance Analytics page and try to activate Application Observation again using the RAM user.

I have enabled monitoring for my code-based application, but why can't I see statistics such as call volume?

Check the following items:

Ensure that the information to be reported is defined in your code. You can use the Tracing module of AgentScope-AI to define this information.
Ensure that you added the --telemetry enable parameter when deploying the code-based application to enable observability.