FAQs
This topic answers frequently asked questions about AgentLoop, covering product overview, getting started, observability, and evaluation.
Product overview
What is AgentLoop?
AgentLoop is a one-stop platform for enterprise AI agents provided by Alibaba Cloud. For more information, see AgentLoop. It drives a self-evolving observe-evaluate-optimize cycle that continuously improves agent quality, efficiency, cost-effectiveness, and security.
How is AgentLoop different from traditional APM and LLMOps tools?
Traditional Application Performance Management (APM) tools monitor conventional microservices but cannot interpret multi-step reasoning chains or token consumption patterns unique to AI agents. Large language model operations (LLMOps) tools focus on individual model calls and provide only fragmented observability.
AgentLoop is purpose-built for the agent paradigm and differs in three ways: full-stack observability with end-to-end auto-instrumentation and automatic Trajectory extraction; an Agent-as-a-Judge evaluation system where the evaluator itself plans and reasons across multiple steps; and a closed-loop data flywheel from observation to optimization, rather than standalone point tools.
Getting started
How do I get started with AgentLoop?
Try the Playground without logging in. To integrate the full workflow, see the AgentLoop QuickStart guide.
How do I distinguish different agent applications in a multi-application deployment?
Set a unique --serviceName startup parameter for each application. Other parameters such as licenseKey, workspace, project, and endpoint can remain the same across applications.
For example, if you run multiple Docker instances for different use cases, configure a different serviceName for each instance. Each application then appears as an independent entry in the AgentLoop console.
How does AgentSpace organize and manage resources?
AgentSpace is the top-level workspace in AgentLoop. Each AgentSpace defines a resource boundary for a team, business unit, or independent project.
Create a separate AgentSpace for each business domain, such as customer service, operations, or development, to achieve resource isolation. You can manage access control within each AgentSpace through Alibaba Cloud Resource Access Management (RAM).
Full-stack observability
What data can AgentLoop observe from agents?
AgentLoop collects data across the entire agent call chain, from user terminals and gateways to the agent and its dependencies, including tools and models:
Trajectory — Complete execution traces that capture the full reasoning and action sequence of each agent run.
Model calls — Input prompts, output responses, token consumption, and latency for each invocation.
Tool calls — Parameters, return values, duration, and status for each invocation.
Conversation records — Dialog history within sessions.
Errors — Exceptions at each stage of the execution chain.
Performance metrics — End-to-end latency, throughput, and resource utilization.
How do I avoid duplicate data collection when using both an agent framework and the OpenAI SDK?
If you enable both the agent framework plugin and the OpenAI plugin, LLM spans are collected twice. Choose only one collection method:
If the agent framework plugin already captures the complete call chain including LLM calls, do not add the OpenAI plugin.
If you call the OpenAI SDK directly without an agent framework, use only the OpenAI plugin.
How do I query session data? Is there an SDK?
AgentLoop does not provide a standalone session query SDK. Query session-level data by using Simple Log Service (SLS) SQL. Log in to the SLS console and aggregate data by the attributes.gen_ai.session.id field to calculate per-session metrics.
The following table describes the available session-level metric fields:
Field | Description |
| Number of conversation turns in the session. |
| Total token consumption for the session. |
| Number of LLM calls in the session. |
| Number of tool calls in the session. |
| Number of agent invocations in the session. |
| Active interaction time, excluding idle intervals. |
| Wall-clock time from session start to the last recorded activity. |
How do I collect data if my agent does not use a mainstream framework shown in AgentLoop?
If your agent uses a mainstream framework, AgentLoop provides built-in GenAI instrumentation out of the box. If your agent is built without a framework, use the OpenTelemetry (OTel) SDK for custom instrumentation to report trace data to AgentLoop.
Why is TTFT sometimes not visible on the dashboard?
Some frameworks (such as Claude Agent SDK) do not expose the underlying model access interface, so these frameworks cannot collect metrics like TTFT.
Evaluation and experimentation
What is Agent-as-a-Judge? How is it different from traditional LLM-as-a-Judge?
Agent-as-a-Judge is an evaluation paradigm introduced by AgentLoop. Unlike LLM-as-a-Judge, which relies on single-turn text judgments, Agent-as-a-Judge uses an evaluator that is itself an AI agent with planning, tool-use, and multi-step reasoning capabilities. The evaluator analyzes complete Trajectory traces to perform in-depth evaluation, including:
Understanding multi-step reasoning processes.
Analyzing the appropriateness and effectiveness of tool calls.
Tracing decision-branch logic across the execution chain.
This approach produces evaluation results that more closely match those of human domain experts.
Is multi-turn session evaluation supported?
The current evaluator supports single-turn evaluation based on individual traces or trajectories. Multi-turn session evaluation is planned for a future release, expected around September 2026.