CreateExperimentPlan-AgentLoop(AgentLoop)-阿里云帮助中心

Creates an experiment plan.

Operation description

Calls CreateExperimentPlan to create an experiment plan in a specified AgentSpace. This operation is used to define the configuration for an offline or online experiment, including the data source, optional evaluators, and experiment groups required for online experiments. After the plan is created, call CreateExperimentRun to initiate the execution.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

agentloop:CreateExperimentPlan

create

*Workspace

acs:agentloop:{#regionId}:{#accountId}:agentspace/{#AgentSpaceName}

None

Request syntax

POST /api/v1/experiments/{agentSpace}/plans HTTP/1.1

Path Parameters

Parameter	Type	Required	Description	Example
agentSpace	string	No	The name of the AgentSpace.	al-playground-cn-hongkong

Request parameters

Parameter	Type	Required	Description	Example
body	object	No	The request body. The experiments parameter is optional for offline experiments. For online experiments, at least one experiment must be specified.	{"planName": "rca_benchmark_eval_experiment", "experimentType": "OFFLINE", "description": "", "datasetId": "rca_benckmark_eval", "evaluators": [{"evaluatorRef": "Builtin.agent_correctness", "name": "Builtin.agent_correctness", "type": "AGENT", "resultName": "Builtin.agent_correctness", "resultType": "score", "variableMapping": {"input": "experiment_input", "output": "experiment_output", "expected_output": "dataset.ground_truth_json"}, "filters": {"query": "", "sample": "100"}, "config": {"variables": [], "prompt": ""}}, {"evaluatorRef": "rca-toxicity-safety-accuracy", "name": "rca-toxicity-safety-accuracy", "type": "AGENT", "resultName": "rca-toxicity-safety-accuracy", "resultType": "score", "variableMapping": {"input": "experiment_input", "output": "experiment_output", "question": "dataset.question", "expected_output": "dataset.ground_truth_json", "payload_json": "dataset.payload_json"}, "filters": {"query": "", "sample": "100"}, "config": {"variables": [], "prompt": ""}}]}
planName	string	Yes	The name of the experiment plan. The name must be unique within the same account in the same AgentSpace.	rca_benchmark_eval_experiment
experimentType	string	Yes	The type of the experiment. Valid values: `OFFLINE` and `ONLINE`. Valid values: OFFLINE : OFFLINE ONLINE : ONLINE	OFFLINE
description	string	No	The description of the experiment plan.	rca_benchmark_eval_experiment offline experiment
datasetId	string	No	The ID of the associated dataset. If this parameter is not specified, the execution phase is processed in simple mode.	rca_benckmark_eval
experiments	array	Yes	The list of experiment configurations. A maximum of five configurations can be specified. This parameter is optional or can be an empty array for offline experiments. For online experiments, at least one configuration must be specified.	[{"label": "A", "name": "experimentA", "modelName": "qwen3.7-plus", "modelProvider": "dashscope", "modelParameters": {"temperature": 0.7, "topP": 0.8, "presencePenalty": 0.0, "frequencyPenalty": 0.0}, "promptTemplate": [{"role": "system", "content": "You are the Alibaba Cloud ARMS product Q&A bot."}, {"role": "user", "content": "{{input}}"}]}, {"label": "B", "name": "experimentB", "modelName": "qwen3.7-max", "modelProvider": "dashscope", "modelParameters": {"temperature": 0.7, "topP": 0.8, "presencePenalty": 0.0, "frequencyPenalty": 0.0}, "promptTemplate": [{"role": "system", "content": "You are the Alibaba Cloud ARMS product Q&A bot."}, {"role": "user", "content": "{{input}}"}]}]
	ExperimentConfig	No	The configuration of a single experiment group. For online experiments, the label, name, and modelName parameters are required. For non-agent mode, the promptTemplate parameter must be provided.
evaluators	array	No	The list of evaluators. After configuration, evaluation can be automatically triggered when the experiment is completed.	[{"evaluatorRef": "Builtin.agent_correctness", "name": "Builtin.agent_correctness", "type": "AGENT", "resultName": "Builtin.agent_correctness", "resultType": "score", "variableMapping": {"input": "experiment_input", "output": "experiment_output", "expected_output": "dataset.ground_truth_json"}, "filters": {"query": "", "sample": "100"}, "config": {"variables": [], "prompt": ""}}, {"evaluatorRef": "rca-toxicity-safety-accuracy", "name": "rca-toxicity-safety-accuracy", "type": "AGENT", "resultName": "rca-toxicity-safety-accuracy", "resultType": "score", "variableMapping": {"input": "experiment_input", "output": "experiment_output", "question": "dataset.question", "expected_output": "dataset.ground_truth_json", "payload_json": "dataset.payload_json"}, "filters": {"query": "", "sample": "100"}, "config": {"variables": [], "prompt": ""}}]
	Evaluator	No	The configuration object of a single evaluator.
selectedItemIds	array	No	The list of selected data item IDs in partial dataset mode. This parameter must be used together with `datasetId`.	["019ef4d5-a0f0-7114-832d-5542d771cd8c", "019f1729-be9b-7769-a006-8e98023ad7ad"]
	string	No	The ID of a single data item in the dataset.	019ef4d5-a0f0-7114-832d-5542d771cd8c
querySql	string	No	The custom query SQL clause in partial dataset mode. This parameter can be used when `selectedItemIds` is empty.	status='OK'
pipelineName	string	No	The name of the associated data processing pipeline. This parameter is optional. After association, when the experiment execution under this plan is completed and the experiment results are written to the Logstore, the system filters by the traceId of the current experiment trace and calls PreviewPipeline to write the results processed by the pipeline together.	my-pipeline
input	object	No	Optional.	{"question": "How do I request a refund?"}

For experimentType, pass OFFLINE or ONLINE in the request. The query returns offline or online.

Regarding experiments:

OFFLINE: This parameter is optional or can be an empty array. Offline groups are specified by using offlineExperiments when CreateExperimentRun is called.
ONLINE: At least one experiment group must be specified, up to a maximum of five.

Sample request for an offline experiment

{
  "planName": "rca_benchmark_eval_experiment",
  "experimentType": "OFFLINE",
  "description": "",
  "datasetId": "rca_benckmark_eval",
  "evaluators": [
    {
      "evaluatorRef": "Builtin.agent_correctness",
      "name": "Builtin.agent_correctness",
      "type": "AGENT",
      "resultName": "Builtin.agent_correctness",
      "resultType": "score",
      "variableMapping": {
        "input": "experiment_input",
        "output": "experiment_output",
        "expected_output": "dataset.ground_truth_json"
      },
      "filters": {
        "query": "",
        "sample": "100"
      },
      "config": {
        "variables": [],
        "prompt": ""
      }
    },
    {
      "evaluatorRef": "rca-toxicity-safety-accuracy",
      "name": "rca-toxicity-safety-accuracy",
      "type": "AGENT",
      "resultName": "rca-toxicity-safety-accuracy",
      "resultType": "score",
      "variableMapping": {
        "input": "experiment_input",
        "output": "experiment_output",
        "question": "dataset.question",
        "expected_output": "dataset.ground_truth_json",
        "payload_json": "dataset.payload_json"
      },
      "filters": {
        "query": "",
        "sample": "100"
      },
      "config": {
        "variables": [],
        "prompt": ""
      }
    }
  ]
}

Sample request for an online experiment

{
  "planName": "arms_agent_experiment",
  "experimentType": "ONLINE",
  "description": "",
  "datasetId": "arms_customer_agent_level1",
  "experiments": [
    {
      "label": "A",
      "name": "experimentA",
      "modelName": "qwen3.7-plus",
      "modelProvider": "dashscope",
      "modelParameters": {
        "temperature": 0.7,
        "topP": 0.8,
        "presencePenalty": 0.0,
        "frequencyPenalty": 0.0
      },
      "promptTemplate": [
        {
          "role": "system",
          "content": "You are the Alibaba Cloud ARMS product Q&A bot."
        },
        {
          "role": "user",
          "content": "{{input}}"
        }
      ]
    },
    {
      "label": "B",
      "name": "experimentB",
      "modelName": "qwen3.7-max",
      "modelProvider": "dashscope",
      "modelParameters": {
        "temperature": 0.7,
        "topP": 0.8,
        "presencePenalty": 0.0,
        "frequencyPenalty": 0.0
      },
      "promptTemplate": [
        {
          "role": "system",
          "content": "You are the Alibaba Cloud ARMS product Q&A bot."
        },
        {
          "role": "user",
          "content": "{{input}}"
        }
      ]
    }
  ],
  "evaluators": [
    {
      "evaluatorRef": "Builtin.agent_correctness",
      "name": "Builtin.agent_correctness",
      "type": "AGENT",
      "resultName": "Builtin.agent_correctness",
      "resultType": "score",
      "variableMapping": {
        "input": "experiment_input",
        "output": "experiment_output",
        "expected_output": "dataset.expected_output"
      },
      "filters": {
        "query": "",
        "sample": "100"
      },
      "config": {
        "variables": [],
        "prompt": ""
      }
    }
  ]
}

Response elements

Element	Type	Description	Example
	object	The response for creating an experiment plan.
requestId	string	The request ID.	3FE4CD1E-FF41-56BE-B590-7A021D9C1524
planId	string	The ID of the experiment plan.	exp-plan-e95bff54685a4ae29ff3a834c1008a71
status	string	The creation result. A value of `created` indicates success. Valid values: created : created error : error	created
message	string	The message.	The experiment plan is created

Returns planId and an HTTP 201 status code upon success. The plan is not automatically executed after creation. You must call CreateExperimentRun to initiate execution.

Examples

Success response

JSON format

{
  "requestId": "3FE4CD1E-FF41-56BE-B590-7A021D9C1524",
  "planId": "exp-plan-e95bff54685a4ae29ff3a834c1008a71",
  "status": "created",
  "message": "The experiment plan is created"
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.