AI annotation

Configure fields of interest

In the Fields of Interest step, add the data required for annotation, such as user questions, model answers, and business context.

1. Click Add Field and select a field in the dataset. To add all fields, click Add All Schema Fields.

2. Select a display type based on the content, and enter a remark that describes the business meaning of the field.

3. To read a subfield in a JSON object, enter a dot-separated path, such as metadata.user.name. Before you select a subfield, make sure that the data is valid JSON and the path matches the data structure.

4. Check the preview on the right. After you confirm the field content, click Next.

We recommend that you add only fields that are related to the annotation goal. For example, to evaluate answer quality, you usually need to provide both the user question and the model answer.

Configure annotation keys

In the Annotation Configuration step, select existing annotation keys, or click New Key to create an annotation field. Each key represents an annotation dimension. For example, intent represents the user intent, and quality_score represents the answer quality.

When you create a key, specify the key name, type, description, and corresponding configurations. The key name must start with a letter, can contain only letters, digits, and underscores (_), and can be up to 64 characters in length. We recommend that you clearly describe the judgment goal, the meaning of each value, and boundary conditions in the description.

A template can contain multiple keys. AI generates a separate result for each key selected in the task.

Preview and save

Go to Annotation View Preview and check the fields of interest, annotation controls, and display order. You can drag the handle on the left side of a card to adjust the order.

After you confirm the settings, click Save Annotation Template to return to the AI annotation configuration. You can also click Save and Go to AI Annotation to continue configuring the task. Operations on controls in the preview are only for viewing the effect and do not submit annotation results.

Select display fields and annotation keys

In the Annotation Configuration section, confirm the input and output of the task.

Select at least one display field and one annotation key. If a required field or key is not in the list, click Configure Annotation Template to add it and save the template first.

Configure custom business recognition rules

To add business criteria, add a Skill in the Custom Business Recognition Rules section. A Skill is an annotation guide written in Markdown that describes field meanings, judgment steps, classification boundaries, and reference examples. If you do not have additional rules, skip this step.

When you write rules, use the same field names, key names, and enumeration options as those in the annotation template. For example, when you define classification rules for an enumeration key, the judgment results must come from the configured options. Rules in a Skill supplement the judgment criteria. To add a key or an enumeration option, modify the annotation configuration first.

Specify how to handle existing annotation results

"Skip Existing Results" is evaluated by key. For example, if a record already has intent but lacks quality_score, the task still fills in quality_score and retains the original intent. Keys that are not selected in this task are not re-annotated.

Configure a scheduling policy

Select an execution method based on your data processing requirements.

For one-time execution, the last 1 hour is selected by default. Adjust the time range based on the data to be annotated. The data time range corresponds to the system time field __time__ of the dataset. Make sure that the time range covers the target data.

The current query conditions or annotation filters on the dataset details page are not automatically applied to the AI annotation task. Confirm the processing scope in the task configuration.

View and correct a single result

1. In the data list, open the annotation panel of the target record and switch to Annotation View.

2. Compare the results of each key against the original content in the display fields.

3. To correct a result, modify the value in the corresponding control and click Submit Annotation. To continue checking subsequent records, click Complete and Next.

Filter by annotation results

Click Annotation Filter on the dataset details page to find data based on the annotation status or the result of a specific key. For example, you can filter records whose is_complaint is "Complaint", or records whose quality_score is within a low score range.

Annotated indicates that a record contains at least one annotation result. It does not mean that all keys in the template are filled in. If multiple annotation conditions are configured, data must meet all of the conditions.

Configure the template

Add question and answer as fields of interest, and create the following annotation keys.

Enter business rules

Enter the following content in the Skill editor. If you use the manual input method, you can replace the sample skeleton with this content and adjust it based on your business requirements.

---
name: customer-service-annotation
description: Identify the intent of customer service conversations and evaluate answer quality
---

# Customer service conversation annotation rules

Fields of interest: question is the user question, and answer is the customer service answer.

1. intent: Identify the main request of the user. Select only from the configured enumeration options.
2. is_complaint: true if the user explicitly expresses dissatisfaction or makes a complaint;
   false if the user only makes a general inquiry.
3. quality_score:
   - 1: The main issue is not resolved. Only a perfunctory or irrelevant answer is given.
   - 3: The question is partially addressed, but key information or a solution is missing.
   - 5: The question is accurately addressed, and a clear and actionable solution is provided.
   - 2 and 4 are used for cases between the adjacent criteria.
4. review_note: Briefly describe the basis for the score based on question and answer.
   Do not exceed 80 characters, and do not add facts that do not exist in the conversation.

Select the two display fields and four annotation keys, select One-time Execution, set the time range to cover the sample data, and then click Start Annotation.

Check the results

Consider the following conversation:

• question: My order is already three days past the estimated delivery time and I still haven't received it. This is really disappointing.

• answer: Please wait patiently.

You can refer to the following example to check whether the annotation results comply with the rules.

Why is the Start Annotation button unavailable?

Make sure that at least one display field and one annotation key are selected and that a valid execution time configuration is specified. If fields are still loading or fail to load, or a Skill is being imported, wait until the operation is complete or reload the page, and then submit the task again.

Why are no new annotation results generated for some data?

Check the task running status and the data time range first. If you use Skip Existing Results, keys that already have values are retained. To update these results, create a new task and select Re-annotate and Overwrite.

How do I re-annotate historical data?

Go to AI Annotation from the dataset page again, confirm the keys and business rules that need to be updated, select Re-annotate and Overwrite, select One-time Execution, set a time range that covers the target data, and then start annotation.

Are images automatically recognized if the display type is set to image?

AI annotation currently analyzes text and structured content. The image display type in a template is used only to view images and does not automatically enable image content recognition.

How do I improve annotation results?

Provide the context required for judgment. In key descriptions and Skills, clearly define classification boundaries, scoring criteria, and how to handle insufficient information. Then, check the results on a small amount of representative data first. For critical results, perform manual reviews to correct them.