Annotation Center (Call Annotation)
This topic describes how to use the Annotation Center module.
Function overview
The effectiveness of an outbound calling service depends on the bot's interaction with the user. Annotating call conversations lets you correct the bot's response errors, which improves interactions and the overall effectiveness of your calls. In the Annotation Center, you can annotate outbound call conversations by creating annotation tasks.
Feature access
From the main interface of Intelligent Outbound Calling, select Business Management in the left-side menu bar. Then, select the desired workspace.
In the workspace, select Annotation Center from the left-side menu bar to open the Annotation Center page.
Add an annotation task
In the Annotation Center, click the Add Task button to open the Add Annotation Task page.
The Annotation Center organizes work into annotation tasks. Each task contains a set of successfully connected outbound or inbound calls, with a maximum of 1,000 calls per task.
On the Add Annotation Task page, configure the task settings. These settings include Task information, Filter criteria, and Sample selection. After you configure the settings, click OK to add the annotation task.
Task information:
Select scenario: After you select a scenario, all annotation data for the task is sourced from outbound call conversations within that scenario.
After you select an outbound calling scenario, the system automatically generates a name for the annotation task.
Filter criteria: You can filter the annotation data.
Call result: You can filter calls by a combination of 11 different call results.
Filter annotated statements: Select this option to exclude previously annotated calls from the task.
Call time range: Filter calls by the time range in which they occurred. A call is included if any part of its duration, from connection to hang-up, falls within the selected time range.
Data source: You can select either an outbound task or a test task as the data source.
Sample selection method: If many calls match the filter criteria, you can use one of the following three sample selection modes:
Full annotation: Includes all calls that match the filter criteria without performing secondary sampling. The total number of calls in a task is limited to 1,000. The system selects calls starting from the earliest record that falls within the specified time range.
Proportional sampling: Randomly samples a specified percentage of calls that match the filter criteria. The total number of samples is limited to 1,000. The actual sampling percentage is displayed.
Custom quantity: Samples a specified number of calls, starting from the earliest call that matches the time range. The total number of samples is limited to 1,000.
Close or delete an annotation task
Definitions:
Close: Closes an unfinished task. A closed task cannot be restarted.
Delete: Deletes a task. This action cannot be undone. This action deletes only the annotation content, not the original call data.
Steps: In the Annotation Center, find the task you want to manage. Click Close or Delete to the right of the task.
Complete an annotation task
Enter the annotation workbench
In the Annotation Center, find the desired annotation task and click Annotate to the right of the task. This opens the annotation workbench.
The left panel of the annotation workbench displays basic call information, which is the same as the call details for an outbound call.
The middle area of the annotation workbench displays the conversation content. In this area, you can perform actions such as speed up audio playback, highlight text to tag, and use keyboard shortcuts to quickly switch between turns.
The right panel of the annotation workbench is the annotation operations area. The content in this area changes as you switch between conversation turns.
Perform annotation
You can annotate the following content in the bot-user conversation: semantic recognition results, intents, correct transcription results, hotwords, and tags.
For each conversation turn, you can process the annotation items as needed. All items are optional. Each item is submitted and saved separately, allowing you to exit at any time and resume annotating later.
When you open a call, the first conversation turn is automatically selected.
After you enter an annotation task, you can start annotating each call. Press Alt+F (Option+F on macOS) to enter full screen. In full screen mode, press Esc to exit.
Semantic recognition result annotation
Definition: Annotate how well the bot recognized the meaning of the user's statement. This type of annotation applies to the conversation interaction. The following methods can be used for this operation:
Keyboard shortcuts:
Use a keyboard shortcut, such as Ctrl+1, to select a label under Semantic recognition result.
Press Alt+Enter (Option+Enter on macOS) to submit the annotation.
Using the mouse: Move the cursor over the desired label under Semantic recognition result. Click to select the label, and then click the Submit button to complete the annotation.
You can select a new label and submit it again. This action will overwrite the previous selection.
Semantic annotations are applied to the similar questions of an intent in the conversation flow and take effect immediately.
Intent annotation
Definition: Annotate the user's question with an intent. This adds the user's question as a similar question for a specific intent.
Keyboard shortcuts:
When the focus is on the Select bot drop-down list, press Enter to expand it. Use the Up and Down arrow keys to navigate the options and press Enter to make a selection.
Press Tab to move the focus to the Select intent drop-down list. Press Enter to expand it. Use the Up and Down arrow keys to navigate the options and press Enter to make a selection.
Press Tab to move the focus to the input box for similar questions. You can either type directly or highlight text in the conversation to Auto Fill the box. Press Enter to temporarily save the entry. You can add multiple entries.
Press Alt+Enter (Option+Enter on macOS) to submit. You can also revoke the submission.
Press Tab to move the focus to the View button. Press Enter to see all temporarily saved content.
Using the mouse: Click to select the corresponding label, and then click the Submit button to complete the annotation. Click the View button to see the corresponding annotation content.
Correct transcription result annotation
Description: Indicates the status of the speech-to-text conversion of the user's query.
In the input field, enter the correct Automatic Speech Recognition (ASR) transcription. You can also highlight text from the conversation to automatically fill the field.
Hotword annotation
Select or add a new hotword list.
Add hotwords to the list. You can add multiple hotwords. Each hotword has a threshold value from -6 to 5. A higher value increases the recognition weight of the hotword but may negatively affect the recognition of other words. The default value is 3.
Annotated hotwords are added to the hotword list in the Voice Dataset. For more information, see the Voice Dataset documentation.
Tag annotation
Tags can be used flexibly. You can apply multiple tags to each conversation turn. The system provides five preset tags, but you can add or delete tags as needed. Tags applied to a call are also included in the task statistics.
The Audio and Semantics tags appear next to each user statement. These tags indicate that you have completed the corresponding annotation items for that statement.
Switching calls
You can switch between calls by clicking the Previous or Next button. When you move to the next call, all unsubmitted annotations for the current call are automatically submitted.
View annotation task progress
Click the blue bar at the top to expand a view that shows the status of all calls in the task. Click the bar again to collapse it.
Finish the annotation task
When the annotation task is complete, click the Finish Annotation button to end the task.
Clicking the Finish Annotation button closes the annotation task, regardless of whether all calls have been annotated.
Click the Save and Exit button to save your progress and continue annotating later.