Create an automated test task-Intelligent Speech Interaction(ISI)-阿里云帮助中心

Prerequisites

Ensure that you have activated Intelligent Speech Interaction. For more information, see Prepare your account.

Create an automated test task

Log on to the Intelligent Speech Interaction console.
In the left navigation pane, click Automated test. On the Automated test page, click Create task.
In the Create task panel, enter a task name. The name must be unique.
Select or upload a test set.

You can use one of the following methods: Upload annotated test set, Upload text only, or Select existing test set.

Important
Errors in the test set will cause both dataset parsing and the test task to fail. To prevent this, review the file requirements carefully.
- Upload annotated test set
  
  Use this method when you have audio files and their corresponding transcripts. The system automatically detects the sample rate.
```
corpus/
├── trans/
│   └── *.txt
└── wav/
    └── *.wav
```
  Requirements:
  - Paths cannot contain Chinese characters.
  - Each WAV filename must be unique.
  - WAV files and transcript TXT files must be in separate directories and use lowercase file extensions.
  - WAV files must be PCM-encoded with a single channel, a sample rate of 8 kHz or 16 kHz, and a 16-bit depth. You can use a tool like SoX to check the Channels, Sample Rate, and Sample Encoding.
  Transcript text
```
声音	文本
070020001.wav	xxx is not a resident of Xinzhou, is it
070020002.wav	Several trade fairs are held here every year and the prices are very low
070020003.wav	This mother is not qualified either. How can she trust xxx to take care of the child
```
  Requirements:
  - The transcript file must be UTF-8 encoded without a Byte Order Mark (BOM). Fields must be separated by a tab character.
  - The trans directory can contain multiple TXT files. Each TXT file can specify the transcripts for multiple WAV files.
  - The audio filenames in the first column must match the filenames in the wav directory. Include the .wav extension.
  - The transcript text must be normalized. For example, write out numbers as words, such as converting "5256" to "five thousand two hundred fifty-six", "2004" to "two thousand four", and "19%" to "nineteen percent". Do not include a directory path with the WAV filename, as each filename is unique.
- Upload text only
  
  Use this method when you have a text corpus but no audio. The system uses speech synthesis to generate the corresponding audio files to create a complete annotated test set. Requirements:
  - Upload a single text file in TXT format (UTF-8 encoded without a BOM).
  - Do not include punctuation. Each line must not exceed 300 characters.
- Select existing test set
  
  You can create a test set in advance and then select it when creating a task. You can select and test multiple test sets together if they have the same sample rate.
Select a model to test. You can select one of your custom language models. Then, click Next.

On the Test model page, the system automatically filters and displays models that match the test set's sample rate. The model list shows the Scene, Language, and Sample rate. Select your target model.
Optionally, select up to one model for comparison, then click OK.
Click Start task on the right of the target task.

Analyze test results

In the test results, focus on the compareResult.txt file.

The compareResult.txt file shows recognition statistics for each audio utterance in the test set. It also provides the overall recognition rate for the entire test set at the end of the file. In this file, ref indicates the reference (ground truth) transcript, and res indicates the recognition result.

Example result

00005(nwords=15,cor=15,ins=0,del=0,sub=0)  corr=100.00%,cer=0.00%
ref:   相 逢 每 醉 还 浮 云 一 别 后 流 水 十 年 间
res:   相 逢 每 醉 还 浮 云 一 别 后 流 水 十 年 间
00019(nwords=15,cor=12,ins=0,del=0,sub=3)  corr=80.00%,cer=20.00%
ref:   流 水 如 有 意 暮 禽 相 与 还 荒 城 临 古 渡
res:   流 水 如 有 意 暮 禽 相 与 还 荒 城 零 五 度
00012(nwords=15,cor=15,ins=0,del=0,sub=0)  corr=100.00%,cer=0.00%
ref:   上 国 随 缘 住 来 途 若 梦 行 浮 天 沧 海 远
res:   上 国 随 缘 住 来 途 若 梦 行 浮 天 沧 海 远
%WER 2.81 [ 8 / 285, 1 ins, 0 del, 7 sub ]
%SER 26.32 [ 5 / 19 ]
Scored 19 sentences, 0 not present in hyp.

Metrics and abbreviations

nwords: The total number of characters in the reference transcript.
cor (correct): The number of correctly recognized characters.

ins (insertion errors): The number of extra characters in the recognition result compared to the reference.
del (deletion errors): The number of missing characters in the recognition result compared to the reference.
sub (substitution errors): The number of characters that were incorrectly replaced in the recognition result.
corr (correction rate): The rate of correctly recognized characters.
- Formula: $corr = \frac{n w or d s - s u b - d e l}{n w or d s} = \frac{cor}{n w or d s}$
cer (character error rate): The percentage of incorrectly recognized characters.
- Formula: $cer = \frac{s u b + d e l + in s}{n w or d s} = \frac{n w or d s - cor + in s}{n w or d s}$
wer (word error rate): The word error rate. This metric is similar to the character error rate but is calculated at the word level. It is commonly used for English text.
ser (sentence error rate): The sentence error rate. This is the percentage of sentences with at least one error in the test set.

Manage automated test tasks

When the Current status of a task is Test completed, you can click View result, Download result, or Start task in the Actions column. You can also Edit or Delete the task.

On the Test task tab of the Automated recognition rate test page, you can filter tasks using the Current status and Associated data set drop-down lists. Click Create task to create a new test task, or select multiple tasks and click Batch delete. The task list includes columns such as Task ID, Task name, Current status, and Update time.