API reference
Real-time speech recognition receives continuous audio streams over WebSocket and returns recognition results. It supports long-running scenarios such as meetings, speeches, and live video streams. This topic describes the endpoints, request parameters, recognition events, and status codes.
Billing and concurrency limits
Real-time speech recognition is available as a trial edition and a commercial edition. For billing details, see Billable items.
For information about upgrading to the commercial edition and billing methods, see Billing methods. For concurrency limits, see Concurrency and QPS.
Usage notes
For the SDKs for Android and iOS, see Mobile SDK API reference.
Before calling the API, make sure that the audio format, sample rate, and project model meet the following requirements.
Supported input: mono audio with a bit depth of 16 bits in PCM, PCM-encoded WAV, Ogg Opus, Ogg Speex, AMR, MP3, or AAC format.
Supported sample rates: 8,000 Hz and 16,000 Hz.
Configurable output features include intermediate recognition results, punctuation added during post-processing, and conversion of Chinese numerals to Arabic numerals.
Emotion recognition is available for Chinese audio sampled at 8 kHz. To use it, disable semantic sentence segmentation by setting enable_semantic_sentence_detection to false.
Speaker diarization and speaker role analysis are not supported.
-
The project model determines the recognition language or dialect. It cannot be specified in a request parameter. For model configuration instructions, see Manage projects.
The following language and dialect models are supported:
-
Automatic routing to the nearest region
Real-time speech recognition supports automatic routing to the nearest region through nls-gateway.aliyuncs.com.
We recommend using this domain name. The service routes requests to the nearest regional server based on the client's location. For example, a request from Beijing is routed to a server in the China (Beijing) region, as if the request specified nls-gateway-cn-beijing.aliyuncs.com.
Endpoints
Access type | Description | URL |
Internet access (Shanghai by default) | All servers can use the public endpoints. The SDKs use a public endpoint by default. |
|
Internal access from ECS | ECS instances in China (Shanghai), China (Beijing), and China (Shenzhen) can use the internal endpoints. ECS instances on the classic network cannot access AnyTunnel or the speech service over the internal network. To use AnyTunnel, create a virtual private cloud (VPC) and access the service from within it. Note
|
|
Interaction workflow
The header.task_id field in a server response identifies the recognition task. Record this value for troubleshooting.
1. Authenticate
Use an NLS token to authenticate when establishing a WebSocket connection to the server.
For instructions, see Obtain a token.
2. Start recognition
Send a StartTranscription command with the recognition parameters. Start streaming audio after the server returns TranscriptionStarted. When using an SDK, configure the parameters through its corresponding methods. The following table describes the parameters.
Parameter | Type | Required | Description |
appkey | String | Yes | The Appkey of a project created in the Intelligent Speech Interaction console. |
format | String | No | The audio format: pcm, wav, opus, speex, amr, mp3, or aac. |
sample_rate | Integer | No | The audio sample rate. Default: 16,000 Hz. In the console, configure the project with a model that supports the audio sample rate and use case. |
enable_intermediate_result | Boolean | No | Specifies whether to return intermediate recognition results. Default: false. |
enable_punctuation_prediction | Boolean | No | Specifies whether to add punctuation during post-processing. Default: false. |
enable_inverse_text_normalization | Boolean | No | Specifies whether to enable inverse text normalization (ITN) to convert Chinese numerals to Arabic numerals. Default: false. |
customization_id | String | No | The ID of the custom linguistic model. |
vocabulary_id | String | No | The ID of the custom hotword vocabulary. |
max_sentence_silence | Integer | No | The silence threshold for sentence segmentation, in milliseconds. After speech is detected, silence longer than this threshold ends the sentence. Valid values: 200 to 6000. Default: 800. When enable_semantic_sentence_detection is enabled, this threshold is not used for silence-based sentence segmentation. Its value must still be within the allowed range. Silence duration is calculated from the audio data, not from the time spent waiting after data transmission stops. To use silence-based segmentation, continue sending audio, including silent segments. In PCM audio, zero-valued samples can represent silence. |
enable_words | Boolean | No | Specifies whether to return word information. Default: false. |
disfluency | Boolean | No | Specifies whether to remove filler words. Default: false (disabled). |
speech_noise_threshold | Float | No | The noise threshold. Valid values: [-1, 1]. The value affects classification as follows:
Important This is an advanced parameter. Adjust it with caution and test the effect thoroughly. |
enable_semantic_sentence_detection | Boolean | No | Specifies whether to enable semantic sentence segmentation. Default: false. When enabling this feature, also enable intermediate results by setting enable_intermediate_result to true. Note Semantic sentence segmentation can improve recognition accuracy with a small increase in latency. It is suitable for scenarios such as meeting transcription. |
special_word_filter | Object (JSON object) | No | Custom sensitive-word filtering. Specify no more than 32 words in total. The specified words can be replaced with empty strings or asterisks (*). For direct WebSocket requests, pass a JSON object in the payload, not a JSON-serialized string. An SDK configuration example follows the table. |
enable_multi_thresh_mod | Boolean | No | This parameter takes effect only when enable_semantic_sentence_detection is false, that is, when VAD-based segmentation is used. Valid values:
|
The following Java SDK snippet configures custom word filtering. transcriber is an initialized recognition object.
// Real-time transcription example.
JSONObject root = new JSONObject();
root.put("system_reserved_filter", true);
// Replace these words with empty strings.
JSONObject root1 = new JSONObject();
JSONArray array1 = new JSONArray();
array1.add("start");
array1.add("occur");
root1.put("word_list", array1);
// Replace these words with asterisks.
JSONObject root2 = new JSONObject();
JSONArray array2 = new JSONArray();
array2.add("test");
root2.put("word_list", array2);
// Set either or both filters.
root.put("filter_with_empty", root1);
root.put("filter_with_signed", root2);
transcriber.addCustomedParam("special_word_filter", root);3. Receive recognition results
Continue streaming audio and receiving recognition events. The following JSON snippets illustrate the event message structures.
Fields in the header object:
|
Parameter |
Type |
Description |
|
namespace |
String |
The namespace of the message. |
|
name |
String |
The event name. |
|
status |
Integer |
The status code, which indicates whether the request succeeded. See the Status codes section. |
|
status_text |
String |
The status message. |
|
task_id |
String |
The globally unique task ID. Record this value for troubleshooting. |
|
message_id |
String |
The message ID. |
SentenceBegin
A SentenceBegin event indicates that the server has detected the start of a sentence. The service automatically detects sentence boundaries. Example:
{
"header": {
"namespace": "SpeechTranscriber",
"name": "SentenceBegin",
"status": 20000000,
"message_id": "a426f3d4618447519c9d85d1a0d1****",
"task_id": "5ec521b5aa104e3abccf3d361822****",
"status_text": "Gateway:SUCCESS:Success."
},
"payload": {
"index": 1,
"time": 0
}
}Fields in the payload object:
|
Parameter |
Type |
Description |
|
index |
Integer |
The sentence index, which starts at 1 and increments for each sentence. |
|
time |
Integer |
The duration of audio processed so far, in milliseconds. |
TranscriptionResultChanged
A TranscriptionResultChanged event contains an updated intermediate result for the current sentence. These events are returned only when enable_intermediate_result is true. Example:
{
"header": {
"namespace": "SpeechTranscriber",
"name": "TranscriptionResultChanged",
"status": 20000000,
"message_id": "dc21193fada84380a3b6137875ab****",
"task_id": "5ec521b5aa104e3abccf3d361822****",
"status_text": "Gateway:SUCCESS:Success."
},
"payload": {
"index": 1,
"time": 1835,
"result": "The weather in",
"confidence": 1.0,
"words": [{
"text": "The",
"startTime": 630,
"endTime": 930
}, {
"text": "weather",
"startTime": 930,
"endTime": 1110
}, {
"text": "in",
"startTime": 1110,
"endTime": 1140
}]
}
} For this event, header.name is TranscriptionResultChanged, indicating an intermediate recognition result.
Fields in the payload object:
|
Parameter |
Type |
Description |
|
index |
Integer |
The sentence index, which starts at 1 and increments for each sentence. |
|
time |
Integer |
The duration of audio processed so far, in milliseconds. |
|
result |
String |
The recognition result for the current sentence. |
|
words |
List< Word > |
Word information for the current sentence. Requires enable_words to be true. |
|
confidence |
Double |
The confidence score for the recognition result, in the range [0.0, 1.0]. A higher value indicates higher confidence. |
SentenceEnd
A SentenceEnd event indicates that the server has detected the end of a sentence and returns the recognition result for that sentence. Example:
The following independent example illustrates emotion recognition for Chinese speech sampled at 8 kHz.
{
"header": {
"namespace": "SpeechTranscriber",
"name": "SentenceEnd",
"status": 20000000,
"message_id": "c3a9ae4b231649d5ae05d4af36fd****",
"task_id": "8fa76b01de2145f39db16d71c283****",
"status_text": "Gateway:SUCCESS:Success."
},
"payload": {
"index": 1,
"time": 1820,
"begin_time": 0,
"result": "北京的天气。",
"confidence": 1.0,
"words": [{
"text": "北京",
"startTime": 630,
"endTime": 930
}, {
"text": "的",
"startTime": 930,
"endTime": 1110
}, {
"text": "天气",
"startTime": 1110,
"endTime": 1380
}],
"emo_tag": "neutral",
"emo_confidence": 0.931
}
}
For this event, header.name is SentenceEnd, indicating the end of a sentence.
Fields in the payload object:
|
Parameter |
Type |
Description |
|
index |
Integer |
The sentence index, which starts at 1 and increments for each sentence. |
|
time |
Integer |
The duration of audio processed so far, in milliseconds. |
|
begin_time |
Integer |
The time of the corresponding SentenceBegin event, in milliseconds. |
|
result |
String |
The recognition result. |
|
words |
List< Word > |
Word information for the current sentence. Requires enable_words to be true. |
|
confidence |
Double |
The confidence score for the recognition result, in the range [0.0, 1.0]. A higher value indicates higher confidence. |
Emotion recognition for Chinese audio sampled at 8 kHz also returns the following fields. Disable semantic sentence segmentation to use this feature.
|
Parameter |
Type |
Description |
|
emo_tag |
String |
The emotion of the current sentence: positive (such as happiness or satisfaction), negative (such as anger, sadness, or disappointment), or neutral (no distinct emotion). |
|
emo_confidence |
Double |
The confidence score for the detected emotion, in the range [0.0, 1.0]. A higher value indicates higher confidence. |
Fields in each words entry:
|
Parameter |
Type |
Description |
|
text |
String |
The word text. |
|
startTime |
Integer |
The word start time, in milliseconds. |
|
endTime |
Integer |
The word end time, in milliseconds. |
4. End recognition
After sending all audio, send StopTranscription to end the recognition task. The server processes the remaining audio and returns TranscriptionCompleted when the task ends. Close the connection only after receiving this event.
StopTranscription does not force a sentence boundary while keeping the task running. If the remaining audio contains valid speech, the server may return SentenceEnd before completing the task. A task containing only silence does not necessarily return SentenceEnd.
Status codes
Use header.status and header.status_text in the response to check the request status. The following tables list common errors and resolutions.
Common error codes
|
Status code |
Status message |
Cause |
Solution |
|
40000000 |
The default client error code. Multiple error messages can use this code. |
Invalid parameters or request logic. |
Compare the request with the documented sample code and test the corrected request. |
|
40000001 |
The token 'xxx' has expired; The token 'xxx' is invalid |
A general client error, usually caused by an expired or invalid token. |
Compare the request with the documented sample code and test the corrected request. |
|
40000002 |
Gateway:MESSAGE_INVALID:Can't process message in state'FAILED'! |
An invalid or malformed message. |
Compare the request with the documented sample code and test the corrected request. |
|
40000003 |
PARAMETER_INVALID; Failed to decode url params |
Invalid request parameters. This error commonly occurs in RESTful requests. |
Compare the request with the documented sample code and test the corrected request. |
|
40000005 |
Gateway:TOO_MANY_REQUESTS:Too many requests! |
Too many concurrent requests. |
For the trial edition, upgrade to the commercial edition to increase concurrency. For the commercial edition, purchase a concurrency resource plan to increase the concurrency quota. |
|
40000009 |
Invalid wav header! |
An invalid header. |
If sending a WAV file with |
|
40000009 |
Too large wav header! |
An invalid WAV header in the transmitted audio. |
Use a format such as PCM or Opus. For WAV audio, check that the header contains the correct data length. |
|
40000010 |
Gateway:FREE_TRIAL_EXPIRED:The free trial has expired! |
The trial has expired and the commercial edition is not activated, or the account has overdue payments. |
Check service activation and the account balance. Purchasing a resource plan does not activate Commercial Edition. Even after purchasing a resource plan, you must upgrade the real-time speech recognition service to Commercial Edition before using it. For upgrade instructions, see Billing methods. |
|
40010001 |
Gateway:NAMESPACE_NOT_FOUND:RESTful url path illegal |
An unsupported API or parameter. |
Check the request parameters against the API documentation and correct them based on the error message. For example, check the URL when making a RESTful request with curl. |
|
40010003 |
Gateway:DIRECTIVE_INVALID:[xxx] |
A general client error. |
The client sent an invalid parameter or command. Use the API-specific error details and documentation to correct the request. |
|
40010004 |
Gateway:CLIENT_DISCONNECT:Client disconnected before task finished! |
The client closed the connection before the request completed. |
Close the connection after receiving TranscriptionCompleted. |
|
40010005 |
Gateway:TASK_STATE_ERROR:Got stop directive while task is stopping! |
The client sent a command that is not supported in the current state. |
Check the command sequence. Do not send StopTranscription again while the task is stopping. |
|
40020105 |
Meta:APPKEY_NOT_EXIST:Appkey not exist! |
The specified Appkey does not exist. |
Check the Appkey in the project configuration in the console. |
|
40020106 |
Meta:APPKEY_UID_MISMATCH:Appkey and user mismatch! |
The Appkey and token belong to different account UIDs. |
Make sure that the Appkey and token belong to the same account. Do not use the Appkey from one account with a token from another. |
|
403 |
Forbidden |
The token is invalid, for example, because it does not exist or has expired. |
Specify a valid token. Obtain a new token before the current token expires. |
|
41000003 |
MetaInfo doesn't have end point info |
Routing information could not be obtained for the Appkey. |
Make sure that the Appkey and token belong to the same account. Do not use the Appkey from one account with a token from another. |
|
41010101 |
UNSUPPORTED_SAMPLE_RATE |
An unsupported sample rate. |
Real-time speech recognition supports sample rates of 8,000 Hz and 16,000 Hz. |
|
41040201 |
Realtime:GET_CLIENT_DATA_TIMEOUT:Client data does not send continuously! |
The server timed out waiting for data from the client. |
Send audio continuously at a real-time rate. After all audio is sent, send StopTranscription and wait for TranscriptionCompleted before closing the connection. |
|
50000000 |
GRPC_ERROR:Grpc error! |
An intermittent error caused by factors such as server load or network conditions. |
Retry the request. |
|
50000001 |
GRPC_ERROR:Grpc error! |
An intermittent error caused by factors such as server load or network conditions. |
Retry the request. |
|
52010001 |
GRPC_ERROR:Grpc error! |
An intermittent error caused by factors such as server load or network conditions. |
Retry the request. |
Real-time speech recognition error codes
|
Status code |
Status message |
Cause |
Solution |
|
40000004 |
Gateway:IDLE_TIMEOUT:Websocket session is idle for too long time |
The client sent no data for more than 10 seconds after establishing the connection. |
After establishing the connection, stream audio continuously as it is captured. After all audio is sent, send StopTranscription and wait for TranscriptionCompleted before closing the connection. |
|
40270002 |
NO_VALID_AUDIO_ERROR |
Invalid audio. |
No valid text was recognized from the audio. |
|
40270003 |
DECODE_ERROR |
Audio decoding failed. |
Set format to match the actual audio format. |
|
41000002 |
APPKEY_KEY_IS_NULL |
The appkey parameter is not set correctly. |
Refer to the API documentation and sample code. |