This service recognizes long audio streams. It is suitable for scenarios that require continuous recognition for extended periods, such as conference speeches and live streaming on ApsaraVideo Live.
Billing and concurrency limits
Real-time Speech Recognition is available in a trial version and a commercial version. For more information, see Trial and commercial versions. To upgrade from the trial version to the commercial version, see Upgrade from the trial version to the commercial version.
For more information about billing methods, see Billing methods.
For more information about concurrency limits, see Concurrency and QPS.
Usage notes
To use the Android or iOS software development kit (SDK), see API reference for mobile devices.
Follow the requirements below when you write your code. Otherwise, recognition may fail and return an empty result.
Supported input formats: Single-channel (mono) audio with a 16 bit audio bit depth. Supported encoding formats include PCM, WAV files with PCM encoding, OPUS in OGG containers, SPEEX in OGG containers, AMR, MP3, and AAC.
Supported audio sample rates: 8000 Hz and 16000 Hz.
Result settings: You can configure whether to return intermediate recognition results, add punctuation in post-processing, and convert Chinese numerals to Arabic numerals.
Sentiment analysis: This feature is available only for Chinese speech at an 8 kHz sample rate. When you use this feature, you must disable semantic sentence detection by setting `enable_semantic_sentence_detection` to `False`.
Speaker diarization is not supported. Role analysis cannot be performed.
Multi-language recognition: You cannot specify language and dialect models in your code. In the Intelligent Speech Interaction console, go to the All Projects page, find the desired project, and click Configure Project Features to select a model. For more information, see Manage projects.
The following tables describe the supported language and dialect models.
Intelligent access to the nearest region
Real-time Speech Recognition supports intelligent access to the nearest region through the domain name nls-gateway.aliyuncs.com.
We recommend that end users use this domain name to access the service. The system automatically resolves requests to the server in the nearest region based on the client's geographical location. For example, if a request is initiated from the China (Beijing) region, the system automatically resolves it to a server in the China (Beijing) region. This is equivalent to using the region-specific domain name nls-gateway-cn-beijing.aliyuncs.com.
Service endpoints
Access type | Description | URL |
Public network access (defaults to the China (Shanghai) region) | All servers can use public network access URLs. The SDK uses public network access URLs by default. |
|
ECS internal network access | If you use an Alibaba Cloud ECS instance in the China (Shanghai), China (Beijing), or China (Shenzhen) region, you can use an internal network access URL. ECS instances in the classic network cannot access AnyTunnel and therefore cannot access the Voice Service over the internal network. To use AnyTunnel, create a VPC and access the service from within the VPC. Note
|
|
Interaction flow
Every server response includes a `Task_id` parameter in the header that identifies the recognition task.
1. Authentication
When the client establishes a WebSocket connection with the server, it uses a token for authentication. For more information about how to obtain a token, see Obtain a token.
2. Start recognition
The client sends a request, and the server validates it. You must set parameters in the request message. To set the parameters, use the `set` method of the `SpeechTranscriber` object in the SDK. The following table describes the parameters.
Parameter | Type | Required | Description |
appkey | String | Yes | The AppKey of the project that you created in the console. To obtain an AppKey, go to the console. |
format | String | No | The audio format. Valid values include PCM, WAV, OPUS, SPEEX, AMR, MP3, and AAC. |
sample_rate | Integer | No | The audio sample rate. The default value is 16000 Hz. Configure a model that supports this sample rate and scenario in the corresponding project in the console. |
enable_intermediate_result | Boolean | No | Specifies whether to return intermediate recognition results. The default value is `false`. |
enable_punctuation_prediction | Boolean | No | Specifies whether to add punctuation in post-processing. The default value is `false`. |
enable_inverse_text_normalization | Boolean | No | Inverse text normalization (ITN), which converts Chinese numerals to Arabic numerals. If you set this parameter to `True`, Chinese numerals are converted to Arabic numerals in the output. The default value is `False`. |
customization_id | String | No | The ID of the custom language model. |
vocabulary_id | String | No | The ID of the custom hotword vocabulary. |
max_sentence_silence | Integer | No | The threshold for voice activity detection (VAD). If the duration of a silence period exceeds this threshold, the sentence is considered complete. The valid range is 200 ms to 6000 ms. The default value is 800 ms. This parameter is invalid if you enable semantic sentence detection by setting enable_semantic_sentence_detection to `true`. |
enable_words | Boolean | No | Specifies whether to return word-level information. The default value is `false`. |
disfluency | Boolean | No | Filters filler words, also known as disfluency detection. The default value is `false`, which disables this feature. |
speech_noise_threshold | Float | No | The noise threshold. The valid range is [-1, 1].
Important This is an advanced parameter. Adjust it with caution and test it thoroughly. |
enable_semantic_sentence_detection | Boolean | No | Specifies whether to enable semantic sentence detection. The default value is `False`. This parameter must be used with the intermediate result parameter. To enable semantic sentence detection, you must also set `enable_intermediate_result` to `true`. Note Enabling semantic sentence detection can improve recognition accuracy but slightly increases latency. It is suitable for scenarios such as meeting transcription. |
special_word_filter | String (JSON format) | No | The sensitive word filtering feature. You can customize up to 32 sensitive words. This parameter lets you perform the following actions: Do not process (default `false`, which displays the original text) or replace with asterisks (*). For more information about how to call this parameter, see the custom word filtering example below. |
enable_multi_thresh_mod | Boolean | No | This parameter takes effect only when `enable_semantic_sentence_detection` is set to `False` (VAD-based sentence detection). Valid values:
|
The following code provides an example of how to call the custom word filtering parameter:
// The following code provides an example for real-time transcription.
JSONObject root = new JSONObject();
root.put("system_reserved_filter", true);
// Replace the following words with an empty string.
JSONObject root1 = new JSONObject();
JSONArray array1 = new JSONArray();
array1.add("start");
array1.add("occur");
root1.put("word_list", array1);
// Replace the following words with asterisks (*).
JSONObject root2 = new JSONObject();
JSONArray array2 = new JSONArray();
array2.add("test");
root2.put("word_list", array2);
// You can configure all or some of the parameters.
root.put("filter_with_empty", root1);
root.put("filter_with_signed", root2);
transcriber.addCustomedParam("special_word_filter", root);3. Receive recognition results
The client continuously sends audio data and receives recognition results.
The `SentenceBegin` event indicates that the server has detected the start of a sentence. The intelligent sentence detection feature of the Real-time Speech Recognition service determines the start and end of a sentence. The following code provides an example:
{ "header": { "namespace": "SpeechTranscriber", "name": "SentenceBegin", "status": 20000000, "message_id": "a426f3d4618447519c9d85d1a0d1****", "task_id": "5ec521b5aa104e3abccf3d361822****", "status_text": "Gateway:SUCCESS:Success." }, "payload": { "index": 1, "time": 0 } }The following table describes the parameters in the `header` object.
Parameter
Type
Description
namespace
String
The namespace of the message.
name
String
The name of the message. `SentenceBegin` indicates the start of a sentence.
status
Integer
The status code that indicates whether the request was successful. For more information, see the service status codes.
status_text
String
The status message.
task_id
String
The globally unique ID (GUID) of the task. Record this value for troubleshooting.
message_id
String
The ID of the message.
The following table describes the parameters in the `payload` object.
Parameter
Type
Description
index
Integer
The sentence number, which increments from 1.
time
Integer
The duration of the audio that has been processed, in milliseconds.
The `TranscriptionResultChanged` event indicates that the recognition result has changed. This message is returned multiple times only if `enable_intermediate_result` is set to `true`. This event provides the intermediate recognition result of a sentence. The following code provides an example:
{ "header": { "namespace": "SpeechTranscriber", "name": "TranscriptionResultChanged", "status": 20000000, "message_id": "dc21193fada84380a3b6137875ab****", "task_id": "5ec521b5aa104e3abccf3d361822****", "status_text": "Gateway:SUCCESS:Success." }, "payload": { "index": 1, "time": 1835, "result": "Beijing's sky", "confidence": 1.0, "words": [{ "text": "Beijing's", "startTime": 630, "endTime": 1110 }, { "text": "sky", "startTime": 1110, "endTime": 1140 }] } }The parameters in the `header` object are the same as those in the preceding table. The `name` is `TranscriptionResultChanged`, which indicates the intermediate recognition result of the sentence.
The following table describes the parameters in the `payload` object.
Parameter
Type
Description
index
Integer
The sentence number, which increments from 1.
time
Integer
The duration of the audio that has been processed, in milliseconds.
result
String
The recognition result of the current sentence.
words
List< Word >
The word-level information of the current sentence. You must set `enable_words` to `true`.
confidence
Double
The confidence level of the recognition result for the current sentence. The valid range is [0.0, 1.0]. A larger value indicates a higher confidence level.
The `SentenceEnd` event indicates that the server has detected the end of a sentence and returns the recognition result for that sentence. The following code provides an example:
{ "header": { "namespace": "SpeechTranscriber", "name": "SentenceEnd", "status": 20000000, "message_id": "c3a9ae4b231649d5ae05d4af36fd****", "task_id": "5ec521b5aa104e3abccf3d361822****", "status_text": "Gateway:SUCCESS:Success." }, "payload": { "index": 1, "time": 1820, "begin_time": 0, "result": "The weather in Beijing.", "confidence": 1.0, "words": [{ "text": "The", "startTime": 630, "endTime": 930 }, { "text": "weather", "startTime": 930, "endTime": 1110 }, { "text": "in Beijing", "startTime": 1110, "endTime": 1380 }], "emo_tag": "neutral", "emo_confidence": 0.931 } }The parameters in the `header` object are the same as those in the preceding table. The `name` is `SentenceEnd`, which indicates the end of a sentence.
The following table describes the parameters in the `payload` object.
Parameter
Type
Description
index
Integer
The sentence number, which increments from 1.
time
Integer
The duration of the audio that has been processed, in milliseconds.
begin_time
Integer
The time of the `SentenceBegin` event corresponding to the current sentence, in milliseconds.
result
String
The current recognition result.
words
List< Word >
The word-level information of the current sentence. You must set `enable_words` to `true`.
confidence
Double
The confidence level of the recognition result for the current sentence. The valid range is [0.0, 1.0]. A larger value indicates a higher confidence level.
emo_tag
String
The emotion of the current sentence. Valid values are `positive` (positive emotions such as happiness and satisfaction), `negative` (negative emotions such as anger, dullness, and disappointment), and `neutral` (no obvious emotion).
emo_confidence
Double
The confidence level of the emotion recognition for the current sentence. The valid range is [0.0, 1.0]. A larger value indicates a higher confidence level.
The following table describes the parameters in the `Words` object.
Parameter
Type
Description
text
String
The text.
startTime
Integer
The start time of the word, in milliseconds.
endTime
Integer
The end time of the word, in milliseconds.
4. End recognition
The client notifies the server that all audio data has been sent. After the server finishes recognition, it notifies the client that the task is complete.
Service status codes
Each response from the service includes a `status` field, which is the service status code. The following tables list general error codes, gateway error codes, and configuration error codes.
General-purpose error codes
Status code | Status message | Cause | Solution |
40000000 | The default client error code. This code corresponds to multiple error messages. | Invalid parameters or call logic was used. | Compare your code with the sample code in the official documentation to test and verify it. |
40000001 | The token 'xxx' has expired. The token 'xxx' is invalid | Invalid parameters or call logic was used. This is a general-purpose client error code that usually indicates an incorrect token, such as an expired or invalid token. | Compare your code with the sample code in the official documentation to test and verify it. |
40000002 | Gateway:MESSAGE_INVALID:Can't process message in state'FAILED'! | The message is invalid or incorrect. | Compare your code with the sample code in the official documentation to test and verify it. |
40000003 | PARAMETER_INVALID Failed to decode url params | The parameters passed by the user are incorrect. This error is common for RESTful API calls. | Compare your code with the sample code in the official documentation to test and verify it. |
40000005 | Gateway:TOO_MANY_REQUESTS:Too many requests! | Too many concurrent requests. | If you are using the Free Edition, you can upgrade to a commercial version to increase the concurrency. If you are already using a commercial version, you can purchase a concurrency resource plan to increase your concurrency quota. |
40000009 | Invalid wav header! | The message header is invalid. | If you send a WAV audio file and set the |
40000009 | Too large wav header! | The WAV header of the transmitted audio is invalid. | You can send the audio stream in a format such as PCM or OPUS. If you use the WAV format, make sure that the WAV header of the audio file contains the correct data length. |
40000010 | Gateway:FREE_TRIAL_EXPIRED:The free trial has expired! | The trial period has ended, and the commercial version is not activated or your account has an overdue payment. | You can log on to the console to check the service activation status and your account balance. |
40010001 | Gateway:NAMESPACE_NOT_FOUND:RESTful url path illegal | The operation or parameter is not supported. | Check whether the parameters passed in the call are consistent with the requirements in the official documentation. You can compare them with the error message to identify and set the correct parameters. For example, if you are using a curl command to make a RESTful API request, check whether the URL you constructed is valid. |
40010003 | Gateway:DIRECTIVE_INVALID:[xxx] | A general-purpose client-side error code. | This error indicates that the client passed an incorrect parameter or instruction. Detailed error messages are available for different operations. You can refer to the corresponding documentation to set the parameters correctly. |
40010004 | Gateway:CLIENT_DISCONNECT:Client disconnected before task finished! | The client actively terminated the connection before the request was processed. | None. Alternatively, you can close the connection after the server responds. |
40010005 | Gateway:TASK_STATE_ERROR:Got stop directive while task is stopping! | The client sent a message instruction that is not currently supported. | Compare your code with the sample code in the official documentation to test and verify it. |
40020105 | Meta:APPKEY_NOT_EXIST:Appkey not exist! | A non-existent Appkey was used. | Confirm whether a non-existent Appkey was used. You can log on to the console and view the project configuration to find the Appkey. |
40020106 | Meta:APPKEY_UID_MISMATCH:Appkey and user mismatch! | The Appkey and token passed in the call were not created by the same Alibaba Cloud account UID. This causes a mismatch. | Check whether you are using resources from two different accounts. Do not use an Appkey from Account A with a token generated from Account B. |
403 | Forbidden | The token is invalid. For example, the token does not exist or has expired. | Set a valid token. Tokens have an expiration period. You must obtain a new token before the current one expires. |
41000003 | MetaInfo doesn't have end point info | Failed to retrieve the routing information for this Appkey. | Check whether you are using resources from two different accounts. Do not use an Appkey from Account A with a token generated from Account B. |
41010101 | UNSUPPORTED_SAMPLE_RATE | The sample rate is not supported. | Real-time speech recognition currently supports only audio with a sample rate of 8000 Hz or 16000 Hz. |
41040201 | Realtime:GET_CLIENT_DATA_TIMEOUT:Client data does not send continuously! | Failed to retrieve data from the client due to a timeout. | When you call real-time speech recognition, the client must send data at a real-time rate and close the connection promptly after the data is sent. |
50000000 | GRPC_ERROR:Grpc error! | An exception caused by factors such as machine load or network issues. This error usually occurs randomly. | You can retry the call to resolve the issue. |
50000001 | GRPC_ERROR:Grpc error! | An exception caused by factors such as machine load or network issues. This error usually occurs randomly. | You can retry the call to resolve the issue. |
52010001 | GRPC_ERROR:Grpc error! | An exception caused by factors such as machine load or network issues. This error usually occurs randomly. | You can retry the call to resolve the issue. |
Real-time speech recognition error codes
Status code | Status message | Cause | Solution |
40000004 | Gateway:IDLE_TIMEOUT:Websocket session is idle for too long time | After a connection is established, the server returns this error message if no data is sent for more than 10 seconds. | After establishing a connection, maintain interaction with the server. For example, continuously send the audio stream. You can send the audio while it is being captured. Close the connection promptly after the audio is sent. |
40010004 | Gateway:CLIENT_DISCONNECT:Client disconnected before task finished! | The client actively terminated the connection before the request was processed. | None. Alternatively, close the connection after the server responds. |
40270002 | NO_VALID_AUDIO_ERROR | The audio is invalid. | No valid text was recognized from the audio. |
40270003 | DECODE_ERROR | Audio decoding failed. | Set the format parameter based on the actual audio format. |
41000002 | APPKEY_KEY_IS_NULL | The appkey was not set correctly. | Refer to the official documentation and sample code. |