WebSocket protocol-Intelligent Speech Interaction(ISI)-阿里云帮助中心

Features

Real-time speech recognition receives audio streams over WebSocket and returns transcription results. It supports long-form audio. Commands and events use JSON text frames, and audio uses binary frames. For details about frame types, see Data Frames.

Supported input formats: PCM, PCM-encoded WAV, OGG-encapsulated OPUS, OGG-encapsulated SPEEX, AMR, MP3, and AAC. Audio must be mono. PCM audio must have a bit depth of 16 bits.
Supported sample rates: 8000 Hz and 16000 Hz. The audio, request parameters, and model configured for the project must use matching sample rates.
The service can return intermediate results, add punctuation, convert Chinese numerals to Arabic numerals, and return word-level information.
Language and dialect models are configured for a project and cannot be switched through the request parameters of this protocol. For configuration instructions, see Manage projects.

Authentication

The server authenticates requests with a temporary token. When establishing a WebSocket connection, pass a valid token in the token query parameter.

For instructions, see Obtain an access token.

Replace <your_token> in the following URLs with an actual token.

Access type	Description	URL
Public access	Connect to the service over the Internet.	`wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1?token=<your_token>`
Private access from an ECS instance in Shanghai	For ECS instances in a VPC in the China (Shanghai) region. Classic networks do not support this access method. Private access does not incur public data transfer charges for the ECS instance.	`ws://nls-gateway-cn-shanghai-internal.aliyuncs.com:80/ws/v1?token=<your_token>`

Interaction flow

Send commands and audio in the following order:

Establish a WebSocket connection with a token.
Send StartTranscription and wait for TranscriptionStarted.
Send audio in binary frames while handling SentenceBegin, TranscriptionResultChanged, and SentenceEnd events. Intermediate results are returned only when the corresponding parameter is enabled.
After all audio has been sent, send StopTranscription.
Close the connection after receiving TranscriptionCompleted. If an error occurs, read the status code and error message in the TaskFailed event.

Commands

Commands are sent as JSON text frames containing a header and an optional payload. The following JSON examples illustrate the protocol structure. Replace the Appkey and generate actual task and message IDs when making requests.

Header format

Parameter	Type	Required	Description
appkey	String	Yes	The Appkey of the project. For instructions on obtaining it, see Manage projects.
message_id	String	Yes	A unique ID for the current message, generated by the client as 32 hexadecimal characters. Use a new ID for each command.
task_id	String	Yes	A unique ID for the recognition task, consisting of 32 hexadecimal characters. Keep this ID unchanged for all commands in the same task.
namespace	String	Yes	Set to `SpeechTranscriber`.
name	String	Yes	The command name: `StartTranscription` or `StopTranscription`.

StartTranscription command

The payload supports the following parameters:

Parameter	Type	Required	Description
format	String	No	The audio format. Default: `pcm`. Supported values: `pcm`, `wav`, `opus`, `speex`, `amr`, `mp3`, and `aac`.
sample_rate	Integer	No	The audio sample rate in Hz. Default: `16000`. Set to `8000` or `16000` to match the audio, and configure a matching model for the project.
enable_intermediate_result	Boolean	No	Specifies whether to return intermediate results. Default: `false`.
enable_punctuation_prediction	Boolean	No	Specifies whether to add punctuation during post-processing. Default: `false`.
enable_inverse_text_normalization	Boolean	No	Specifies whether to enable inverse text normalization (ITN) to convert Chinese numerals to Arabic numerals. Default: `false`.
customization_id	String	No	The ID of the custom linguistic model.
vocabulary_id	String	No	The ID of the custom hotword vocabulary.
max_sentence_silence	Integer	No	The silence threshold for sentence segmentation, in milliseconds. Silence longer than this threshold marks a sentence boundary. Valid values: `200`–`2000`. Default: `800`.
enable_words	Boolean	No	Specifies whether to return word-level information. Default: `false`.
disfluency	Boolean	No	Specifies whether to remove filler words from the transcript. Default: `false`.
speech_noise_threshold	Float	No	The noise threshold, from `-1` to `1`. Values closer to `-1` make noise more likely to be classified as speech. Values closer to `1` make speech more likely to be classified as noise. This is an advanced parameter. Test recognition accuracy after adjusting it.
enable_semantic_sentence_detection	Boolean	No	Specifies whether to enable semantic sentence segmentation. Default: `false`.

{
  "header": {
    "message_id": "05450bf69c53413f8d88aed1ee600001",
    "task_id": "640bc797bb684bd69601856513070001",
    "namespace": "SpeechTranscriber",
    "name": "StartTranscription",
    "appkey": "<your_appkey>"
  },
  "payload": {
    "format": "pcm",
    "sample_rate": 16000,
    "enable_intermediate_result": true,
    "enable_punctuation_prediction": true,
    "enable_inverse_text_normalization": true
  }
}

StopTranscription command

Notifies the server that all audio has been sent and requests that transcription stop. This command does not require a payload.

{
  "header": {
    "message_id": "05450bf69c53413f8d88aed1ee600002",
    "task_id": "640bc797bb684bd69601856513070001",
    "namespace": "SpeechTranscriber",
    "name": "StopTranscription",
    "appkey": "<your_appkey>"
  }
}

Events

The server returns events as JSON text frames. The header identifies the task, event name, and status. The payload contains event-specific data. In a successful response, header.status is 20000000 and header.status_text contains the status message. The following examples omit some fields. Text and timestamps are illustrative.

TranscriptionStarted event

Indicates that the server is ready to receive audio. Wait for this event before sending binary audio frames. Use header.task_id to identify the task; do not depend on the presence of payload.session_id.

{
  "header": {
    "namespace": "SpeechTranscriber",
    "name": "TranscriptionStarted",
    "task_id": "640bc797bb684bd69601856513070001",
    "status": 20000000,
    "status_text": "Gateway:SUCCESS:Success."
  }
}

SentenceBegin event

Indicates that the server has detected the start of a sentence.

Payload parameter	Type	Description
index	Integer	The sentence index, starting from `1`.
time	Integer	The sentence start time relative to the start of the audio stream, in milliseconds.

{
  "header": {"name": "SentenceBegin", "status": 20000000},
  "payload": {"index": 1, "time": 0}
}

TranscriptionResultChanged event

Returned when an intermediate result changes, if enable_intermediate_result is enabled.

Payload parameter	Type	Description
index	Integer	The sentence index, starting from `1`.
time	Integer	The duration of audio processed so far, in milliseconds.
result	String	The current intermediate recognition result.
words	`Array<Word>`	An array of word information. Word information is returned when `enable_words` is enabled.
status	Integer	The sentence-level status code, if returned. Handle it separately from `header.status`.

Word structure

Parameter	Type	Description
text	String	The word text.
startTime	Integer	The word start time, in milliseconds.
endTime	Integer	The word end time, in milliseconds.

Example

{
  "header": {"name": "TranscriptionResultChanged", "status": 20000000},
  "payload": {
    "index": 1,
    "time": 1000,
    "result": "The weather",
    "words": [
      {"text": "The", "startTime": 0, "endTime": 500},
      {"text": "weather", "startTime": 500, "endTime": 1000}
    ]
  }
}

SentenceEnd event

Indicates that the server has detected the end of a sentence.

Payload parameter	Type	Description
index	Integer	The sentence index, starting from `1`.
time	Integer	The duration of audio processed so far, in milliseconds.
begin_time	Integer	The time of the corresponding `SentenceBegin` event, in milliseconds.
result	String	The final recognition result for the sentence.
confidence	Double	The confidence score, from `0.0` to `1.0`. Higher values indicate greater confidence.
words	`Array<Word>`	An array of word information. Word information is returned when `enable_words` is enabled.
status	Integer	The sentence-level status code. A normal recognition result can return `0`. Do not confuse this field with `header.status`.
stash_result	StashResult	The stashed result. With semantic sentence segmentation enabled, it can contain an intermediate result for the next sentence before a boundary is detected. The text can also be empty.

StashResult structure

Parameter	Type	Description
sentenceId	Integer	The sentence index, starting from `1`.
beginTime	Integer	The sentence start time.
text	String	The stashed transcript.
currentTime	Integer	The current processing time.

Example

{
  "header": {"name": "SentenceEnd", "status": 20000000},
  "payload": {
    "index": 1,
    "time": 1500,
    "begin_time": 0,
    "result": "The weather is nice today."
  }
}

TranscriptionCompleted event

After receiving StopTranscription, the server finishes transcription and returns this event.

{
  "header": {
    "namespace": "SpeechTranscriber",
    "name": "TranscriptionCompleted",
    "task_id": "640bc797bb684bd69601856513070001",
    "status": 20000000,
    "status_text": "Gateway:SUCCESS:Success."
  }
}

TaskFailed event

If a request fails, inspect header.status and header.status_text. For example, an invalid message ID returns 40000002. This is an application-level error code, not a WebSocket Close frame status code.

FAQ

How do I send audio, and how large should each chunk be?

Send audio as binary frames, not in the JSON payload of a command. Size chunks by audio duration. For example, 100 ms of 16-bit mono PCM audio is 3200 bytes at 16000 Hz or 1600 bytes at 8000 Hz. These are not the only valid chunk sizes. Match the sending rate to the audio duration.

Why is Invalid message id returned?

Check that message_id contains 32 hexadecimal characters. A 32-character string containing non-hexadecimal characters is rejected. The task_id must also meet the format requirements.

How do I troubleshoot a connection that closes after audio is sent?

Record the TaskFailed event, task_id, status code, and error message returned by the server. Then check that the token is valid, request fields are correct, the audio format is supported, and the client continues sending audio according to the protocol. If no error event arrives, inspect the client's error and connection-close logs. Setting status on the client does not change the server's result.

How do I generate message_id and task_id?

Generate unique IDs of 32 hexadecimal characters on the client. Keep the same task_id throughout a recognition task, and generate a new message_id for each command.

How do I continuously send real-time audio?

Continuously capture and send audio chunks while receiving server events. After stopping capture and sending all remaining audio, send StopTranscription. For code that simulates real-time streaming from a local file, see Java SDK.