How to develop custom code to access Alibaba Cloud Voice Service-Intelligent Speech Interaction(ISI)-阿里云帮助中心

If you prefer not to use the Intelligent Speech Interaction SDK, or if the provided SDKs do not meet your needs, this guide explains how to access the Speech Service directly using the WebSocket protocol.

Overview

The Intelligent Speech Interaction service uses the WebSocket protocol to provide real-time speech-to-text for long-form audio. Commands and events are sent as text data frames, and audio data is streamed as binary data frames. You must follow the protocol's required interaction flow. To send audio data, use WebSocket binary frames. For more information, see Data Frames.

You must send audio data at a real-time or near-real-time rate. Sending large amounts of audio data in a single, rapid burst is not supported. To process offline audio files quickly, use the file transcription API instead.

Supported input format: 16-bit, single-channel (mono) PCM audio, either as uncompressed data or in a WAV file.
Supported audio sample rates: 8000 Hz and 16000 Hz.
Result settings: You can configure the service to return an intermediate result, add punctuation during post-processing, and convert Chinese numerals to Arabic numerals.
You can select a multi-language recognition model when editing a project in the console. For more information, see Manage projects.

Authentication

The service uses a temporary token for authentication. You must include this token as a parameter in the request URL. To get a token, see Get token. Once you have a token, use one of the following URLs to connect to the Speech Service.

Access type	Description	URL
Public access	All servers can use the public access URL.	wss://nls-gateway.cn-shanghai.aliyuncs.com/ws/v1?token=<your token>
Private network access from an ECS instance in China (Shanghai)	You can use this URL to connect from an ECS instance in the China (Shanghai) region. Private network access is supported only for instances within a Virtual Private Cloud (VPC) and is not available for instances in a classic network. Note Private network access does not incur public network traffic fees for your ECS instance. For more information about ECS network types, see Network types.	ws://nls-gateway.cn-shanghai-internal.aliyuncs.com:80/ws/v1?token=<your token>

Interaction flow

To avoid errors, you must send commands and the audio stream in the order shown in the following diagram.

交互过程

Commands

Commands control the start and end of a transcription task and define its boundaries. You send commands as JSON-formatted text frames. Basic request information must be set in the header. Each command consists of a header and a payload. The header has a consistent format, while the payload format varies by command.

1. Header format

The header has the following format:

Parameter	Type	Required	Description
`appkey`	String	Yes	The appkey of the project that you created in the console.
`message_id`	String	Yes	A unique, randomly generated 32-character ID for the message request.
`task_id`	String	Yes	A unique 32-character ID for the entire real-time speech-to-text session. This ID must remain the same for all requests within the same session.
`namespace`	String	Yes	The name of the service being accessed. This must be set to "SpeechTranscriber".
`name`	String	Yes	The name of the command. Valid values are `StartTranscription` and `StopTranscription`. For more information, see StartTranscription command and StopTranscription command.

2. StartTranscription command

Payload parameters:

Parameter	Type	Required	Description
`format`	String	No	The audio encoding format. Supported formats are PCM, OPUS, and OPU. The default is PCM (16-bit, single-channel, uncompressed data or a WAV file).
`sample_rate`	Integer	No	The audio sample rate in Hz. The default is 16000. Ensure that the model configured in your project supports the specified sample rate and scenario.
`enable_intermediate_result`	Boolean	No	Specifies whether to return an intermediate result. The default is `false`.
`enable_punctuation_prediction`	Boolean	No	Specifies whether to add punctuation during post-processing. The default is `false`.
`enable_inverse_text_normalization`	Boolean	No	Specifies whether to perform Inverse Text Normalization (ITN) during post-processing. If set to `true`, Chinese numerals in the output are converted to Arabic numerals. The default is `false`. Note ITN is not applied to word-level information.
`customization_id`	String	No	The ID of the custom model.
`vocabulary_id`	String	No	The ID of the custom vocabulary.
`max_sentence_silence`	Integer	No	The silence duration in milliseconds that the service uses to detect the end of a sentence. The service considers a period of silence longer than this value to be a sentence break. Valid range: 200 to 2000. Default: 800.
`enable_words`	Boolean	No	Specifies whether to return word-level information. The default is `false`.
`enable_ignore_sentence_timeout`	Boolean	No	Specifies whether to ignore sentence-level timeouts during real-time recognition. The default is `false`.
`disfluency`	Boolean	No	Specifies whether to smooth the transcript by removing filler words and repetitions. The default is `false`.
`speech_noise_threshold`	Float	No	The threshold for classifying audio as speech or noise. The valid range is from -1.0 to 1.0. A value closer to -1.0 increases the probability of noise being classified as speech. A value closer to 1.0 increases the probability of speech being classified as noise. Note This is an advanced parameter. Adjust it with caution and test your changes thoroughly.
`enable_semantic_sentence_detection`	Boolean	No	Specifies whether to enable semantic sentence detection. The default is `false`.

Example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "SpeechTranscriber",
        "name": "StartTranscription",
        "appkey": "17d4c634****"
    },
    "payload": {
        "format": "opus",
        "sample_rate": 16000,
        "enable_intermediate_result": true,
        "enable_punctuation_prediction": true,
        "enable_inverse_text_normalization": true
    }
}

3. StopTranscription command

The StopTranscription command instructs the server to stop the transcription task. Its payload is empty. Example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "SpeechTranscriber",
        "name": "StopTranscription",
        "appkey": "17d4c634****"
    }
}

Events

1. TranscriptionStarted event

The TranscriptionStarted event indicates the server is ready for transcription. You can now start sending audio data.

Parameter	Type	Description
`session_id`	String	If the client provides a `session_id` in the request, the server returns it. Otherwise, the server generates a unique 32-character ID.

Example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "SpeechTranscriber",
        "name": "TranscriptionStarted",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    },
    "payload": {
        "session_id": "1231231dfdf****"
    }
}

2. SentenceBegin event

The SentenceBegin event indicates the start of a sentence.

Parameter	Type	Description
`index`	Integer	The sequential index of the sentence, starting from 1.
`time`	Integer	The start time of the sentence, in milliseconds, relative to the beginning of the audio stream.

Example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "SpeechTranscriber",
        "name": "SentenceBegin",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    },
    "payload": {
        "index": 1,
        "time": 320
    }
}

3. TranscriptionResultChanged event

The TranscriptionResultChanged event indicates a change in the intermediate recognition result.

Parameter	Type	Description
`index`	Integer	The sequential index of the sentence, starting from 1.
`time`	Integer	The duration of audio processed so far, in milliseconds.
`result`	String	The current intermediate transcription result for the sentence.
`words`	Word	An array of objects that contains word-level information.
`status`	Integer	The status code.

Word structure:

Parameter	Type	Description
`text`	String	The transcribed word.
`startTime`	Integer	The start time of the word.
`endTime`	Integer	The end time of the word.

Example:

{
    "header":{
        "message_id":"05450bf69c53413f8d88aed1ee60****",
        "task_id":"640bc797bb684bd6960185651307****",
        "namespace":"SpeechTranscriber",
        "name":"TranscriptionResultChanged",
        "status":20000000,
        "status_message":"GATEWAY|SUCCESS|Success."
    },
    "payload":{
        "index":1,
        "time":1800,
        "result":"good morning",
        "words":[
            {
                "text":"good",
                "startTime":1,
                "endTime":2
            },
            {
                "text":"morning",
                "startTime":2,
                "endTime":3
            }
        ]
    }
}

4. SentenceEnd event

The SentenceEnd event indicates the end of a sentence.

Parameter	Type	Description
`index`	Integer	The sequential index of the sentence, starting from 1.
`time`	Integer	The duration of the audio, in milliseconds, that has been processed so far.
`begin_time`	Integer	The timestamp, in milliseconds, of the corresponding `SentenceBegin` event for this sentence.
`result`	String	The final transcription result for the sentence.
`confidence`	Double	The confidence score for the result, ranging from 0.0 to 1.0. Higher values indicate greater confidence.
`words`	Word	An array of objects that contains word-level information.
`status`	Integer	The default status code is 20000000. If `enable_ignore_sentence_timeout` is enabled and a timeout occurs, the system returns error code 51040104 and maintains the current connection.
`stash_result`	StashResult	The stashed result. If semantic sentence detection is enabled, this field contains the unfinalized intermediate result for the next sentence.

StashResult structure:

Parameter	Type	Description
`sentenceId`	Integer	The sequential index of the sentence, starting from 1.
`beginTime`	Integer	The start time of the sentence.
`text`	String	The transcribed content.
`currentTime`	Integer	The timestamp of the last processed audio chunk.

Example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "SpeechTranscriber",
        "name": "SentenceEnd",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    },
    "payload": {
        "index": 1,
        "time": 3260,
        "begin_time": 1800,
        "result": "good morning everyone"
    }
}

5. TranscriptionCompleted event

The TranscriptionCompleted event indicates that the transcription task is complete. Example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "SpeechTranscriber",
        "name": "TranscriptionCompleted",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    }
}