API reference for the real-time speech recognition mobile SDK-Intelligent Speech Interaction(ISI)-阿里云帮助中心

The Natural User Interaction (NUI) SDK provides real-time speech recognition for mobile clients over a persistent WebSocket connection. It is designed for long-running, uninterrupted recognition scenarios such as conference speeches and live streaming.

Key concepts

Compared to general-purpose SDKs, the NUI SDK is smaller in size and offers more comprehensive status management. It provides a full suite of speech processing capabilities through a unified API and can also function as an atomic SDK to meet diverse integration requirements.

How real-time recognition works:

Submit audio data to the server over WebSocket. The server applies voice activity detection (VAD) to detect sentence boundaries, then streams recognition results back as events. For each sentence:

SentenceBegin fires when the server detects the start of a sentence.
TranscriptionResultChanged fires repeatedly as partial results accumulate (if enabled).
SentenceEnd fires when the sentence is complete, returning the final result.

Note

The server adds task_id to every response header to identify the recognition task. Record this value — you will need it when filing a support ticket.

Features

Accepts pulse-code modulation (PCM) encoded 16-bit mono audio.
Supports audio sampling rates of 8,000 Hz and 16,000 Hz.
Returns intermediate results, adds punctuation, and converts Chinese numerals to Arabic numerals (each configurable).
Supports custom speech models and hotword vocabularies.
Supports linguistic model selection per project in the Intelligent Speech Interaction console. For details, see Manage projects.

Endpoint

Access type	URL
Internet	`wss://nls-gateway-ap-southeast-1.aliyuncs.com/ws/v1`

The Internet endpoint is built into the SDK by default.

Interaction process

Step 1: Authenticate and initialize the SDK

To establish a WebSocket connection, authenticate with a token. For details, see Obtain an access token.

Pass the following parameters when initializing the SDK:

Parameter	Type	Required	Description
`workspace`	String	Yes	The working directory from which the SDK reads the configuration file.
`app_key`	String	Yes	The appkey of the project created in the Intelligent Speech Interaction console.
`token`	String	Yes	The authentication token for Intelligent Speech Interaction. Keep the token valid. You can set it at initialization and update it when setting request parameters.
`device_id`	String	Yes	A unique device identifier, such as a media access control (MAC) address, serial number, or pseudo unique ID.
`debug_path`	String	No	The directory for audio files generated during debugging. Requires `save_log=true` at initialization.
`save_wav`	String	No	Whether to save debug audio files to `debug_path`. Requires `save_log=true` and a writable directory.

Step 2: Set request parameters

Set request parameters using the setParams method in JSON format. These parameters apply to all service requests.

Parameter	Type	Required	Description
`appkey`	String	No	The project appkey. Generally set during SDK initialization.
`token`	String	No	The authentication token. Update as needed.
`service_type`	Int	Yes	The speech service type. Set to `4` for real-time speech recognition.
`direct_ip`	String	No	An IP address resolved from a Domain Name System (DNS) domain name. The client resolves the DNS name and uses the resulting IP address.
`nls_config`	JsonObject	No	Service configuration parameters. See the table below.

Parameters inside `nls_config`:

Parameter	Type	Default	Description
`sr_format`	String	`OPUS`	The audio encoding format. Accepts `OPUS` or `PCM`. Set to `PCM` if `sample_rate` is `8000`.
`sample_rate`	Integer	`16000`	The audio sampling rate, in Hz. After changing this value, update the matching model or scene in the Intelligent Speech Interaction console.
`enable_intermediate_result`	Boolean	`false`	Whether to return intermediate results during recognition.
`enable_punctuation_prediction`	Boolean	`false`	Whether to add punctuation during post-processing.
`enable_inverse_text_normalization`	Boolean	`false`	Whether to enable inverse text normalization (ITN) during post-processing. When `true`, Chinese numerals are converted to Arabic numerals. ITN does not apply to individual words.
`customization_id`	String	—	The ID of a custom speech training model.
`vocabulary_id`	String	—	The vocabulary ID for custom hotwords.
`max_sentence_silence`	Integer	`800`	The silence threshold for sentence-end detection, in milliseconds. Valid values: 200–2000.
`enable_words`	Boolean	`false`	Whether to return word-level timing information.
`enable_ignore_sentence_timeout`	Boolean	`false`	Whether to ignore single-sentence recognition timeouts.
`disfluency`	Boolean	`false`	Whether to enable disfluency detection.
`vad_model`	String	—	The ID of the voice activity detection (VAD) model on the server.
`speech_noise_threshold`	Float	—	The noise detection threshold. Valid values: -1 to 1. Values closer to `-1` treat more audio as valid speech; values closer to `1` treat more audio as noise. Adjust with caution and test after any change.

Step 3: Send audio data

The client cyclically sends audio data to the server. The server streams recognition events back via the onNuiEventCallback method.

SentenceBegin

The server fires SentenceBegin when it detects the start of a sentence (via VAD).

Example response:

{
    "header": {
        "namespace": "SpeechTranscriber",
        "name": "SentenceBegin",
        "status": 20000000,
        "message_id": "a426f3d4618447519c9d85d1a0d1****",
        "task_id": "5ec521b5aa104e3abccf3d361822****",
        "status_text": "Gateway:SUCCESS:Success."
    },
    "payload": {
        "index": 1,
        "time": 0
    }
}

Header fields:

Field	Type	Description
`namespace`	String	The message namespace.
`name`	String	The message name. `SentenceBegin` indicates the server detected a sentence start.
`status`	Integer	The HTTP status code. `20000000` indicates success. For other codes, see Error codes.
`message_id`	String	The message ID, automatically generated by the SDK.
`task_id`	String	The task globally unique identifier (GUID). Record this value to facilitate troubleshooting.
`status_text`	String	The status message.

Payload fields:

Field	Type	Description
`index`	Integer	The sentence sequence number, starting from 1.
`time`	Integer	The duration of processed audio, in milliseconds.

TranscriptionResultChanged

If enable_intermediate_result is true, the server fires EVENT_ASR_PARTIAL_RESULT events repeatedly, delivering partial results via onNuiEventCallback as recognition progresses.

Example response:

{
    "header": {
        "namespace": "SpeechTranscriber",
        "name": "TranscriptionResultChanged",
        "status": 20000000,
        "message_id": "dc21193fada84380a3b6137875ab****",
        "task_id": "5ec521b5aa104e3abccf3d361822****",
        "status_text": "Gateway:SUCCESS:Success."
    },
    "payload": {
        "index": 1,
        "time": 1835,
        "result": "Sky in Beijing",
        "confidence": 1.0,
        "words": [{
            "text": "Sky",
            "startTime": 630,
            "endTime": 930
        }, {
            "text": "in",
            "startTime": 930,
            "endTime": 1110
        }, {
            "text": "Beijing",
            "startTime": 1110,
            "endTime": 1140
        }]
    }
}

The header fields are identical to those in SentenceBegin, with name set to TranscriptionResultChanged.

Payload fields:

Field	Type	Description
`index`	Integer	The sentence sequence number, starting from 1.
`time`	Integer	The duration of processed audio, in milliseconds.
`result`	String	The current recognition result for the sentence.
`confidence`	Double	The recognition confidence score. Valid values: 0.0–1.0. Higher values indicate greater confidence.
`words`	List\<Word\>	Word-level timing data. Returned only when `enable_words` is `true`.

Word fields (`words` object):

Field	Type	Description
`text`	String	The recognized word text.
`startTime`	Integer	The time when the word appears in the sentence. Unit: milliseconds.
`endTime`	Integer	The time when the word ends in the sentence. Unit: milliseconds.

SentenceEnd

The server fires SentenceEnd when it detects the end of a sentence, returning the final recognition result.

Example response:

{
    "header": {
        "namespace": "SpeechTranscriber",
        "name": "SentenceEnd",
        "status": 20000000,
        "message_id": "c3a9ae4b231649d5ae05d4af36fd****",
        "task_id": "5ec521b5aa104e3abccf3d361822****",
        "status_text": "Gateway:SUCCESS:Success."
    },
    "payload": {
        "index": 1,
        "time": 1820,
        "begin_time": 0,
        "result": "Weather in Beijing.",
        "confidence": 1.0,
        "words": [{
            "text": "Weather",
            "startTime": 630,
            "endTime": 930
        }, {
            "text": "in",
            "startTime": 930,
            "endTime": 1110
        }, {
            "text": "Beijing",
            "startTime": 1110,
            "endTime": 1380
        }]
    }
}

The header fields are identical to those in SentenceBegin, with name set to SentenceEnd.

Payload fields:

Field	Type	Description
`index`	Integer	The sentence sequence number, starting from 1.
`time`	Integer	The duration of processed audio, in milliseconds.
`begin_time`	Integer	The time at which the server returned the `SentenceBegin` message for this sentence, in milliseconds.
`result`	String	The final recognition result for the sentence.
`confidence`	Double	The recognition confidence score. Valid values: 0.0–1.0. Higher values indicate greater confidence.
`words`	List\<Word\>	Word-level timing data. Returned only when `enable_words` is `true`. See the word fields table in the TranscriptionResultChanged section.

Step 4: Complete the recognition task

After sending all audio data, the client notifies the server. The server finalizes the recognition task and notifies the client that the task is complete.

Error codes

For error codes returned by the real-time speech recognition service, see Error codes.