Python SDK for real-time speech recognition-Intelligent Speech Interaction(ISI)-阿里云帮助中心

The Python SDK sends audio streams to the real-time speech recognition service and returns intermediate and sentence recognition results through callbacks.

Prerequisites

Prepare a Python 3 environment. The SDK does not support Python 2.
Obtain a project Appkey and an access token. For information about obtaining an Appkey and configuring a model, see Manage projects. For information about obtaining a token, see Obtain an access token.

For the protocol and audio requirements, see the real-time speech recognition API reference.

Download and install

Obtain the source code from GitHub, or download and extract the NLS Python SDK package.
Install the setuptools build tool.
```
python -m pip install setuptools
```

Install the dependencies and the SDK in the SDK root directory.

python -m pip install -r requirements.txt
python -m pip install .

Import the SDK in your code.
```
import nls
```

Code example

This example uses Python 3.12 and SDK package version 1.1.0 to send audio from tests/test1.pcm in the SDK package to the real-time speech recognition service.

The audio is 16000 Hz, 16-bit, mono PCM. Configure the General model for the project associated with the Appkey. For other audio files, choose a model that matches the language, sample rate, and use case. For configuration instructions, see Manage projects.
Set the NLS_APP_KEY and NLS_TOKEN environment variables to the project Appkey and access token, respectively. The example reads these two variables.
Set AUDIO_FILE to the local path of test1.pcm.

The example accesses the service over the Internet. To use the internal network from an ECS instance in the China (Shanghai) region, set URL to ws://nls-gateway-cn-shanghai-internal.aliyuncs.com/ws/v1.

import json
import os
import threading
import time

import nls

APPKEY = os.environ["NLS_APP_KEY"]
TOKEN = os.environ["NLS_TOKEN"]
URL = "wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1"
AUDIO_FILE = "tests/test1.pcm"

ready = threading.Event()
finished = threading.Event()

def on_start(message, *args):
    ready.set()

def on_result_changed(message, *args):
    result = json.loads(message)
    print("Intermediate result:", result["payload"]["result"])

def on_sentence_end(message, *args):
    result = json.loads(message)
    print("Sentence result:", result["payload"]["result"])

def on_completed(message, *args):
    finished.set()

def on_error(message, *args):
    print("Recognition failed:", message)
    finished.set()

def on_close(*args):
    finished.set()

nls.enableTrace(False)  # Disable SDK debug logging

with open(AUDIO_FILE, "rb") as audio:
    transcriber = nls.NlsSpeechTranscriber(
        url=URL,
        appkey=APPKEY,
        token=TOKEN,
        on_start=on_start,
        on_result_changed=on_result_changed,
        on_sentence_end=on_sentence_end,
        on_completed=on_completed,
        on_error=on_error,
        on_close=on_close,
    )
    try:
        transcriber.start(
            aformat="pcm",
            sample_rate=16000,
            enable_intermediate_result=True,
            enable_punctuation_prediction=True,
            enable_inverse_text_normalization=True,
        )
        if ready.is_set() and not finished.is_set():
            while not finished.is_set():
                chunk = audio.read(640)
                if not chunk:
                    break
                transcriber.send_audio(chunk)
                time.sleep(len(chunk) / (16000 * 2))
            if not finished.is_set():
                transcriber.stop()
    finally:
        transcriber.shutdown()

SDK reference

The real-time speech recognition client is nls.NlsSpeechTranscriber. The following descriptions apply to SDK package version 1.1.0.

Initialization (init)

Create a client and configure credentials, the gateway, and callbacks.

Parameter	Type	Description
`url`	str	The gateway WebSocket URL. Default: `wss://nls-gateway.cn-shanghai.aliyuncs.com/ws/v1`.
`appkey`	str	The project Appkey. For information about obtaining an Appkey, see Manage projects.
`token`	str	The access token. For information about obtaining a token, see Obtain an access token.
`on_start`	callable	Called when `TranscriptionStarted` is received, indicating that the recognition task is ready.
`on_sentence_begin`	callable	Called when `SentenceBegin` is received, indicating that recognition of a sentence has begun.
`on_sentence_end`	callable	Called when `SentenceEnd` is received. The message contains the complete recognition result for the sentence in `payload.result`.
`on_result_changed`	callable	Called when `TranscriptionResultChanged` is received. The message contains an intermediate recognition result. Set `enable_intermediate_result` to `True` in `start` to enable this callback.
`on_completed`	callable	Called when `TranscriptionCompleted` is received, indicating that the entire recognition task has completed. Sentence results are returned through `on_sentence_end`.
`on_error`	callable	Called when a `TaskFailed` error message is received from the service.
`on_close`	callable	Called when the connection to the service closes.
`callback_args`	list	A list of custom arguments. The elements are expanded in order and passed to each callback as `*args`.

Except for on_close, each callback receives a JSON-formatted message string as its first argument, followed by the expanded callback_args. The on_close callback receives only the custom arguments. For example, with callback_args=["first", "second"], message callbacks receive callback(message, "first", "second"), and the close callback receives on_close("first", "second").

Client parameter validation errors and connection or call timeouts may also be reported as exceptions. Handle these exceptions in the calling code.

start

Start real-time speech recognition and synchronously wait for the task to be ready. The on_start callback is invoked when recognition is ready, and on_error is invoked if the service reports a task failure. Send audio after the task is ready.

Parameter	Type	Description
`aformat`	str	The audio format. Default: `pcm`. PCM, OPUS, and OPU are supported. For other formats and audio requirements, see the real-time speech recognition API reference. The SDK does not convert audio encodings. Encode audio into OPUS or OPU before sending it.
`sample_rate`	int	The audio sample rate. Default: 16000 Hz.
`ch`	int	The number of audio channels. Default: `1`. Only mono audio is supported.
`enable_intermediate_result`	bool	Specifies whether to return intermediate results. Default: `False`.
`enable_punctuation_prediction`	bool	Specifies whether to add punctuation to recognition results. Default: `False`.
`enable_inverse_text_normalization`	bool	Specifies whether to enable inverse text normalization (ITN) to convert Chinese numerals to Arabic numerals. Default: `False`.
`timeout`	int	The timeout for waiting for the task to be ready. Default: 10 seconds. A timeout raises `StartTimeoutException`.
`ping_interval`	int or float	The interval for sending Ping frames. Default: 8 seconds. Set to `0` or `None` to disable periodic Ping frames.
`ping_timeout`	int or float	The timeout for receiving a Pong after sending a Ping. Default: `None`, which disables the Pong timeout check. When enabled, the value must be positive. If periodic Ping frames are also enabled, this value must be less than `ping_interval`.
`ex`	dict	Additional request parameters. Default: `None`. The dictionary's key-value pairs are merged into the request `payload`. For request parameters, see the real-time speech recognition API reference.

Return value: None. Recognition status and results are returned through callbacks. Parameter or connection errors may raise exceptions. Connection closure or task failure also ends the wait.

stop

Finish sending audio and synchronously wait for the recognition task to end. The on_completed callback is invoked on normal completion. A task failure or connection closure also ends the wait.

Parameter	Type	Description
`timeout`	int	The timeout for waiting for the task to end. Default: 10 seconds. A timeout raises `StopTimeoutException`.

Return value: None.

shutdown

Forcibly close the connection created by start. Calling this method repeatedly on a connection that has been created has no additional effect.

Parameters: None. Return value: None.

send_audio

Send binary audio data in the format specified by aformat in start.

Parameter	Type	Description
`pcm_data`	bytes	The audio data to send. Encode audio into OPUS or OPU before sending it in either format. The SDK does not automatically convert PCM to another format.

Return value: None. Calls are ignored when the task is not ready or has ended. After recognition is ready, send audio continuously at real-time speed.

ctrl

Send a control command using ctrl(**kwargs).

Parameter	Type	Description
`kwargs`	Keyword arguments	Control command parameters, which are merged directly into `payload`. To pass a dictionary, use Python's `**` unpacking syntax.

Call this method after the task is ready. Calling it without arguments raises InvalidParameter. Return value: None.

Multi-threading and concurrency

In CPython builds with the Global Interpreter Lock (GIL) enabled, only one thread executes Python code at a time, although some performance-oriented libraries can bypass this limitation. To make full use of multiple CPU cores, we recommend multiprocessing or concurrent.futures.ProcessPoolExecutor. Multi-threading remains suitable for multiple I/O-bound tasks.

Too many threads in a single interpreter increase thread-switching overhead and may cause SDK errors. We recommend using no more than 200 threads. For more concurrent tasks, use multiple processes or create multiple interpreters through a script.