WebSocket protocol

Prerequisites

Before you connect using the WebSocket protocol, see the API reference.

Authentication

The service authenticates requests using a temporary token. You must include the token as a parameter in the request URL. For more information about how to obtain a token, see Obtain a token. After you obtain the token, you can access the Voice Service using one of the following URLs.

Request instructions

Request instructions control the start and end of a speech synthesis task and define its boundaries. You can send instructions to the service as a JSON-formatted text frame. The basic request information is set in the header. An instruction consists of a Header and a Payload. The Header has a standard format, but the Payload format varies depending on the instruction.

1. Header format

The header has the following format:

2. StartSynthesis instruction

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "StartSynthesis",
        "appkey": "17d4c634****"
    },
    "payload": {
        "voice": "xiaoyun",
        "format": "wav",
        "sample_rate": 16000,
        "volume": 50,
        "speech_rate": 0,
        "pitch_rate": 0,
        "enable_subtitle": true
    }
}

3. RunSynthesis instruction

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "RunSynthesis",
        "appkey": "17d4c634****"
    },
    "payload": {
        "text": "Streaming input text"
    }
}

4. StopSynthesis instruction

The `StopSynthesis` instruction tells the service to stop speech synthesis and synthesize any cached text.

The Payload is empty. The following code provides an example:

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "StopSynthesis",
        "appkey": "17d4c634****"
    }
}

Downstream data

WebSocket data frames include text frames, binary frames, close frames, Ping frames, and Pong frames. The service uses text frames to send events and binary frames to send the audio data stream.

The following example code for `websocket-client` in Python shows how to parse data received from the WebSocket:

audio_data = None


# Callback function for the message listener
def on_message(self, ws, message):
    if isinstance(message, str):
        # Parse the text frame as JSON
        try:
            json_data = json.loads(message)
            # TODO: Parse the event
        except json.JSONDecodeError:
            print("Failed to parse message as JSON.")
    elif isinstance(message, (bytes, bytearray)):
        # Save the binary frame as an audio frame
        # TODO: Save the audio or play it using a player that supports streaming input, such as pyaudio
        if audio_data is None:
            audio_data = bytes(message)
        else:
            audio_data = self._audio_data + bytes(message)

ws = websocket.WebSocketApp(
    url,
    header={
        "X-NLS-Token": token,
    },
    on_message=on_message,
    on_error=None,
    on_close=None,
)

For more information about WebSocket, see the documentation at this link.

Events

Events are progress notifications sent from the service to the client. They represent different processing stages. The client can use these events to implement various types of business logic. Events are returned in JSON format and consist of a Header and a Payload. The Header has a standard format, but the Payload format may vary depending on the event.

1. SynthesisStarted event

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "SynthesisStarted",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    }
}

2. SentenceBegin event

The `SentenceBegin` event indicates that the service has detected the beginning of a sentence.

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "SentenceBegin",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    },
    "payload": {
        "index": 1
    }
}

3. SentenceSynthesis event

The `SentenceSynthesis` event indicates that a new synthesis result is available. This event contains the latest audio and timestamps. The data is complete for the current sentence and is provided incrementally across multiple sentences.

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "SentenceSynthesis",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    },
    "payload": {
        "subtitles": [
            {
                "text": "",
                "begin_time": 0,
                "end_time": 0,
                "begin_index": 0,
                "end_index": 1,
                "sentence": true,
                "phoneme_list": []
            },
            {
                "text": "jīn",
                "begin_time": 0,
                "end_time": 175,
                "begin_index": 0,
                "end_index": 1,
                "sentence": false,
                "phoneme_list": [
                    {
                        "begin_time": 0,
                        "end_time": 120,
                        "text": "j_c",
                        "tone": "1"
                    },
                    {
                        "begin_time": 120,
                        "end_time": 170,
                        "text": "in_c",
                        "tone": "1"
                    }
                ]
            }
        ]
    }
}

4. SentenceEnd event

The `SentenceEnd` event indicates that the service has detected the end of a sentence. It returns the complete timestamp information for that sentence.

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "SentenceEnd",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    },
    "payload": {
        "subtitles": [
            {
                "text": "",
                "begin_time": 0,
                "end_time": 0,
                "begin_index": 0,
                "end_index": 1,
                "sentence": true,
                "phoneme_list": []
            },
            {
                "text": "jin",
                "begin_time": 0,
                "end_time": 175,
                "begin_index": 0,
                "end_index": 1,
                "sentence": false,
                "phoneme_list": [
                    {
                        "begin_time": 0,
                        "end_time": 120,
                        "text": "j_c",
                        "tone": "1"
                    },
                    {
                        "begin_time": 120,
                        "end_time": 170,
                        "text": "in_c",
                        "tone": "1"
                    }
                ]
            },
            {
                "text": "tian",
                "begin_time": 175,
                "end_time": 320,
                "begin_index": 1,
                "end_index": 2,
                "sentence": false,
                "phoneme_list": [
                    {
                        "begin_time": 0,
                        "end_time": 120,
                        "text": "t_c",
                        "tone": "1"
                    },
                    {
                        "begin_time": 120,
                        "end_time": 170,
                        "text": "ian_c",
                        "tone": "1"
                    }
                ]
            }
        ]
    }
}

5. SynthesisCompleted event

The `SynthesisCompleted` event indicates that the service has stopped speech synthesis and all audio data has been sent.

This event is returned only after you send a `StopSynthesis` instruction.

{
    "header": {
        "message_id": "05450bf69c53413f8d88aed1ee60****",
        "task_id": "640bc797bb684bd6960185651307****",
        "namespace": "FlowingSpeechSynthesizer",
        "name": "SynthesisCompleted",
        "status": 20000000,
        "status_message": "GATEWAY|SUCCESS|Success."
    }
}

Downstream audio stream

In real-time speech synthesis, the audio is sent in frames as a data stream. Together, all the sent audio frames constitute a single, complete audio file. You can play the stream in real time using a player that supports streaming playback, such as FFmpeg, PyAudio (Python), AudioFormat (Java), or MediaSource (JavaScript).

JavaScript code example

For an example of how to implement the real-time speech synthesis protocol and playback in JavaScript, see JS Playback Example for Real-time Speech Synthesis.

Before you open `index.html`, you must replace the `appkey` and `token` in `app.js`. Then, you can run the following Python command to start a simple HTTP server in the current directory. Open http://localhost:8000 in your browser to view the webpage.

python -m http.server 8000 

On the webpage, you can click buttons to send the corresponding instructions. Clicking `RunSynthesis` sends the text from the text box to the server. You can click `RunSynthesis` multiple times in a single session.

Streaming player details

In audio_player.js, the Web Audio API is used to develop the PCMAudioPlayer player, which plays streaming audio in PCM format. This player converts 16-bit sampling points to float values, writes them to an audio buffer for playback, and immediately plays the next audio segment in the `onended` callback after the previous segment finishes playing.

Testing tool

While you develop your interface based on the WebSocket protocol, you can download the stream_input_tts_mock_server.py script. Run the following commands to install dependencies and run a local mock server for testing. The mock server simulates the real-time speech synthesis service of the public cloud.

pip install websocket-client
python NlsStreamInputTtsMockServer.py

After the script runs successfully, a mock service runs by default at `ws://127.0.0.1:12345`. You can test your implementation against the local mock service before you switch to the online service.