Sambert speech synthesis Python SDK

Updated at:

This topic describes the parameters and interfaces of the Sambert speech synthesis Python SDK.

ImportantAlibaba Cloud Model Studio has released a workspace-specific domain for the China (Beijing) region. The new dedicated domain delivers superior performance and higher stability for inference requests. We recommend migrating from dashscope.aliyuncs.com to {WorkspaceId}.cn-beijing.maas.aliyuncs.com.

Replace {WorkspaceId} with your actual Workspace ID. The existing domain remains fully functional.

User guide: For more information about model introductions and selection suggestions, see Real-time Speech Synthesis - CosyVoice/Sambert.

Online experience: Not supported.

Prerequisites

You have activated the service and obtained and configured an API key. To prevent security risks from code leaks, set the API key as an environment variable instead of hard-coding it in your code.

Getting started

The core resource (client.audio.tts) provides interfaces for non-streaming calls and one-way streaming calls. Choose the appropriate method based on your scenario:

  • Non-streaming call: Submit the text. The server processes it and returns the complete speech synthesis result. The entire process is blocking, which means the client must wait for the server to finish before it can proceed. This method is suitable for short text synthesis.
  • One-way streaming call: Send the text to the server in a single request and receive the speech synthesis result in real time. You cannot send the text in segments. This method is suitable for scenarios that require low latency.

Non-streaming call

Submit a single speech synthesis task for speech synthesis (intermediate results are not streamed) and retrieve the complete result at once.

image

Call the synthesize method of the core resource (client.audio.tts) to synthesize speech. You can use the synthesize method to set request parameters.

After the task is complete, the method returns the complete audio data (bytes). Timestamp information is available only through a streaming session. For more information, see Response results.

Click to view the complete example

The following example shows how to use the synchronous interface to call the Zhichu (sambert-zhichu-v1) model to synthesize the text "How is the weather today?". The synthesized audio is in the WAV format with a sample rate of 48 kHz and is saved to a file named output.wav.

# coding=utf-8
import os
import sys

import dashscope

# If you do not configure the API key as an environment variable, replace the following line with your API key: api_key="sk-xxx"
# The following configuration is for the China (Beijing) region. Replace "{WorkspaceId}" with your actual workspace ID when you call the API. The configuration varies by region.
client = dashscope.Client(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    workspace="{WorkspaceId}",
    region="cn-beijing",
)
audio = client.audio.tts.synthesize(
    'How is the weather today?',
    model='sambert-zhichu-v1',
    sample_rate=48000,
    format='wav')
with open('output.wav', 'wb') as f:
    f.write(audio)
print('SUCCESS: get audio data: %dbytes in output.wav' % (sys.getsizeof(audio)))

One-way streaming call

Submit a single speech synthesis task. Intermediate results are streamed back, and the synthesis results are retrieved by iterating events()/audio_chunks() of the streaming synthesis session.

image
  1. Call the connect method of the core resource (client.audio.tts) to create a streaming synthesis session. The connection is automatically established when the with context is entered.
  2. Call the send_text method of the session to send the text to be synthesized, and call the stop method to notify the server that the input is complete.
  3. In the receiving thread, iterate events() (audio bytes and JSON events, including timestamps) or audio_chunks() (audio only) of the session to retrieve the synthesis results.

Click to view the complete example

The following example shows how to use the streaming interface to call the Zhichu (sambert-zhichu-v1) model. It synthesizes the text "How is the weather today?" into streaming audio in the default WAV format with a 48 kHz sample rate, and retrieves the corresponding timestamp.

# coding=utf-8

import os
import sys

import dashscope

# If you do not configure the API key as an environment variable, replace the following line with your API key: api_key="sk-xxx"
# The following configuration is for the China (Beijing) region. Replace "{WorkspaceId}" with your actual workspace ID when you call the API. The configuration varies by region.
client = dashscope.Client(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    workspace="{WorkspaceId}",
    region="cn-beijing",
)

# Create a streaming synthesis session: the connection is automatically established when the with context is entered
with client.audio.tts.connect(
        model='sambert-zhichu-v1',
        sample_rate=48000,
        parameters={'word_timestamp_enabled': True,
                    'phoneme_timestamp_enabled': True},
) as session:
    session.send_text('How is the weather today?')
    # Notify the server that the input is complete
    session.stop()
    # events() yields both audio bytes and server JSON events (including timestamps)
    for event in session.events():
        if isinstance(event, bytes):
            print('audio result length:', sys.getsizeof(event))
        else:
            output = (event.get('payload') or {}).get('output') or {}
            if output.get('timestamp') is not None:
                print('timestamp result:', str(output.get('timestamp')))

Request parameters

You can set the request parameters in the synthesize, stream, and connect methods of the core resource (client.audio.tts). Named parameters are passed directly, and word_timestamp_enabled and phoneme_timestamp_enabled are passed in the parameters dictionary.

ParameterTypeDefault valueRequiredDescription

model

str

Yes

The name of the voice model to use for speech synthesis. For a complete list, see Model list.

text

str

Yes

The text to be synthesized. The text must be UTF-8 encoded and cannot be empty.

Maximum number of characters: 10,000.

Character calculation rule: One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.

The SSML format is supported. For more information, see Introduction to SSML.

format

str

wav

No

The encoding format of the synthesized audio. The pcm, wav, and mp3 formats are supported.

sample_rate

int

16000

No

The sample rate of the synthesized audio in Hz. We recommend that you use the model's default sample rate (see the Model list). If there is a mismatch, the service performs the necessary upsampling or downsampling.

volume

int

50

No

The volume of the synthesized audio. Valid values: 0 to 100.

rate

float

1.0

No

The speech rate of the synthesized audio. Valid values: 0.5 to 2.

  • 0.5: indicates 0.5 times the default speech rate.
  • 1: indicates the default speech rate. The default speech rate is the synthesis rate of the default output of the model. The speech rate may vary slightly depending on the speaker. The rate is about four characters per second.
  • 2: indicates twice the default speech rate.

pitch

float

1.0

No

The pitch of the synthesized audio. Valid values: 0.5 to 2.

word_timestamp_enabled

bool

False

No

Specifies whether to enable word-level timestamps. The default value is false.

phoneme_timestamp_enabled

bool

False

No

Specifies whether to generate phoneme-level timestamp information based on word-level timestamps (word_timestamp_enabled). The default value is false.

Key interfaces

Core resource (client.audio.tts)

The speech synthesis capability is mounted on the client.audio.tts resource: synthesize returns the complete audio at once, stream yields the audio chunks, and connect creates a streaming synthesis session. The key interfaces are as follows:

Interface/MethodParametersReturn valueDescription
def synthesize(self,
               text: str,
               *,
               model: str,
               voice: str = "longxiaochun",
               format: str = "mp3",
               sample_rate: int = 22050,
               volume: int = None,
               rate: float = None,
               pitch: float = None,
               parameters: Dict = None,
               ...) -> bytes
  • text: The text to be synthesized.
  • model: The name of the model. For more information, see Model list.
  • The other parameters are request parameters, such as format.

The complete audio data (bytes)

Starts a speech synthesis task and returns the complete audio data after the synthesis is complete.

You can use the synthesize method to set the request parameters.

def stream(self,
           text: str,
           *,
           model: str,
           ...) -> Iterator[bytes]

Same as synthesize

The audio data yielded chunk by chunk (Iterator[bytes])

Starts a speech synthesis task, sends the text in a single request, and yields the audio chunks one by one. It yields only audio and does not expose per-event JSON metadata. To obtain timestamps, use the connect session.

def connect(self,
            *,
            model: str,
            format: str = "mp3",
            sample_rate: int = 22050,
            parameters: Dict = None,
            ...) -> SynthesisSession

The SynthesisSession streaming synthesis session

Creates a streaming synthesis session. The connection is automatically established when the with context is entered. Use it together with send_text, stop, and events()/audio_chunks(). For more information, see Event iteration.

Event iteration (events()/audio_chunks())

When you make a one-way streaming call, the server returns key process information and data to the client as events. In the receiving thread, iterate events() or audio_chunks() of the session to process the information and data returned by the server.

Click to view the example

# events() yields both audio bytes and server JSON events (including timestamps)
for event in session.events():
    if isinstance(event, bytes):
        print('Received binary audio data:', event)
    else:
        output = (event.get('payload') or {}).get('output') or {}
        if output.get('timestamp') is not None:
            print('Received timestamp data:', str(output.get('timestamp')))

# Alternatively, iterate only the audio (the session still updates last_response and usage)
for chunk in session.audio_chunks():
    print('Received binary audio data:', chunk)
Interface/MethodParametersReturn valueDescription
def send_text(self, text: str) -> None

text: The text to be synthesized

None

Sends the text to be synthesized. Sending the text and receiving the results must be performed concurrently.

def stop(self) -> None

None

None

Notifies the server that the input is complete. Then, continue to consume events()/audio_chunks() until all the remaining audio is returned.

def events(self) -> Generator[Union[Dict, bytes], None, None]

None

Dict (JSON event) or bytes (audio)

Yields items as the server returns the synthesis data: audio as bytes frames and metadata (such as timestamps) as JSON event dict objects. Only one reader of events() or audio_chunks() is allowed per session.

def audio_chunks(self) -> Iterator[bytes]

None

bytes

Iterates only the audio data. The session still updates its last_response and usage.

session.first_package_delay

None

float

The delay (in milliseconds) from sending the text to receiving the first audio data chunk. Read this property after the task is complete.

Response results

Synthesis result (audio data and timestamps)

synthesize returns the complete audio bytes. In streaming calls, you receive the audio bytes chunk by chunk through stream or the audio_chunks() method of the session. Timestamp information is retrieved from the JSON events yielded by events() of the session (payload.output.timestamp).

Interface/MethodParametersReturn valueDescription
audio = client.audio.tts.synthesize(...)

None

The complete binary audio data

In non-streaming synthesis, obtains the complete binary audio data.

for chunk in session.audio_chunks():

None

The current segment of synthesized binary audio data

In streaming synthesis, receives the synthesized audio frame data chunk by chunk.

output = (event.get('payload') or {}).get('output') or {}
output.get('timestamp')

None

The currently synthesized sentence's timestamp information

In streaming synthesis, obtains the timestamp information of the currently synthesized sentence from the JSON events yielded by events().

timestamps = [
    ((e.get('payload') or {}).get('output') or {}).get('timestamp')
    for e in json_events if ...
]

None

The timestamp information of all sentences

In streaming synthesis, accumulating the timestamps of the sentences yielded by events() gives the timestamp information of all sentences.

Timestamp information

In the JSON events yielded by events() of the session, payload.output.timestamp is the timestamp information of the currently synthesized sentence. Accumulating the timestamps of all sentences gives the timestamp information of all sentences.

The following code provides an example of the timestamp information for a single sentence. words corresponds to word-level timestamp information, and phonemes corresponds to phoneme-level timestamp information:

{
    "begin_time":0,
    "end_time":1412,
    "words":[
        {
            "text":"jin",
            "begin_time":0,
            "end_time":200,
            "phonemes":[
                {
                    "begin_time":0,
                    "end_time":82,
                    "text":"j_c",
                    "tone":1
                },
                {
                    "begin_time":82,
                    "end_time":200,
                    "text":"in_c",
                    "tone":1
                }
            ]
        }
    ]
}

The parameters are as follows:

Parameter

Type

Description

begin_time

int

The start time of a sentence, word, or phoneme in milliseconds.

end_time

int

The end time of a sentence, word, or phoneme in milliseconds.

words

list

The word-level timestamp information. The word_timestamp_enabled parameter in the request must also be set to true.

text

str

The text information.

phonemes

list

The phoneme-level timestamp information. The phoneme_timestamp_enabled parameter in the request must also be set to true.

tone

str

The tone.

  • In English, 0, 1, and 2 represent unstressed, primary stress, and secondary stress, respectively.

  • In Pinyin, 1, 2, 3, 4, and 5 represent the first, second, third, fourth, and neutral tones, respectively.

Error codes

If an API call fails and returns an error message, see Error codes to resolve the issue.

More examples

For more examples, see GitHub.

FAQ

For more information, see the QA on GitHub.

Model list

NoteThe default sample rate is the optimal sample rate for the current model. By default, the output is based on this sample rate. Downsampling and upsampling are also supported. For example, for the Zhimiao voice, the default sample rate is 16 kHz. You can downsample it to 8 kHz, but upsampling it to 48 kHz does not provide additional benefits.

TimbreAudio sample (Right-click to save the audio)model parameterTimestamp supportScenariosFeaturesLanguageDefault sample rate (Hz)

Zhinan

sambert-zhinan-v1

Yes

General scenarios

Advertising male voice

Chinese and English

48 kHz

Zhiqi

sambert-zhiqi-v1

Yes

General scenarios

Gentle female voice

Chinese and English

48 kHz

Zhichu

sambert-zhichu-v1

Yes

News broadcasting

A Bite of China male voice

Chinese and English

48 kHz

Zhide

sambert-zhide-v1

Yes

News broadcasting

News male voice

Chinese and English

48 kHz

Zhijia

sambert-zhijia-v1

Yes

News broadcasting

Standard female voice

Chinese and English

48 kHz

Zhiru

sambert-zhiru-v1

Yes

News broadcasting

News female voice

Chinese and English

48 kHz

Zhiqian

sambert-zhiqian-v1

Yes

Dubbing and news broadcasting

Information female voice

Chinese and English

48 kHz

Zhixiang

sambert-zhixiang-v1

Yes

Dubbing

Magnetic male voice

Chinese and English

48 kHz

Zhiwei

sambert-zhiwei-v1

Yes

Reading product introductions

Lolita female voice

Chinese and English

48 kHz

Zhihao

sambert-zhihao-v1

Yes

General scenarios

Consultation male voice

Chinese and English

16 KB

Zhijing

sambert-zhijing-v1

Yes

General scenarios

Strict female voice

Chinese and English

16 k

Zhiming

sambert-zhiming-v1

Yes

General scenarios

Humorous male voice

Chinese and English

16 KB

Zhimo

sambert-zhimo-v1

Yes

General scenarios

Emotional male voice

Chinese and English

16 kHz

Zhina

sambert-zhina-v1

Yes

General scenarios

Zhejiang Mandarin female voice

Chinese and English

16 kHz

Zhishu

sambert-zhishu-v1

Yes

General scenarios

Information male voice

Chinese and English

16 KB

Zhisha

sambert-zhistella-v1

Yes

General scenarios

Intellectual female voice

Chinese and English

16 kHz

Zhiting

sambert-zhiting-v1

Yes

General scenarios

Radio female voice

Chinese and English

16 kHz

Zhixiao

sambert-zhixiao-v1

Yes

General scenarios

Information female voice

Chinese and English

16 kHz

Zhiya

sambert-zhiya-v1

Yes

General scenarios

Strict female voice

Chinese and English

16 kHz

Zhiye

sambert-zhiye-v1

Yes

General scenarios

Young male voice

Chinese and English

16 kHz

Zhiying

sambert-zhiying-v1

Yes

General scenarios

Cute child voice

Chinese and English

16 kHz

Zhiyuan

sambert-zhiyuan-v1

Yes

General scenarios

Caring sister voice

Chinese and English

16 kHz

Zhiyue

sambert-zhiyue-v1

Yes

Customer service

Gentle female voice

Chinese and English

16 kHz

Zhigui

sambert-zhigui-v1

Yes

Reading product introductions

Livestreaming female voice

Chinese and English

16 kHz

Zhishuo

sambert-zhishuo-v1

Yes

Digital human

Natural male voice

Chinese and English

16 kHz

Zhimiao (multi-emotional)

sambert-zhimiao-emo-v1

Yes

Reading product introductions, digital humans, and livestreaming

Multi-emotional female voice

Chinese and English

16 kHz

Zhimao

sambert-zhimao-v1

Yes

Reading product introductions, dubbing, digital humans, and livestreaming

Livestreaming female voice

Chinese and English

16 kHz

Zhilun

sambert-zhilun-v1

Yes

Dubbing

Suspense narration

Chinese and English

16 kHz

Zhifei

sambert-zhifei-v1

Yes

Dubbing

Passionate narration

Chinese and English

16 KB

Zhida

sambert-zhida-v1

Yes

News broadcasting

Standard male voice

Chinese and English

16 KB

Camila

sambert-camila-v1

No

General scenarios

Spanish female voice

Spanish

16 kHz

Perla

sambert-perla-v1

No

General scenarios

Italian female voice

Italian

16 k

Indah

sambert-indah-v1

No

General scenarios

Indonesian female voice

Indonesian

16 kHz

Clara

sambert-clara-v1

No

General scenarios

French female voice

French

16 kHz

Hanna

sambert-hanna-v1

No

General scenarios

German female voice

German

16 kHz

Beth

sambert-beth-v1

Yes

General scenarios

Consultation female voice

American English

16 kHz

Betty

sambert-betty-v1

Yes

General scenarios

Customer service female voice

American English

16 kHz

Cally

sambert-cally-v1

Yes

General scenarios

Natural female voice

American English

16 kHz

Cindy

sambert-cindy-v1

Yes

General scenarios

Conversational female voice

American English

16 kHz

Eva

sambert-eva-v1

Yes

General scenarios

Companion female voice

American English

16 KB

Donna

sambert-donna-v1

Yes

General scenarios

Education female voice

American English

16 kHz

Brian

sambert-brian-v1

Yes

General scenarios

Customer service male voice

American English

16 kHz

Waan

sambert-waan-v1

No

General scenarios

Thai female voice

Thai

16 kHz