Sambert speech synthesis Java SDK-Alibaba Cloud Model Studio(Model Studio)-阿里云帮助中心

This topic describes the parameters and interface details of the Sambert speech synthesis Java software development kit (SDK).

User guide: For more information about model introductions and selection suggestions, see Real-time Speech Synthesis - CosyVoice/Sambert.

Online experience: Not supported.

Service address

You must set the SDK service endpoint to the following address (which includes WorkspaceId) before initialization.

Important

Sambert is only available in the China (Beijing) region.

Important

Alibaba Cloud Model Studio has released a workspace-specific domain for the China (Beijing) region. The new dedicated domain delivers superior performance and higher stability for inference requests. We recommend migrating from dashscope.aliyuncs.com to {WorkspaceId}.cn-beijing.maas.aliyuncs.com.

Replace {WorkspaceId} with your actual Workspace ID. The existing domain remains fully functional.

Service endpoint:

wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference

Replace {WorkspaceId} with your actual Workspace ID.

Prerequisites

You have activated the service and obtained and configured an API key. To prevent security risks from code leaks, set the API key as an environment variable instead of hard-coding it in your code.
Install the latest version of the DashScope SDK.

Getting started

The SpeechSynthesizer class provides interfaces for non-streaming and one-way streaming calls. You can select a call method based on your business scenario:

Non-streaming call: After you submit the text, the server immediately processes it and returns the complete speech synthesis result. The entire process is blocking. The client must wait for the server to finish processing before proceeding to the next operation. This method is suitable for short text synthesis scenarios.
One-way streaming call: You can send the text to the server at one time and receive the speech synthesis results in real time. You cannot send the text in segments. This method is suitable for scenarios that require high real-time performance.

Non-streaming call

You can submit a single speech synthesis task to obtain the complete result at once. This method does not require a callback and does not stream intermediate results.

Instantiate the SpeechSynthesizer class. Call the call method to attach the request parameters. Then, you can perform synthesis and retrieve the binary audio data.

Click to view the complete example

The following example shows how to use the non-streaming interface to call the Zhichu (sambert-zhichu-v1) speaker model to synthesize the text "What's the weather like today?" into audio with a sample rate of 48 kHz in the WAV audio format and save it to a file named output.wav.

import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisParam;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisAudioFormat;
import com.alibaba.dashscope.utils.Constants;

import java.io.*;
import java.nio.ByteBuffer;

public class Main {
    public static void syncAudioDataToFile() {
        SpeechSynthesizer synthesizer = new SpeechSynthesizer();
        SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                // If you do not set the API key as an environment variable, uncomment the following line and replace yourApiKey with your actual API key.
                // .apiKey("yourApiKey")
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .format(SpeechSynthesisAudioFormat.WAV)
                .build();

        File file = new File("output.wav");
        // Submit a non-streaming synthesis task to get the complete audio data.
        ByteBuffer audio = synthesizer.call(param);
        try (FileOutputStream fos = new FileOutputStream(file)) {
            fos.write(audio.array());
            System.out.println("Synthesis done!");
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    public static void main(String[] args) {
        // Replace "{WorkspaceId}" with your actual Workspace ID.
        Constants.baseWebsocketApiUrl = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference";
        syncAudioDataToFile();
        System.exit(0);
    }
}

One-way streaming call

You can submit a single speech synthesis task and stream the intermediate results through a callback. The synthesis results are streamed through the callback method in ResultCallback.

Instantiate the SpeechSynthesizer class. Call the call method to attach the request parameters and the callback interface (ResultCallback) to start speech synthesis. You can retrieve the synthesis results in real time through the onEvent method of the callback interface (ResultCallback).

After speech synthesis is complete (after the onComplete method of the ResultCallback interface is called), you can also call the getAudioData and getTimestamps methods of the SpeechSynthesizer class to retrieve the complete audio and timestamp results at once.

Click to view the complete example

The following example shows how to use the streaming interface to call the Zhichu (sambert-zhichu-v1) speaker model to synthesize the text "What's the weather like today?" into streaming audio with a sample rate of 48 kHz and the default WAV audio format, and obtain the corresponding timestamps.

import com.alibaba.dashscope.audio.tts.SpeechSynthesisParam;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisResult;
import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;
import com.alibaba.dashscope.common.ResultCallback;
import com.alibaba.dashscope.utils.Constants;

import java.util.concurrent.CountDownLatch;

public class Main {
    public static void main(String[] args) {
        // Replace "{WorkspaceId}" with your actual Workspace ID.
        Constants.baseWebsocketApiUrl = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference";
        CountDownLatch latch = new CountDownLatch(1);
        SpeechSynthesizer synthesizer = new SpeechSynthesizer();
        SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                // If you do not set the API key as an environment variable, uncomment the following line and replace yourApiKey with your actual API key.
                // .apiKey("yourApiKey")
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .enableWordTimestamp(true)
                .enablePhonemeTimestamp(true)
                .build();

        class ReactCallback extends ResultCallback<SpeechSynthesisResult> {
            @Override
            public void onEvent(SpeechSynthesisResult result) {
                if (result.getAudioFrame() != null) {
                    // do something with the audio frame
                    System.out.println("Audio result length: " + result.getAudioFrame().array().length);
                }
                if (result.getTimestamp() != null) {
                    // do something with the timestamp
                    System.out.println("Timestamp: " + result.getTimestamp());
                }
            }

            @Override
            public void onComplete() {
                // do something when the synthesis is done
                System.out.println("onComplete!");
                latch.countDown();
            }

            @Override
            public void onError(Exception e) {
                // do something when an error occurs
                System.out.println("onError:" + e);
                latch.countDown();
            }
        }

        synthesizer.call(param, new ReactCallback());
        try {
            latch.await();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.exit(0);
    }
}

Call through Flowable

Flowable is an open source framework for workflow and business process management (BPM). It is released under the Apache 2.0 license. For more information, see the Flowable API details.

Click to view the complete example

The following example shows how to use the blockingForEach interface of the Flowable object to obtain the audio data and timestamp information (SpeechSynthesisResult) that is returned in each stream in a blocking manner.

After all streaming data from Flowable is returned, you can also call the getAudioData and getTimestamps methods of the SpeechSynthesizer class to retrieve the complete synthesis result and timestamps.

import com.alibaba.dashscope.audio.tts.SpeechSynthesisParam;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisResult;
import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;
import com.alibaba.dashscope.utils.Constants;
import io.reactivex.Flowable;

public class Main {
    public static void main(String[] args) {
        // Replace "{WorkspaceId}" with your actual Workspace ID.
        Constants.baseWebsocketApiUrl = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference";
        SpeechSynthesizer synthesizer = new SpeechSynthesizer();
        SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                // If you do not set the API key as an environment variable, uncomment the following line and replace yourApiKey with your actual API key.
                // .apiKey("yourApiKey")
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .enableWordTimestamp(true)
                .build();

        Flowable<SpeechSynthesisResult> flowable = synthesizer.streamCall(param);
        flowable.blockingForEach(
                msg -> {
                    if (msg.getAudioFrame() != null) {
                        // do something with the audio frame
                        System.out.println("getAudioFrame");
                    }
                    if (msg.getTimestamp() != null) {
                        // do something with the timestamp
                        System.out.println("getTimestamp");
                    }
                }
        );
        System.exit(0);
    }
}

High-concurrency calls

The DashScope Java SDK uses the connection pool technology of OkHttp3 to reduce the overhead of repeatedly establishing connections. For more information, see High-concurrency scenarios.

Request parameters

You can use the chained method of SpeechSynthesisParam to configure parameters, such as the model and the text to be synthesized. The configured object is passed to the call method of the SpeechSynthesizer class.

Click to view an example

SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .enableWordTimestamp(true)
                .build();

Parameter	Type	Default value	Required	Description
model	String	-	Yes	The name of the timbre model used for speech synthesis. For a complete list, see Model list.
text	String	-	Yes	The text to be synthesized. The text must be UTF-8 encoded and cannot be empty. Maximum number of characters: 10,000. Character calculation rule: One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character. The Speech Synthesis Markup Language (SSML) format is supported. For more information about how to use SSML, see SSML overview.
format	enum	WAV	No	The encoding format of the synthesized audio. The following formats are supported: SpeechSynthesisAudioFormat.PCM SpeechSynthesisAudioFormat.WAV SpeechSynthesisAudioFormat.MP3 Import `SpeechSynthesisAudioFormat` using `import com.alibaba.dashscope.audio.tts.SpeechSynthesisAudioFormat;`.
sampleRate	int	16000	No	The sample rate of the synthesized audio in Hz. We recommend that you use the default sample rate of the model. For more information, see Model list. If the sample rate does not match, the service performs necessary upsampling or downsampling.
volume	int	50	No	The volume of the synthesized audio. Valid values: 0 to 100.
rate	float	1.0	No	The speech rate of the synthesized audio. Valid values: 0.5 to 2. 0.5: indicates 0.5 times the default speech rate. 1: indicates the default speech rate. The default speech rate is the default output speech rate of the model. The speech rate may vary slightly depending on the speaker. The default speech rate is about four characters per second. 2: indicates twice the default speech rate.
pitch	float	1.0	No	The pitch of the synthesized audio. Valid values: 0.5 to 2.
enableWordTimestamp	boolean	false	No	Specifies whether to enable word-level timestamps. The default value is false.
enablePhonemeTimestamp	boolean	false	No	Specifies whether to display phoneme-level timestamps when word-level timestamps are enabled (`enableWordTimestamp` is `true`). The default value is false.
apiKey	String	-	No	The user's API key.

Key interfaces

`SpeechSynthesizer` class

You can import SpeechSynthesizer using import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;. The key interfaces are as follows:

Interface/Method	Parameter	Return value	Description
`public ByteBuffer call(SpeechSynthesisParam param)`	`param`: request parameters	Binary audio	Sends the text to be synthesized and obtains the speech synthesis result. This method blocks the current thread until all results are returned.
`public void call(SpeechSynthesisParam param, ResultCallback<SpeechSynthesisResult> callback)`	`param`: request parameters `callback`: callback interface (ResultCallback)	None	Asynchronously starts a speech synthesis task. After the task is started, the server calls the method of the `ResultCallback` instance through a callback to return key process information and data to the client.
`public ByteBuffer getAudioData()`	None	Binary audio	Obtains the complete binary audio data. When you use a one-way streaming call, call this method to get the complete audio at once after the callback is complete (after the `onComplete` method of `ResultCallback` is called).
`public List<Sentence> getTimestamps()`	None	`List` collection of sentence-level timestamp information (Sentence)	Obtains the complete sentence-level timestamp information (Sentence). When you use a one-way streaming call, call this method to get the complete timestamp at once after the callback is complete (after the `onComplete` method of `ResultCallback` is called).
`public String getLastRequestId()`	None	Request ID of the current task	Obtains the request ID of the current task. You can call this method after you call `call` to start a new task.
`public long getFirstPackageDelay()`	None	First package latency of the current task	Obtains the first package latency of the current task. Use this method after the task is complete.

Callback interface (`ResultCallback`)

When you use a one-way streaming call, you can retrieve the synthesis result through the ResultCallback interface.

Click to view an example

ResultCallback<SpeechSynthesisResult> callback = new ResultCallback<SpeechSynthesisResult>() {
    @Override
    public void onEvent(SpeechSynthesisResult result) {
        System.out.println("Request ID: " + result.getRequestId());
        // Implement the logic to process the speech synthesis result here.
    }

    @Override
    public void onComplete() {
        System.out.println("Task complete");
    }

    @Override
    public void onError(Exception e) {
        System.out.println("Task failed: " + e.getMessage());
    }
};

Interface/Method	Parameter	Return value	Description
`public void onEvent(SpeechSynthesisResult result)`	`result`: audio data and timestamp information (SpeechSynthesisResult)	None	This method is called back when the server returns synthetic data.
`public void onComplete()`	None	None	This method is called back after all synthetic data is returned.
`public void onError(Exception e)`	`e`: exception information	None	This method is called back when an exception occurs during the call or the service returns an error.

Response results

Non-streaming call: The response is binary audio data.

One-way streaming call: The response is audio data and timestamp information (SpeechSynthesisResult).

Audio data and timestamp information (`SpeechSynthesisResult`)

SpeechSynthesisResult encapsulates the speech synthesis result. The commonly used interfaces are getAudioFrame and getTimestamp.

Interface/Method	Parameter	Return value	Description
`public ByteBuffer getAudioFrame()`	None	Binary audio data	Returns the incremental binary audio data of a streaming synthesis segment. The value can be empty.
`public List<Sentence> getTimestamp()`	None	`List` collection of sentence-level timestamp information (Sentence)	Obtains sentence-level timestamp information in batches. The value can be empty.

Sentence-level timestamp information (`Sentence`)

Sentence encapsulates sentence-level timestamp information.

Interface/Method	Parameter	Return value	Description
`public int getBeginTime()`	None	Start time of the sentence in ms	Returns the start time of the sentence.
`public int getEndTime()`	None	End time of the sentence in ms	Returns the end time of the sentence.
`public List<Word> getWords()`	None	`List` collection of word-level timestamp information (Word)	Obtains word-level timestamp information in batches. The value can be empty.

Word-level timestamp information (`Word`)

Word encapsulates word-level timestamp information.

Interface/Method	Parameter	Return value	Description
`public int getBeginTime()`	None	Start time of the word in ms	Returns the start time of the word.
`public int getEndTime()`	None	End time of the word in ms	Returns the end time of the word.
`public String getText()`	None	Text information	Returns the text information.
`public List<Phoneme> getPhonemes()`	None	`List` collection of phoneme-level timestamp information (Phoneme)	Obtains phoneme-level timestamp information in batches. The value can be empty.

Phoneme-level timestamp information (`Phoneme`)

Phoneme encapsulates phoneme-level timestamp information.

Interface/Method	Parameter	Return value	Description
`public int getBeginTime()`	None	Start time of the phoneme in ms	Returns the start time of the phoneme.
`public int getEndTime()`	None	End time of the phoneme in ms	Returns the end time of the phoneme.
`public String getText()`	None	Text information	Returns the text information.
`public String getTone()`	None	Tone	Returns the tone. In English, 0, 1, and 2 represent unstressed, primary stress, and secondary stress, respectively. In Pinyin, 1, 2, 3, 4, and 5 represent the first, second, third, fourth, and neutral tones, respectively.

Error codes

If an API call fails and returns an error message, see Error codes to resolve the issue.

More examples

For more examples, see GitHub.

FAQ

For more information, see the QA on GitHub.

Model list

Note

The default sample rate is the optimal sample rate for the current model. By default, the output is based on this sample rate. Downsampling and upsampling are also supported. For example, for the Zhimiao voice, the default sample rate is 16 kHz. You can downsample it to 8 kHz, but upsampling it to 48 kHz does not provide additional benefits.

Timbre	Audio sample (Right-click to save the audio)	model parameter	Timestamp support	Scenarios	Features	Language	Default sample rate (Hz)
Zhinan		sambert-zhinan-v1	Yes	General scenarios	Advertising male voice	Chinese and English	48 kHz
Zhiqi		sambert-zhiqi-v1	Yes	General scenarios	Gentle female voice	Chinese and English	48 kHz
Zhichu		sambert-zhichu-v1	Yes	News broadcasting	A Bite of China male voice	Chinese and English	48 kHz
Zhide		sambert-zhide-v1	Yes	News broadcasting	News male voice	Chinese and English	48 kHz
Zhijia		sambert-zhijia-v1	Yes	News broadcasting	Standard female voice	Chinese and English	48 kHz
Zhiru		sambert-zhiru-v1	Yes	News broadcasting	News female voice	Chinese and English	48 kHz
Zhiqian		sambert-zhiqian-v1	Yes	Dubbing and news broadcasting	Information female voice	Chinese and English	48 kHz
Zhixiang		sambert-zhixiang-v1	Yes	Dubbing	Magnetic male voice	Chinese and English	48 kHz
Zhiwei		sambert-zhiwei-v1	Yes	Reading product introductions	Lolita female voice	Chinese and English	48 kHz
Zhihao		sambert-zhihao-v1	Yes	General scenarios	Consultation male voice	Chinese and English	16 KB
Zhijing		sambert-zhijing-v1	Yes	General scenarios	Strict female voice	Chinese and English	16 k
Zhiming		sambert-zhiming-v1	Yes	General scenarios	Humorous male voice	Chinese and English	16 KB
Zhimo		sambert-zhimo-v1	Yes	General scenarios	Emotional male voice	Chinese and English	16 kHz
Zhina		sambert-zhina-v1	Yes	General scenarios	Zhejiang Mandarin female voice	Chinese and English	16 kHz
Zhishu		sambert-zhishu-v1	Yes	General scenarios	Information male voice	Chinese and English	16 KB
Zhisha		sambert-zhistella-v1	Yes	General scenarios	Intellectual female voice	Chinese and English	16 kHz
Zhiting		sambert-zhiting-v1	Yes	General scenarios	Radio female voice	Chinese and English	16 kHz
Zhixiao		sambert-zhixiao-v1	Yes	General scenarios	Information female voice	Chinese and English	16 kHz
Zhiya		sambert-zhiya-v1	Yes	General scenarios	Strict female voice	Chinese and English	16 kHz
Zhiye		sambert-zhiye-v1	Yes	General scenarios	Young male voice	Chinese and English	16 kHz
Zhiying		sambert-zhiying-v1	Yes	General scenarios	Cute child voice	Chinese and English	16 kHz
Zhiyuan		sambert-zhiyuan-v1	Yes	General scenarios	Caring sister voice	Chinese and English	16 kHz
Zhiyue		sambert-zhiyue-v1	Yes	Customer service	Gentle female voice	Chinese and English	16 kHz
Zhigui		sambert-zhigui-v1	Yes	Reading product introductions	Livestreaming female voice	Chinese and English	16 kHz
Zhishuo		sambert-zhishuo-v1	Yes	Digital human	Natural male voice	Chinese and English	16 kHz
Zhimiao (multi-emotional)		sambert-zhimiao-emo-v1	Yes	Reading product introductions, digital humans, and livestreaming	Multi-emotional female voice	Chinese and English	16 kHz
Zhimao		sambert-zhimao-v1	Yes	Reading product introductions, dubbing, digital humans, and livestreaming	Livestreaming female voice	Chinese and English	16 kHz
Zhilun		sambert-zhilun-v1	Yes	Dubbing	Suspense narration	Chinese and English	16 kHz
Zhifei		sambert-zhifei-v1	Yes	Dubbing	Passionate narration	Chinese and English	16 KB
Zhida		sambert-zhida-v1	Yes	News broadcasting	Standard male voice	Chinese and English	16 KB
Camila		sambert-camila-v1	No	General scenarios	Spanish female voice	Spanish	16 kHz
Perla		sambert-perla-v1	No	General scenarios	Italian female voice	Italian	16 k
Indah		sambert-indah-v1	No	General scenarios	Indonesian female voice	Indonesian	16 kHz
Clara		sambert-clara-v1	No	General scenarios	French female voice	French	16 kHz
Hanna		sambert-hanna-v1	No	General scenarios	German female voice	German	16 kHz
Beth		sambert-beth-v1	Yes	General scenarios	Consultation female voice	American English	16 kHz
Betty		sambert-betty-v1	Yes	General scenarios	Customer service female voice	American English	16 kHz
Cally		sambert-cally-v1	Yes	General scenarios	Natural female voice	American English	16 kHz
Cindy		sambert-cindy-v1	Yes	General scenarios	Conversational female voice	American English	16 kHz
Eva		sambert-eva-v1	Yes	General scenarios	Companion female voice	American English	16 KB
Donna		sambert-donna-v1	Yes	General scenarios	Education female voice	American English	16 kHz
Brian		sambert-brian-v1	Yes	General scenarios	Customer service male voice	American English	16 kHz
Waan		sambert-waan-v1	No	General scenarios	Thai female voice	Thai	16 kHz