Prerequisites
-
Before you use the software development kit (SDK), read the Product introduction.
-
You must obtain an authentication key. To obtain an Appkey, see Manage projects. To obtain a token, see Obtain a token using an SDK.
Java code sample
Download and install
You can download the latest version of the SDK from the Maven server.
<dependency>
<groupId>com.alibaba.nls</groupId>
<artifactId>nls-sdk-tts</artifactId>
<version>2.2.14</version>
</dependency>
<dependency>
<groupId>com.alibaba.nls</groupId>
<artifactId>nls-sdk-common</artifactId>
<version>2.2.14</version>
</dependency>Important
Starting from version 2.1.7 of the Java SDK, the timeout unit for the SpeechSynthesizer.waitForComplete method has changed from seconds to milliseconds. This change applies to the speech synthesis SDK and the real-time long-text speech synthesis SDK.
Sample call
The following Java code sample shows how to stream text input, request speech synthesis, and play the resulting audio. To save the synthesized audio to a local file, append the received binary audio stream to a file within the onAudioData method.
Important
Before you run the code, replace your-appkey and your-token with your actual AppKey and token.
package org.example;
import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.StreamInputTts;
import com.alibaba.nls.client.protocol.tts.StreamInputTtsListener;
import com.alibaba.nls.client.protocol.tts.StreamInputTtsResponse;
import javax.sound.sampled.*;
import java.nio.ByteBuffer;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicBoolean;
class PlaybackRunnable implements Runnable {
// Set the audio format. Configure the format based on your device, synthesis parameters, and platform.
// Here, 24k, 16-bit, and mono-channel are selected. Select other sample rates and formats based on the model's sample rate and your device's compatibility.
private AudioFormat af;
private DataLine.Info info;
private SourceDataLine targetSource;
private AtomicBoolean runFlag;
private ConcurrentLinkedQueue<ByteBuffer> queue;
public PlaybackRunnable(int sample_rate) {
af = new AudioFormat(sample_rate, 16, 1, true, false);
info = new DataLine.Info(SourceDataLine.class, af);
targetSource = null;
runFlag = new AtomicBoolean(true);
queue = new ConcurrentLinkedQueue<>();
}
// Prepare the player.
public void prepare() throws LineUnavailableException {
targetSource = (SourceDataLine) AudioSystem.getLine(info);
targetSource.open(af, 4096);
targetSource.start();
}
public void put(ByteBuffer buffer) {
queue.add(buffer);
}
// Stop playback.
public void stop() {
runFlag.set(false);
}
@Override
public void run() {
if (targetSource == null) {
return;
}
while (runFlag.get()) {
if (queue.isEmpty()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
}
continue;
}
ByteBuffer buffer = queue.poll();
if (buffer == null) {
continue;
}
byte[] data = buffer.array();
targetSource.write(data, 0, data.length);
}
// Play all cached data.
if (!queue.isEmpty()) {
ByteBuffer buffer = null;
while ((buffer = queue.poll()) != null) {
byte[] data = buffer.array();
targetSource.write(data, 0, data.length);
}
}
// Release the player.
targetSource.drain();
targetSource.stop();
targetSource.close();
}
}
public class StreamInputTtsPlayableDemo {
private static long startTime;
NlsClient client;
private String appKey;
public StreamInputTtsPlayableDemo(String appKey, String token, String url) {
this.appKey = appKey;
// Create an NlsClient instance. One instance is sufficient for the entire application. The lifecycle of the instance can be the same as the application's lifecycle. The default endpoint is the Alibaba Cloud online service endpoint.
if (url.isEmpty()) {
client = new NlsClient(token);
} else {
client = new NlsClient(url, token);
}
}
private static StreamInputTtsListener getSynthesizerListener(final PlaybackRunnable audioPlayer) {
StreamInputTtsListener listener = null;
try {
listener = new StreamInputTtsListener() {
private boolean firstRecvBinary = true;
// Stream-based text-to-speech synthesis starts.
@Override
public void onSynthesisStart(StreamInputTtsResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus());
}
// The server detects the beginning of a sentence.
@Override
public void onSentenceBegin(StreamInputTtsResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus());
System.out.println("Sentence Begin");
}
// The server detects the end of a sentence and obtains the start and end positions and all timestamps for the sentence.
@Override
public void onSentenceEnd(StreamInputTtsResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus() + ", subtitles: " + response.getObject("subtitles"));
}
// Stream-based text-to-speech synthesis is complete.
@Override
public void onSynthesisComplete(StreamInputTtsResponse response) {
// When onSynthesisComplete is called, it indicates that all text-to-speech (TTS) data has been received. All text has been synthesized into audio and returned.
System.out.println("name: " + response.getName() + ", status: " + response.getStatus());
audioPlayer.stop();
}
// Received binary audio data from speech synthesis.
@Override
public void onAudioData(ByteBuffer message) {
if (firstRecvBinary) {
// Calculate the latency of the first audio stream packet here. You can start audio playback as soon as you receive the first packet. This improves response speed, especially in real-time interactive scenarios.
firstRecvBinary = false;
long now = System.currentTimeMillis();
System.out.println("tts first latency : " + (now - StreamInputTtsPlayableDemo.startTime) + " ms");
}
byte[] bytesArray = new byte[message.remaining()];
message.get(bytesArray, 0, bytesArray.length);
System.out.println("recv audio bytes:" + bytesArray.length);
audioPlayer.put(ByteBuffer.wrap(bytesArray));
}
// Received incremental audio timestamps from speech synthesis.
@Override
public void onSentenceSynthesis(StreamInputTtsResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus() + ", subtitles: " + response.getObject("subtitles"));
}
@Override
public void onFail(StreamInputTtsResponse response) {
// The task_id is the unique identifier for communication between the caller and the server. Provide this task_id when you encounter issues to facilitate troubleshooting.
System.out.println(
"session_id: " + getStreamInputTts().getCurrentSessionId() +
", task_id: " + response.getTaskId() +
// Status code
", status: " + response.getStatus() +
// Error message
", status_text: " + response.getStatusText());
audioPlayer.stop();
}
};
} catch (Exception e) {
e.printStackTrace();
}
return listener;
}
public static void main(String[] args) throws Exception {
String appKey = "your-appkey";
String token = "your-token";
// Use the default URL.
String url = "wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1";
String[] textArray = {"The stream-based text-to-speech synthesis SDK ", "can convert input text ", "into binary audio data. ",
"Compared with non-streaming speech synthesis, ", "streaming synthesis provides stronger ", "real-time performance. Users can hear ",
"nearly synchronized audio output ", "while entering text, ", "which greatly improves the interactive experience ",
"and reduces user waiting time. ", "This is suitable for scenarios where ", "a large language model (LLM) is called ",
"to perform speech synthesis ", "with streaming text input."};
StreamInputTtsPlayableDemo demo = new StreamInputTtsPlayableDemo(appKey, token, url);
demo.process(textArray);
demo.shutdown();
}
public void process(String[] textArray) throws InterruptedException {
StreamInputTts synthesizer = null;
PlaybackRunnable playbackRunnable = new PlaybackRunnable(24000);
try {
playbackRunnable.prepare();
} catch (LineUnavailableException e) {
throw new RuntimeException(e);
}
Thread playbackThread = new Thread(playbackRunnable);
// Start the playback thread.
playbackThread.start();
try {
// Create an instance and establish a connection.
synthesizer = new StreamInputTts(client, getSynthesizerListener(playbackRunnable));
synthesizer.setAppKey(appKey);
// Set the encoding format of the returned audio.
synthesizer.setFormat(OutputFormatEnum.PCM);
// Set the sample rate of the returned audio.
synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_24K);
synthesizer.setVoice("longxiaochun");
// Volume. The range is 0 to 100. This parameter is optional. The default value is 50.
synthesizer.setVolume(50);
// Pitch. The range is -500 to 500. This parameter is optional. The default value is 0.
synthesizer.setPitchRate(0);
// Speech rate. The range is -500 to 500. The default value is 0.
synthesizer.setSpeechRate(0);
// This method serializes the preceding parameter settings into JSON format, sends them to the server, and waits for confirmation from the server.
long start = System.currentTimeMillis();
synthesizer.startStreamInputTts();
System.out.println("tts start latency " + (System.currentTimeMillis() - start) + " ms");
StreamInputTtsPlayableDemo.startTime = System.currentTimeMillis();
// Set the minimum interval in milliseconds between two consecutive text transmissions. If the time elapsed since the last call is less than this value when you call send, the process is blocked until the condition is met.
synthesizer.setMinSendIntervalMS(100);
for (String text : textArray) {
// Send streaming text data.
synthesizer.sendStreamInputTts(text);
}
// Notify the server that the streaming text data has been sent. The process is blocked until the server finishes processing.
synthesizer.stopStreamInputTts();
} catch (Exception e) {
e.printStackTrace();
} finally {
// Close the connection.
if (null != synthesizer) {
synthesizer.close();
playbackThread.join();
}
}
}
public void shutdown() {
client.shutdown();
}
}
Python code sample
Download and install
-
Download the Python SDK.
You can obtain the Python SDK from GitHub or download streamInputTts-github-python directly.
-
Install SDK dependencies.
Go to the root directory of the SDK and run the following command:
python -m pip install -r requirements.txt-
Install the SDK.
After the dependencies are installed, run the following command:
python -m pip install .-
After the installation is complete, you can import the SDK using the following code.
# -*- coding: utf-8 -*-
import nlsImportant
You must run all the preceding commands from the root directory of the SDK.
Sample call
Important
Before you run the code, replace your-appkey and your-token with your actual AppKey and token.
# coding=utf-8
#
# Installation instructions for pyaudio:
# APPLE Mac OS X
# brew install portaudio
# pip install pyaudio
# Debian/Ubuntu
# sudo apt-get install python-pyaudio python3-pyaudio
# or
# pip install pyaudio
# CentOS
# sudo yum install -y portaudio portaudio-devel && pip install pyaudio
# Microsoft Windows
# python -m pip install pyaudio
import nls
import time
# Enable log output.
nls.enableTrace(False)
# Save the audio to a file.
SAVE_TO_FILE = True
# Play the audio in real time using a player. A sound card is required. If you run the code on a server, disable this switch.
PLAY_REALTIME_RESULT = True
if PLAY_REALTIME_RESULT:
import pyaudio
test_text = [
"The stream-based text-to-speech synthesis SDK ",
"can convert input text ",
"into binary audio data. ",
"Compared with non-streaming speech synthesis, ",
"streaming synthesis provides stronger ",
"real-time performance. Users can hear ",
"nearly synchronized audio output ",
"while entering text, ",
"which greatly improves the interactive experience ",
"and reduces user waiting time. ",
"This is suitable for scenarios where ",
"a large language model (LLM) is called ",
"to perform speech synthesis ",
"with streaming text input.",
]
if __name__ == "__main__":
if SAVE_TO_FILE:
file = open("output.wav", "wb")
if PLAY_REALTIME_RESULT:
player = pyaudio.PyAudio()
stream = player.open(
format=pyaudio.paInt16, channels=1, rate=24000, output=True
)
# Create an SDK instance.
# Configure the callback function.
def test_on_data(data, *args):
if SAVE_TO_FILE:
file.write(data)
if PLAY_REALTIME_RESULT:
stream.write(data)
def test_on_message(message, *args):
print("on message=>{}".format(message))
def test_on_close(*args):
print("on_close: args=>{}".format(args))
def test_on_error(message, *args):
print("on_error message=>{} args=>{}".format(message, args))
sdk = nls.NlsStreamInputTtsSynthesizer(
# Because the large model voice is currently available only in the China (Beijing) region, you must change the URL to the endpoint of the China (Beijing) region.
url="wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1",
token="your-token",
appkey="your-appkey",
on_data=test_on_data,
on_sentence_begin=test_on_message,
on_sentence_synthesis=test_on_message,
on_sentence_end=test_on_message,
on_completed=test_on_message,
on_error=test_on_error,
on_close=test_on_close,
callback_args=[],
)
# Send a text message.
sdk.startStreamInputTts(
voice="longxiaochun", # The speaker for speech synthesis.
aformat="wav", # The format of the synthesized audio.
sample_rate=24000, # The sample rate of the synthesized audio.
volume=50, # The volume of the synthesized audio.
speech_rate=0, # The speech rate of the synthesized audio.
pitch_rate=0, # The pitch of the synthesized audio.
)
for text in test_text:
sdk.sendStreamInputTts(text)
time.sleep(0.05)
sdk.stopStreamInputTts()
if SAVE_TO_FILE:
file.close()
if PLAY_REALTIME_RESULT:
stream.stop_stream()
stream.close()
player.terminate()
Mobile client code sample
Note
For sample calls, see the demo project.
Download and install
-
Download the Android and iOS sample project or the HarmonyNext SDK sample project.
ImportantAfter you download the project, you must replace the placeholder values for your Alibaba Cloud account information, AppKey, and token in the sample initialization code before you run the project.
Category
Android compatibility
iOS compatibility
HarmonyNext compatibility
System
Android 4.0 or later, API level 14
iOS 12 or later.
Support for IDE 5.0.3.600
Architecture
armeabi-v7a, arm64-v8a, x86, x86_64
arm64, x86_64
arm64-v8a
-
Decompress the ZIP package and add the code library.
-
For Android, obtain the SDK package in AAR format from the
app/libsdirectory and add the AAR package to your project as a dependency. -
For iOS, add nuisdk.framework from the ZIP package to your project. Then, add nuisdk.framework to the Link Binary With Libraries section of the Build Phases tab.
-
For HarmonyNext, the neonui.har file in the entry/libs/ directory of the compressed package is the HAR package generated by the SDK. You can import and call this package in your project.
-
-
Open the project file to run the demo project.
-
For Android, you can use Android Studio to open this project and review the reference code. The sample code for stream-based speech synthesis is in the StreamInputTtsBasicActivity.java file. After you replace the Appkey and token, you can run the code.
-
For iOS, you can use Xcode to open this project. The project provides reference code and ready-to-use utility classes, such as classes for audio playback, recording, and file operations. You can copy the source code into your own project. The sample code for stream-based speech synthesis is in the ViewController file. After you replace the Appkey and token, you can run the code.
-
For HarmonyNext, use DevEco Studio to open the project. The cosyvoice sample code is in the StreamTTSPage.ets file. After you replace the Appkey and token in the UserKeyStreamTTS class in UserKey.ets, you can run the code.
-
Sample call
Note
For more information about how to use the mobile SDK, see the official Android and iOS demo projects.
A sample for the HarmonyOS Next system is also provided.
-
Write your own callback function.
@Override public void onStreamInputTtsEventCallback( INativeStreamInputTtsCallback.StreamInputTtsEvent event, String task_id, String session_id, int ret_code, String error_msg, String timestamp, String all_response) { Log.i(TAG, "stream input tts event:" + event + " session id " + session_id + " session id " + task_id + " ret " + ret_code); switch (event) { case STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED: // TODO: Process the SynthesisStarted instruction. break; case STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN: // TODO: Process the SentenceBegin instruction. break; case STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS: // TODO: Process the SentenceSynthesis instruction. break; case STREAM_INPUT_TTS_EVENT_SENTENCE_END: // TODO: Process the SynthesisEnd instruction. break; case STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE: // TODO: Process the SynthesisComplete instruction. break; case STREAM_INPUT_TTS_EVENT_TASK_FAILED: // TODO: Process the TaskFailed instruction. break; default: break; } } @Override public void onStreamInputTtsDataCallback(byte[] data) { if (data.length > 0) { if (mEncodeType.equals("pcm")) { mAudioTrack.setAudioData(data); } } }/** * Event callback. This prints the complete information returned by the service. You can process the corresponding event types as needed. */ - (void)onStreamInputTtsEventCallback:(StreamInputTtsCallbackEvent)event taskId:(char *)taskid sessionId:(char *)sessionId ret_code:(int)ret_code error_msg:(char *)error_msg timestamp:(char *)timestamp all_response:(char *)all_response { // [self logFormattedString:@"\n[Event callback] (event_code : %d), %s", event, // all_response]; NSLog(@"\n[Event callback] (event_code : %d), %s", event, all_response); // You can process various event types. switch (event) { case TTS_EVENT_SYNTHESIS_STARTED: // TODO: Process the SynthesisStarted instruction. break; case TTS_EVENT_SENTENCE_BEGIN: // TODO: Process the SentenceBegin instruction. break; case TTS_EVENT_SENTENCE_SYNTHESIS: // TODO: Process the SentenceSynthesis instruction. break; case TTS_EVENT_SENTENCE_END: // TODO: Process the SynthesisEnd instruction. break; case TTS_EVENT_SYNTHESIS_COMPLETE: // TODO: Process the SynthesisComplete instruction. break; case TTS_EVENT_TASK_FAILED: // TODO: Process the TaskFailed instruction. break; default: break; } } /** * Audio callback function. This writes the received audio frames to the audio player. */ - (void)onStreamInputTtsDataCallback:(char *)buffer len:(int)len { NSLog(@"\n[Audio received] %d bytes", len); if (len > 0) { [_voicePlayer write:(char *)buffer Length:(unsigned int)len]; } }function cb_tts_event_callback(event:StreamInputTtsEvent, task_id:string, session_id:string, ret_code:number, error_msg:string, timestamp:string, all_response:string):void{ console.info( "stream input tts event:" + event + " session id " + session_id + " session id " + task_id + " ret " + ret_code); if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED) { console.info("STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED"); console.info("start play"); } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS) { console.info("STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS:" + timestamp); } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE || event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_TASK_FAILED) { /* * Note: This indicates that TTS has completed synthesis and returned all audio data through the callback. It does not indicate that the player has finished playing all audio data. */ console.info("play end"); // If synthesis is complete or an error occurs, and you have saved the audio data to a local file, you can close the file. // if (filesave){ // fs.closeSync(filesave) // filesave=undefined // } // Notify the player that data synthesis is complete. Wait for the player to finish playing all cached data. // playerVoiceEnd() if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_TASK_FAILED) { console.info("STREAM_INPUT_TTS_EVENT_TASK_FAILED error_code:" + ret_code + " errmsg:" + error_msg); // If an error is generated, stop the player directly. //playerVoiceStop(true) } else { console.info("STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE" ); } } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN) { console.info("STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN" ); } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_END) { console.info("STREAM_INPUT_TTS_EVENT_SENTENCE_END"); } } function cb_tts_user_data_callback(buffer:ArrayBuffer|null):void{ if (buffer){ // Save the generated audio data to a local audio file. // if (filesave){ // fs.writeSync(filesave.fd, buffer) // } // Send the generated audio data to the player module for real-time playback. if (buffer.byteLength > 0) { playerSetVoiceArrayBuffer(buffer as ArrayBuffer) } } else { console.info("womx cb_tts_user_data_callback undefined"); } } const g_ttscallback_instance:INativeStreamInputTtsCallback = { onStreamInputTtsEventCallback: cb_tts_event_callback, onStreamInputTtsDataCallback: cb_tts_user_data_callback }; -
Perform authentication, set parameters, connect to the network, configure the callback function, and start the stream-based speech synthesis task.
String ticket = genTicket(); String parameters = genParameters(); NativeNui stream_input_tts_instance = new NativeNui(Constants.ModeType.MODE_STREAM_INPUT_TTS); // The callback is an INativeTtsCallback object with configured callbacks. int ret = streamInput_tts_instance.startStreamInputTts( callback, ticket, parameters, "", 1, false);// Obtain authentication information and speech synthesis parameters. NSString *ticket = [self genTicket]; NSString *parameters = [self genParameters]; // Obtain the stream-based TTS instance and configure the delegate. _streamInputTtsSdk = [StreamInputTts get_instance]; _streamInputTtsSdk.delegate = self; // Establish a connection and start the speech synthesis task. int ret = [_streamInputTtsSdk startStreamInputTts:[ticket UTF8String] parameters:[parameters UTF8String] sessionId:nil logLevel:0 saveLog:NO];stream_input_tts_instance:NativeNui = new NativeNui(Constants.ModeType.MODE_STREAM_INPUT_TTS, "streamtts") ticket:string = genTicketTTS(); parameters:string = genParameters(this.fontname); startTTS():number{ let ret:number = this.stream_input_tts_instance.startStreamInputTts( g_ttscallback_instance, this.ticket, this.parameters, "", Constants.LogLevel.toInt(Constants.LogLevel.LOG_LEVEL_VERBOSE), false ) if (Constants.NuiResultCode.SUCCESS != ret) { // An error occurred during startup. console.log("start tts failed"); // showToast("start tts failed") } else { // Success. } return ret }The ticket parameter is a JSON string that is used to generate authentication information. It includes the following fields.
{ "appkey": "your-app-key", // Required. The AppKey. For more information about how to obtain an AppKey, see the relevant documentation. "token": "yout-token", // Required. The token. For more information about how to obtain a token, see the relevant documentation. "url": "wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1", "complete_waiting_ms": "10000", }The parameters parameter is a JSON string that is used to set speech synthesis parameters. It includes the following fields.
{ "voice": "longxiaochun", "format": "wav", "sample_rate": "24000", "volume": "50", "speech_rate": "0", "pitch_rate": "0", "enable_subtitle": "0", "session_id": "", } -
Send streaming text. You can send text in multiple batches. The callback function is executed each time a response is returned.
String text = "Hello"; int ret = streamInput_tts_instance.sendStreamInputTts(text);ret = [_streamInputTtsSdk sendStreamInputTts:[oneLine UTF8String]];let ttstext:string = "Hello" this.stream_input_tts_instance.sendStreamInputTts(ttstext) -
Stop sending text. This action blocks the process until all audio data is returned and the current stream-based speech synthesis task is complete.
int ret = streamInput_tts_instance.stopStreamInputTts();ret = [_streamInputTtsSdk stopStreamInputTts];// Note: For the HarmonyNext platform, the parameters in the stop interface can control whether the stop operation is synchronous or asynchronous. Use asynchronous stop. let retcode: number = this.stream_input_tts_instance.stopStreamInputTts(true) if (retcode == 0) { console.info('womx stopStreamInputTts() and return success') } else { // An error occurred. console.info('womx stopStreamInputTts() but return error:%d', retcode) showToast(this.errorinfo) }
该文章对您有帮助吗？