This topic describes how to use the Java software development kit (SDK) for Intelligent Speech Interaction streaming text-to-speech. It includes installation instructions and code examples.
Prerequisites
Before you use the SDK, see the API reference.
Download and install
Download the latest version of the SDK, nls-sdk-java-demo+flowingtts+3.zip, from the Maven server.
<dependency> <groupId>com.alibaba.nls</groupId> <artifactId>nls-sdk-tts</artifactId> <version>2.2.14</version> </dependency> <dependency> <groupId>com.alibaba.nls</groupId> <artifactId>nls-sdk-common</artifactId> <version>2.2.14</version> </dependency>ImportantStarting from Java SDK version 2.1.7, the timeout unit for the waitForComplete method of SpeechSynthesizer is milliseconds instead of seconds. This applies to all speech synthesis, including real-time long-text synthesis.
Unzip the ZIP file.
In the directory that contains the pom.xml file, run the
mvn packagecommand. This command generates an executable JAR file named nls-example-tts-2.0.0-jar-with-dependencies.jar in the target folder.You can copy the JAR package to your application server to quickly authenticate and stress test the service.
Authenticate the service. Run the following command and provide the parameters as prompted. After the command runs, the logs/nls.log file is created in the directory from which you ran the command. The synthesized audio is saved as flowingTts.wav.
java -cp nls-example-flowing-tts-2.0.0-jar-with-dependencies.jar com.alibaba.nls.client.FlowingSpeechSynthesizerDemo <your-api-key> <your-token>
Key interfaces
NlsClient is the speech processing client. Use this client for speech processing tasks such as short sentence recognition, real-time speech recognition, and speech synthesis. This client is thread-safe. Create only one global instance for your application.
FlowingSpeechSynthesizer is the class for processing streaming text-to-speech. Use this interface to set request parameters and send requests. This class is not thread-safe.
FlowingSpeechSynthesizerListener is the listener class for streaming text-to-speech. It listens for returned results. This class is not thread-safe. Implement the following abstract methods:
/** * The server detects the start of a sentence. * @param response */ abstract public void onSentenceBegin(FlowingSpeechSynthesizerResponse response) ; /** * Receives the synthesized audio data stream. * @param message The binary audio data. */ abstract public void onAudioData(ByteBuffer message); /** * The server detects the end of a sentence and returns the start and end positions and all timestamps for the sentence. * @param response */ abstract public void onSentenceEnd(FlowingSpeechSynthesizerResponse response) ; /** * The synthesis is complete. * @param response */ abstract public void onSynthesisComplete(FlowingSpeechSynthesizerResponse response) ; /** * Handles a failure. * @param response */ abstract public void onFail(FlowingSpeechSynthesizerResponse response) ; /** * Returns incremental timestamps in the response=>payload. * @param response */ abstract public void onSentenceSynthesis(FlowingSpeechSynthesizerResponse response) ;
Note the following when you call the SDK:
NlsClient uses the Netty framework. Creating an NlsClient object is resource-intensive, but the object can be reused. Align the creation and shutdown of the NlsClient object with your program's lifecycle.
SpeechSynthesizer objects cannot be reused. Each speech synthesis task requires a new SpeechSynthesizer object. For example, for N conversations between a large model and a user, you must create N SpeechSynthesizer objects for N synthesis tasks.
SpeechSynthesizerListener and SpeechSynthesizer objects have a one-to-one correspondence. Do not assign a single SpeechSynthesizerListener object to multiple SpeechSynthesizer objects. If you do, you cannot distinguish between the different synthesis tasks.
The Java SDK depends on the Netty network library. If your application also uses Netty, update your Netty version to 4.1.17.Final or later.
Code example
package com.alibaba.nls.client;
import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.FlowingSpeechSynthesizer;
import com.alibaba.nls.client.protocol.tts.FlowingSpeechSynthesizerListener;
import com.alibaba.nls.client.protocol.tts.FlowingSpeechSynthesizerResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
/**
* This demo shows how to call the streaming text-to-speech API.
*/
public class FlowingSpeechSynthesizerDemo {
private static final Logger logger = LoggerFactory.getLogger(FlowingSpeechSynthesizerDemo.class);
private static long startTime;
private String appKey;
NlsClient client;
public FlowingSpeechSynthesizerDemo(String appKey, String token, String url) {
this.appKey = appKey;
// Create an NlsClient instance. We recommend that you create only one global instance. The lifecycle of the instance can be the same as your application's lifecycle. The default endpoint is the public Alibaba Cloud endpoint.
if(url.isEmpty()) {
client = new NlsClient(token);
} else {
client = new NlsClient(url, token);
}
}
private static FlowingSpeechSynthesizerListener getSynthesizerListener() {
FlowingSpeechSynthesizerListener listener = null;
try {
listener = new FlowingSpeechSynthesizerListener() {
File f=new File("flowingTts.wav");
FileOutputStream fout = new FileOutputStream(f);
private boolean firstRecvBinary = true;
// Streaming text-to-speech starts.
public void onSynthesisStart(FlowingSpeechSynthesizerResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus());
}
// The server detects the start of a sentence.
public void onSentenceBegin(FlowingSpeechSynthesizerResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus());
System.out.println("Sentence Begin");
}
// The server detects the end of a sentence and obtains the start and end positions and all timestamps for the sentence.
public void onSentenceEnd(FlowingSpeechSynthesizerResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus() + ", subtitles: " + response.getObject("subtitles"));
}
// Streaming text-to-speech ends.
@Override
public void onSynthesisComplete(FlowingSpeechSynthesizerResponse response) {
// When onSynthesisComplete is called, all TTS data has been received, and all text has been synthesized into audio and returned.
System.out.println("name: " + response.getName() + ", status: " + response.getStatus()+", output file :"+f.getAbsolutePath());
}
// Receives the binary audio data from speech synthesis.
@Override
public void onAudioData(ByteBuffer message) {
try {
if(firstRecvBinary) {
// Calculate the latency of the first audio packet. You can start playback when the first audio stream packet is received. This improves response speed, especially in real-time interaction scenarios.
firstRecvBinary = false;
long now = System.currentTimeMillis();
logger.info("tts first latency : " + (now - FlowingSpeechSynthesizerDemo.startTime) + " ms");
}
byte[] bytesArray = new byte[message.remaining()];
message.get(bytesArray, 0, bytesArray.length);
System.out.println("write array:" + bytesArray.length);
fout.write(bytesArray);
} catch (IOException e) {
e.printStackTrace();
}
}
// Receives the incremental audio timestamps from speech synthesis.
@Override
public void onSentenceSynthesis(FlowingSpeechSynthesizerResponse response) {
System.out.println("name: " + response.getName() +
", status: " + response.getStatus() + ", subtitles: " + response.getObject("subtitles"));
}
@Override
public void onFail(FlowingSpeechSynthesizerResponse response){
// The task_id is the unique identifier for communication between the caller and the server. Provide this task_id when you troubleshoot issues.
System.out.println(
"session_id: " + getFlowingSpeechSynthesizer().getCurrentSessionId() +
", task_id: " + response.getTaskId() +
// Status code
", status: " + response.getStatus() +
// Error message
", status_text: " + response.getStatusText());
}
};
} catch (Exception e) {
e.printStackTrace();
}
return listener;
}
public void process(String[] textArray) {
FlowingSpeechSynthesizer synthesizer = null;
try {
// Create an instance and establish a connection.
synthesizer = new FlowingSpeechSynthesizer(client, getSynthesizerListener());
synthesizer.setAppKey(appKey);
// Set the encoding format of the returned audio.
synthesizer.setFormat(OutputFormatEnum.WAV);
// Set the sample rate of the returned audio.
synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_16K);
// The voice.
synthesizer.setVoice("siyue");
// The volume. Valid values: 0 to 100. Optional. Default value: 50.
synthesizer.setVolume(50);
// The pitch. Valid values: -500 to 500. Optional. Default value: 0.
synthesizer.setPitchRate(0);
// The speech rate. Valid values: -500 to 500. Default value: 0.
synthesizer.setSpeechRate(0);
// This method serializes the preceding parameters into JSON format, sends them to the server, and waits for confirmation.
long start = System.currentTimeMillis();
synthesizer.start();
logger.info("tts start latency " + (System.currentTimeMillis() - start) + " ms");
FlowingSpeechSynthesizerDemo.startTime = System.currentTimeMillis();
// Set the minimum interval (in milliseconds) between two consecutive text sends. If the time since the last call is less than this value when send() is called, the call is blocked until the interval condition is met.
synthesizer.setMinSendIntervalMS(100);
for(String text :textArray) {
// Send streaming text data.
synthesizer.send(text);
}
// Notify the server that the streaming text data has been sent. The call is blocked until the server finishes processing.
synthesizer.stop();
} catch (Exception e) {
e.printStackTrace();
} finally {
// Close the connection.
if (null != synthesizer) {
synthesizer.close();
}
}
}
public void shutdown() {
client.shutdown();
}
public static void main(String[] args) throws Exception {
String appKey = "your-api-key";
String token = "your-token";
// Use the default URL.
String url = "wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1";
if (args.length == 2) {
appKey = args[0];
token = args[1];
} else if (args.length == 3) {
appKey = args[0];
token = args[1];
url = args[2];
} else {
System.err.println("run error, need params(url is optional): " + "<app-key> <token> [url]");
System.exit(-1);
}
String[] textArray = {"From the Hundred-Plant Garden to the Three-", "Flavors Study by Lu Xun \nBehind my house was a large", "garden, known as the Hundred-Plant Garden. It has long since been sold, along with the house, to", "the descendants of Zhu Wengong. Even my last glimpse of it was seven or eight",
"years ago. Now it seems to contain only wild grass, but back then, it was my paradise.\nNeedless to say, there were green vegetable plots, smooth well railings, tall soapberry trees, and purple mulberries. Nor need I mention the cicadas singing in the leaves, the fat wasps clinging to the vegetable flowers,",
"or the nimble skylarks that would suddenly shoot up from the grass into the clouds.\nJust along the foot of the low mud wall, there was endless fun. Tree crickets chirped softly, and other crickets played their tunes. Turning over a broken brick, you might find a centipede. There were also blister",
"beetles; if you pressed on their backs, they would puff out a cloud of smoke from behind with a pop.\nThe fleeceflower and magnolia vines were intertwined. The magnolia had fruit like a lotus pod, and the fleeceflower had a swollen root. Some said that if a fleeceflower root was shaped like a person, eating it would",
"make you an immortal. So I often pulled them up, one after another,\neven damaging the mud wall in the process, but I never found one that looked like a person! If you were not afraid of thorns, you could also pick raspberries, like little balls of tiny coral beads, both sour and sweet,",
"Their color and taste were far better than mulberries..."};
FlowingSpeechSynthesizerDemo demo = new FlowingSpeechSynthesizerDemo(appKey, token, url);
demo.process(textArray);
demo.shutdown();
}
}FAQ
How do I resolve the "idle timeout" error returned by the server?
This error occurs if the server does not receive a message from the client for more than 10 s. As a result, the connection is closed and an idle timeout error is returned.
You can call FlowingSpeechSynthesizer.getConnection().sendPing() to periodically send ping packets to the server and keep the connection alive.