Timestamp feature introduction
The real-time long-text speech synthesis service outputs an audio stream and a timestamp for each Chinese character or English word. The timestamp indicates the position of the word in the audio. This feature is also known as the word-level phoneme boundary interface. You can use this time information to drive virtual human lip movements or create video captions.
Function overview
The real-time long-text speech synthesis service splits long text into sentences. For each sentence, the service streams the audio along with two types of timestamps: one for the entire sentence and one for each word within it. Timestamps are returned in blocks, where each block corresponds to a sentence. Timestamps are synchronized with the synthesized audio and are accurately calculated only after the audio is generated.
For example, the text "Alibaba. DAMO Academy." is a long text that contains two sentences:
The first sentence: "Alibaba."
The second sentence: "DAMO Academy."
The following code shows a sample timestamp output. This example is for demonstration purposes only and does not indicate that each subtitles element synthesizes audio for only one word.
// "sentence":true indicates a sentence timestamp. "sentence":false indicates a word timestamp.
{"subtitles":[{"begin_index":0,"end_index":1,"begin_time":0,"end_time":0,"phoneme":"null","text":"","sentence":true},{"begin_index":0,"end_index":1,"begin_time":0,"end_time":150,"phoneme":"null","text":"A","sentence":false}]}
{"subtitles":[{"begin_index":0,"end_index":1,"begin_time":0,"end_time":0,"phoneme":"null","text":"","sentence":true},{"begin_index":0,"end_index":1,"begin_time":0,"end_time":150,"phoneme":"null","text":"A","sentence":false},{"begin_index":1,"end_index":2,"begin_time":150,"end_time":325,"phoneme":"null","text":"li","sentence":false}]}
{"subtitles":[{"begin_index":0,"end_index":1,"begin_time":0,"end_time":0,"phoneme":"null","text":"","sentence":true},{"begin_index":0,"end_index":1,"begin_time":0,"end_time":150,"phoneme":"null","text":"A","sentence":false},{"begin_index":1,"end_index":2,"begin_time":150,"end_time":325,"phoneme":"null","text":"li","sentence":false},{"begin_index":2,"end_index":3,"begin_time":325,"end_time":525,"phoneme":"null","text":"ba","sentence":false}]}
// After the entire sentence is synthesized, the sentence timestamp is accurately calculated.
{"subtitles":[{"begin_index":0,"end_index":1,"begin_time":0,"end_time":850,"phoneme":"null","text":"","sentence":true},{"begin_index":0,"end_index":1,"begin_time":0,"end_time":150,"phoneme":"null","text":"A","sentence":false},{"begin_index":1,"end_index":2,"begin_time":150,"end_time":325,"phoneme":"null","text":"li","sentence":false},{"begin_index":2,"end_index":3,"begin_time":325,"end_time":525,"phoneme":"null","text":"ba","sentence":false},{"begin_index":3,"end_index":4,"begin_time":525,"end_time":788,"phoneme":"null","text":"ba","sentence":false}]}
// The begin_index and end_index of a sentence timestamp are the same as those of the first word in that sentence.
{"subtitles":[{"begin_index":4,"end_index":5,"begin_time":850,"end_time":850,"phoneme":"null","text":"","sentence":true},{"begin_index":4,"end_index":5,"begin_time":850,"end_time":1025,"phoneme":"null","text":"Da","sentence":false}]}
{"subtitles":[{"begin_index":4,"end_index":5,"begin_time":850,"end_time":850,"phoneme":"null","text":"","sentence":true},{"begin_index":4,"end_index":5,"begin_time":850,"end_time":1025,"phoneme":"null","text":"Da","sentence":false},{"begin_index":5,"end_index":6,"begin_time":1025,"end_time":1200,"phoneme":"null","text":"mo","sentence":false}]}
{"subtitles":[{"begin_index":4,"end_index":5,"begin_time":850,"end_time":1512,"phoneme":"null","text":"","sentence":true},{"begin_index":4,"end_index":5,"begin_time":850,"end_time":1025,"phoneme":"null","text":"Da","sentence":false},{"begin_index":5,"end_index":6,"begin_time":1025,"end_time":1200,"phoneme":"null","text":"mo","sentence":false},{"begin_index":6,"end_index":7,"begin_time":1200,"end_time":1450,"phoneme":"null","text":"yuan","sentence":false}]}This feature is available only for voices that support the word-level phoneme boundary interface.
The captions returned by the Text-to-Speech (TTS) service are based on pronunciation. Do not use them directly for on-screen display. Use your original text instead.
To display captions, you can use the returned results to find the start and end timestamps for each sentence.
Parameter settings
To enable the timestamp feature, set the enable_subtitle request parameter to true on the client.
The following example shows how to set this parameter in the Java software development kit (SDK).
// Specifies whether to enable the caption feature, which returns timestamps for the text. This feature is disabled by default.
synthesizer.addCustomedParam("enable_subtitle", true);Server-side response
The server returns a `MetaInfo` event response that contains caption information.
Parameter | Type | Description |
subtitles | List | Timestamp information. |
The `SubtitleItem` object has the following format.
Parameter | Type | Description |
text | String | Text information. |
begin_time | Integer | The start timestamp of the speech that corresponds to the text. Unit: ms. |
end_time | Integer | The end timestamp of the speech that corresponds to the text. Unit: ms. |
phoneme | String | The phoneme series for the character is not supported. The default output is |
begin_index | Integer | The start index of the character in the sentence. The index starts from 0. |
end_index | Integer | The end index of the character in the sentence. The index starts from 0. |
sentence | Boolean | Indicates whether the timestamp is for a sentence. A value of `true` indicates a sentence timestamp. |
Response example
{
"header":{
"namespace":"SpeechLongSynthesizer",
"name":"MetaInfo",
"status":20000000,
"message_id":"49818960d4ca40d88ebxxxxxxxxxxx",
"task_id":"326f3b9d9cfa47f3a692xxxxxxxxxx",
"status_text":"Gateway:SUCCESS:Success."
},
"payload":{
"subtitles":[
{
"text":"",
"phoneme":"null",
"sentence":true,
"begin_index":0,
"end_index":1,
"begin_time":0,
"end_time":498
},
{
"text":"ni",
"phoneme":"null",
"sentence":false,
"begin_index":0,
"end_index":1,
"begin_time":0,
"end_time":118
},
{
"text":"hao",
"phoneme":"null",
"sentence":false,
"begin_index":1,
"end_index":2,
"begin_time":118,
"end_time":439
}
]
}
}
{
"header":{
"namespace":"SpeechLongSynthesizer",
"name":"MetaInfo",
"status":20000000,
"message_id":"bb4e791f1dff464e9997xxxxxxxxxxxxxx",
"task_id":"326f3b9d9cfa47f3a6921xxxxxxxxxxxx",
"status_text":"Gateway:SUCCESS:Success."
},
"payload":{
"subtitles":[
{
"text":"",
"phoneme":"null",
"sentence":true,
"begin_index":2,
"end_index":3,
"begin_time":498,
"end_time":1067
},
{
"text":"ming",
"phoneme":"null",
"sentence":false,
"begin_index":2,
"end_index":3,
"begin_time":498,
"end_time":687
},
{
"text":"tian",
"phoneme":"null",
"sentence":false,
"begin_index":3,
"end_index":4,
"begin_time":687,
"end_time":1008
}
]
}
}Word-level timestamp code example
The example uses the default public endpoint for the speech synthesis service that is built into the SDK. If you use an Elastic Compute Service (ECS) instance in the China (Shanghai) region and need to access the service over the internal network, set the internal endpoint when you create the `NlsClient` object:
client = new NlsClient("ws://nls-gateway.cn-shanghai-internal.aliyuncs.com/ws/v1", accessToken);The example saves the synthesized audio to a file. To play the audio with low latency, you can use streaming playback. Streaming playback plays the audio as data is received, which reduces the delay.
package com.alibaba.nls.client;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizer;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerListener;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* This example demonstrates the following:
* Calling the long-text speech synthesis API (setLongText).
* Streaming TTS synthesis.
* First-packet latency calculation.
*
* Note: This example differs from SpeechSynthesizerLongTextDemo in the nls-example-tts package. Long-text speech synthesis is a separate feature that sends a long string of text directly to the server for synthesis.
* In contrast, SpeechSynthesizerLongTextDemo shows how to split a long string of text on the client side and then call the speech synthesis API in segments.
*/
public class SpeechLongSynthesizerDemo {
private static final Logger logger = LoggerFactory.getLogger(SpeechLongSynthesizerDemo.class);
private static long startTime;
private String appKey;
NlsClient client;
public SpeechLongSynthesizerDemo(String appKey, String token, String url) {
this.appKey = appKey;
// Create an NlsClient instance. A single global instance is sufficient for an application. The instance lifecycle can be the same as the application lifecycle. The default endpoint is the Alibaba Cloud public endpoint.
if(url.isEmpty()) {
client = new NlsClient(token);
} else {
client = new NlsClient(url, token);
}
}
private static SpeechSynthesizerListener getSynthesizerListener() {
SpeechSynthesizerListener listener = null;
try {
listener = new SpeechSynthesizerListener() {
File f=new File("ttsForLongText.wav");
FileOutputStream fout = new FileOutputStream(f);
private boolean firstRecvBinary = true;
// Speech synthesis is complete.
@Override
public void onComplete(SpeechSynthesizerResponse response) {
// When onComplete is called, all TTS data has been received. This represents the latency for the entire synthesis process. This latency can be long and may not be suitable for real-time scenarios.
System.out.println("name: " + response.getName() + ", status: " + response.getStatus()+", output file :"+f.getAbsolutePath());
}
// Binary audio data from speech synthesis.
@Override
public void onMessage(ByteBuffer message) {
try {
if(firstRecvBinary) {
// Calculate the latency of the first audio stream packet here. When the first audio stream packet is received, you can start audio playback to improve response speed, especially in real-time interactive scenarios.
firstRecvBinary = false;
long now = System.currentTimeMillis();
logger.info("tts first latency : " + (now - SpeechLongSynthesizerDemo.startTime) + " ms");
}
byte[] bytesArray = new byte[message.remaining()];
message.get(bytesArray, 0, bytesArray.length);
//System.out.println("write array:" + bytesArray.length);
fout.write(bytesArray);
} catch (IOException e) {
e.printStackTrace();
}
}
@Override
public void onMetaInfo(SpeechSynthesizerResponse response) {
System.out.println("name: " + response.getName() + ", taskId: " + response.getTaskId());
JSONArray subtitles = (JSONArray)response.getObject("subtitles");
List<Map> subtitleList = subtitles.toJavaList(Map.class);
for (Map word : subtitleList) {
System.out.println("current subtitle: " + word);
}
}
@Override
public void onFail(SpeechSynthesizerResponse response){
// The task_id is the unique identifier for communication between the client and the server. Provide this task_id when you report an issue.
System.out.println(
"task_id: " + response.getTaskId() +
// Status code
", status: " + response.getStatus() +
// Error message
", status_text: " + response.getStatusText());
}
};
} catch (Exception e) {
e.printStackTrace();
}
return listener;
}
public void process(String text) {
SpeechSynthesizer synthesizer = null;
try {
// Create an instance and establish a connection.
synthesizer = new SpeechSynthesizer(client, getSynthesizerListener());
synthesizer.setAppKey(appKey);
// Set the encoding format for the returned audio.
synthesizer.setFormat(OutputFormatEnum.WAV);
// Set the sample rate for the returned audio.
synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_16K);
// The voice. Note: The Java SDK does not support voices for ultra-high definition (UHD) scenarios, such as "zhiqi". To use these voices, call the RESTful API.
synthesizer.setVoice("siyue");
// The pitch. The range is -500 to 500. This parameter is optional. The default value is 0.
synthesizer.setPitchRate(0);
// The speech rate. The range is -500 to 500. The default value is 0.
synthesizer.setSpeechRate(0);
// Set the text for speech synthesis.
// The setLongText method is called here. The original speech synthesis method is setText.
synthesizer.setLongText(text);
// This method serializes the preceding parameters into a JSON object, sends the object to the server, and waits for confirmation.
long start = System.currentTimeMillis();
synthesizer.start();
logger.info("tts start latency " + (System.currentTimeMillis() - start) + " ms");
SpeechLongSynthesizerDemo.startTime = System.currentTimeMillis();
// Wait for speech synthesis to complete.
synthesizer.waitForComplete();
logger.info("tts stop latency " + (System.currentTimeMillis() - start) + " ms");
} catch (Exception e) {
e.printStackTrace();
} finally {
// Close the connection.
if (null != synthesizer) {
synthesizer.close();
}
}
}
public void shutdown() {
client.shutdown();
}
public static void main(String[] args) throws Exception {
String appKey = "";
String token = "your_token";
// Use the default URL.
String url = "wss://nls-gateway.cn-shanghai.aliyuncs.com/ws/v1";
if (args.length == 2) {
appKey= args[0];
token = args[1];
} else if (args.length == 3) {
appKey = args[0];
token = args[1];
url = args[2];
} else {
System.err.println("run error, need params(url is optional): " + "<app-key> <token> [url]");
System.exit(-1);
}
String ttsTextLong = "Alibaba Cloud provides a comprehensive suite of global cloud computing services to power both our international customers' online businesses and Alibaba Group's own e-commerce ecosystem. " +
"Alibaba Cloud's services are available on a pay-as-you-go basis and include data storage, relational databases, big-data processing, Anti-DDoS protection and content delivery networks (CDN). " +
"This long text is used as an example to demonstrate the long-text speech synthesis feature. The system will split this text into multiple sentences and synthesize them into speech, providing timestamps for each word and sentence. " +
"This allows for applications such as synchronized captions and virtual human lip-syncing. The service is designed to be highly responsive and scalable, catering to a wide range of use cases from simple notifications to complex interactive voice response systems.";
SpeechLongSynthesizerDemo demo = new SpeechLongSynthesizerDemo(appKey, token, url);
demo.process(ttsTextLong);
demo.shutdown();
}
}