Introduction to the speech synthesis timestamp feature
Speech synthesis timestamps mark the start and end of each Chinese character or English word in synthesized audio. Use them to synchronize video subtitles, text highlighting, and virtual character lip movements. Enable timestamps in the synthesis request to receive timing data alongside the audio stream.
Limitations
-
Word-level timestamps are available only for voices that support word-level phoneme boundaries.
-
The short-text speech synthesis RESTful API does not return timestamps. Use the WebSocket API or a corresponding SDK to receive them.
-
Subtitle text follows pronunciation and may not match the original text character by character. For example, in Chinese speech,
12can be represented by two subtitle units:十and二. Display the original text and use the returned timing data to determine sentence boundaries or highlighting times. Do not treat subtitle indexes as character offsets into the original string.
Code example
This Java example uses nls-sdk-tts 2.1.6. It reads the project AppKey and a valid NLS token from environment variables, prints subtitle information, and saves the audio to tts_test.wav. An existing file with that name is overwritten.
Add dependencies and configure credentials
Add the following dependency to the dependencies section of the Maven project:
<dependency>
<groupId>com.alibaba.nls</groupId>
<artifactId>nls-sdk-tts</artifactId>
<version>2.1.6</version>
</dependency>
Set NLS_APP_KEY and NLS_TOKEN in the runtime environment to the project AppKey and NLS token. The token must be valid and belong to the same account as the project. Do not substitute an STS token or an API key for another product.
Synthesize speech and receive timestamps
import com.alibaba.fastjson.JSONArray;
import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizer;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerListener;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerResponse;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
public class SpeechSynthesizerDemo {
private static String requiredEnv(String name) {
String value = System.getenv(name);
if (value == null || value.trim().isEmpty()) {
throw new IllegalArgumentException("Missing environment variable: " + name);
}
return value;
}
public static void main(String[] args) throws Exception {
String appKey = requiredEnv("NLS_APP_KEY");
String token = requiredEnv("NLS_TOKEN");
String endpoint = "wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1";
NlsClient client = new NlsClient(endpoint, token);
try (OutputStream audio = Files.newOutputStream(Paths.get("tts_test.wav"))) {
AtomicReference<String> failure = new AtomicReference<>();
AtomicBoolean completed = new AtomicBoolean(false);
AtomicBoolean firstAudio = new AtomicBoolean(true);
final long start = System.nanoTime();
SpeechSynthesizerListener listener = new SpeechSynthesizerListener() {
@Override
public void onMessage(ByteBuffer message) {
if (firstAudio.compareAndSet(true, false)) {
System.out.println("First audio latency (ms): "
+ (System.nanoTime() - start) / 1_000_000);
}
byte[] bytes = new byte[message.remaining()];
message.get(bytes);
try {
audio.write(bytes);
} catch (IOException e) {
failure.compareAndSet(null, "Failed to write audio: " + e.getMessage());
}
}
@Override
public void onMetaInfo(SpeechSynthesizerResponse response) {
JSONArray subtitles = (JSONArray) response.getObject("subtitles");
if (subtitles != null) {
System.out.println("MetaInfo: " + subtitles.toJSONString());
}
}
@Override
public void onComplete(SpeechSynthesizerResponse response) {
completed.set(true);
System.out.println("SynthesisCompleted: " + response.getTaskId());
}
@Override
public void onFail(SpeechSynthesizerResponse response) {
failure.set("task_id=" + response.getTaskId()
+ ", status=" + response.getStatus()
+ ", status_text=" + response.getStatusText());
}
};
SpeechSynthesizer synthesizer = new SpeechSynthesizer(client, listener);
try {
synthesizer.setAppKey(appKey);
synthesizer.setFormat(OutputFormatEnum.WAV);
synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_16K);
synthesizer.setVoice("siyue");
synthesizer.setPitchRate(100);
synthesizer.setSpeechRate(100);
synthesizer.setText("Hello world. I have 12 apples.");
synthesizer.addCustomedParam("enable_subtitle", true);
synthesizer.start();
synthesizer.waitForComplete();
if (failure.get() != null) {
throw new IllegalStateException(failure.get());
}
if (!completed.get()) {
throw new IllegalStateException("Synthesis did not complete");
}
System.out.println("Audio saved to tts_test.wav");
} finally {
synthesizer.close();
}
} finally {
client.shutdown();
}
}
}
On success, the program prints MetaInfo subtitle arrays, SynthesisCompleted, and an audio-file confirmation. First-audio latency is measured from before synthesis starts until the first audio chunk arrives. It differs from the time needed to receive all audio.
The example saves audio to a file. For latency-sensitive playback, play audio chunks as they arrive in onMessage instead of waiting for synthesis to finish.
For internal network access from an ECS instance in the China (Shanghai) region, use the following endpoint when creating NlsClient:
NlsClient client = new NlsClient(
"ws://nls-gateway.cn-shanghai-internal.aliyuncs.com/ws/v1", token);
Word-level timestamps
Request parameters
Before starting synthesis, set enable_subtitle to true. Timestamps are disabled by default. In the Java SDK, use:
synthesizer.addCustomedParam("enable_subtitle", true);
Server response
The service returns timestamps in MetaInfo events. Each event contains a subtitle array in payload.subtitles. The Java SDK delivers these events through the onMetaInfo callback.
Fields in each subtitles element
|
Field |
Type |
Description |
|
text |
String |
A pronunciation-based text unit, such as a Chinese character or an English word. |
|
begin_time |
Integer |
The start time of the text unit in the synthesized audio, in milliseconds. |
|
end_time |
Integer |
The end time of the text unit in the synthesized audio, in milliseconds. |
|
phoneme |
String |
When only word-level timestamps are enabled, this field contains the string |
|
begin_index |
Integer |
The zero-based start index of the text unit. This is not a character offset into the original string. |
|
end_index |
Integer |
The exclusive end index of the text unit. |
For example, the two words in Hello world! occupy index ranges [0, 1) and [1, 2). In the Chinese input 你好,世界。, the character 世 has a start index of 2.
A synthesis request can produce multiple MetaInfo events. Subtitle arrays can repeat or include previously returned entries. Update existing data by index and timestamp instead of appending each entire array.
Sample response
The following sample payload is returned for Hello world!. Timing values vary with the voice, speech rate, and text.
{
"subtitles": [
{
"text": "Hello",
"phoneme": "null",
"begin_index": 0,
"end_index": 1,
"begin_time": 0,
"end_time": 328
},
{
"text": "world",
"phoneme": "null",
"begin_index": 1,
"end_index": 2,
"begin_time": 328,
"end_time": 775
}
]
}
Phoneme-level timestamps
Phoneme-level timestamps provide the start and end times of individual phonemes within each text unit for more precise lip synchronization. Availability depends on the voice. For example, aida, aiqi, aiyue, ailun, ava, and emily return phoneme_list. With the feature enabled, siyue and aicheng return phoneme strings but not phoneme timestamp arrays.
Phoneme-level timestamps are not currently supported on mobile devices.
Request parameters
Before starting synthesis, set both enable_subtitle and enable_phoneme_timestamp to true. For example, to synthesize Chinese text with aida:
synthesizer.setVoice("aida");
synthesizer.addCustomedParam("enable_subtitle", true);
synthesizer.addCustomedParam("enable_phoneme_timestamp", true);
Server response
Phoneme information is returned in payload.subtitles in MetaInfo events. The text, timing, and index fields have the same meanings as for word-level timestamps. The phoneme fields are:
|
Field |
Type |
Description |
|
phoneme |
String |
The space-separated phoneme sequence for the text unit. The string |
|
phoneme_list |
List |
The phoneme timestamp array. Availability depends on the voice. A populated |
Fields in each phoneme_list element
|
Field |
Type |
Description |
|
index |
Integer |
The zero-based index of the phoneme within the current text unit. |
|
begin_time |
Integer |
The start time of the phoneme in the synthesized audio, in milliseconds. |
|
end_time |
Integer |
The end time of the phoneme in the synthesized audio, in milliseconds. |
|
phoneme |
String |
The phoneme name. For mappings, see Phoneme mappings to Pinyin and IPA. |
|
tone |
String |
The tone or stress marker. For English, |
Sample response
The following excerpt from payload shows the Chinese characters 今 and 天 when aida synthesizes 今天阳光很好。Hello world..
{
"subtitles": [
{
"text": "今",
"phoneme": "j_c in_c",
"begin_index": 0,
"end_index": 1,
"begin_time": 0,
"end_time": 200,
"phoneme_list": [
{
"index": 0,
"begin_time": 0,
"end_time": 104,
"phoneme": "j_c",
"tone": "1"
},
{
"index": 1,
"begin_time": 104,
"end_time": 200,
"phoneme": "in_c",
"tone": "1"
}
]
},
{
"text": "天",
"phoneme": "t_c ian_c",
"begin_index": 1,
"end_index": 2,
"begin_time": 200,
"end_time": 370,
"phoneme_list": [
{
"index": 0,
"begin_time": 200,
"end_time": 280,
"phoneme": "t_c",
"tone": "1"
},
{
"index": 1,
"begin_time": 280,
"end_time": 370,
"phoneme": "ian_c",
"tone": "1"
}
]
}
]
}
Phoneme mappings to Pinyin and IPA
Chinese phonemes use the symbols in the mapping table. For example, 天 maps to t_c ian_c. Some Pinyin syllables use special symbols, so appending _c to Pinyin letters does not always produce the correct phonemes. For the complete mapping, see the Pinyin-to-phoneme mapping table.
The following tables map English phonemes to International Phonetic Alphabet (IPA) symbols. Some phoneme names include a numeric suffix. For example, ah1 is the stressed form of ah, and ah0 is the unstressed form. Suffixes 0, 1, and 2 have the same meanings as the tone values.
American English
|
No. |
American English phoneme |
IPA symbol |
|
1 |
aa |
ɑ |
|
2 |
ae |
æ |
|
3 |
ah1 |
ʌ |
|
4 |
ah2 |
ə |
|
5 |
ah0 |
ə |
|
6 |
ao |
ɔ |
|
7 |
aw |
aʊ |
|
8 |
ay |
aɪ |
|
9 |
b |
b |
|
10 |
ch |
tʃ |
|
11 |
d |
d |
|
12 |
dh |
ð |
|
13 |
eh |
ɛ |
|
14 |
er |
ɝ |
|
15 |
ey |
eɪ |
|
16 |
f |
f |
|
17 |
g |
g |
|
18 |
hh |
h |
|
19 |
ih |
ɪ |
|
20 |
iy |
i |
|
21 |
jh |
dʒ |
|
22 |
k |
k |
|
23 |
l |
l |
|
24 |
m |
m |
|
25 |
n |
n |
|
26 |
ng |
ŋ |
|
27 |
ow |
oʊ |
|
28 |
oy |
ɔɪ |
|
29 |
p |
p |
|
30 |
r |
ɹ |
|
31 |
s |
s |
|
32 |
sh |
ʃ |
|
33 |
t |
t |
|
34 |
th |
θ |
|
35 |
uh |
ʊ |
|
36 |
uw |
u |
|
37 |
v |
v |
|
38 |
w |
w |
|
39 |
y |
j |
|
40 |
z |
z |
|
41 |
zh |
ʒ |
British English
|
No. |
British English phoneme |
IPA symbol |
|
1 |
@ |
ə |
|
2 |
@@ |
ɜː |
|
3 |
a |
æ |
|
4 |
aa |
ɑː |
|
5 |
e |
ɛ |
|
6 |
i |
ɪ |
|
7 |
ii |
iː |
|
8 |
o |
ɒ |
|
9 |
oo |
ɔː |
|
10 |
u |
ʊ |
|
11 |
uh |
ʌ |
|
12 |
uu |
uː |
|
13 |
ai |
aɪ |
|
14 |
au |
aʊ |
|
15 |
e@ |
ɛə |
|
16 |
ei |
eɪ |
|
17 |
i@ |
ɪə |
|
18 |
oi |
ɔɪ |
|
19 |
ou |
əʊ |
|
20 |
u@ |
ʊə |
|
21 |
p |
p |
|
22 |
b |
b |
|
23 |
t |
t |
|
24 |
d |
d |
|
25 |
k |
k |
|
26 |
g |
g |
|
27 |
ch |
tʃ |
|
28 |
jh |
dʒ |
|
29 |
s |
s |
|
30 |
z |
z |
|
31 |
v |
v |
|
32 |
f |
f |
|
33 |
sh |
ʃ |
|
34 |
th |
θ |
|
35 |
dh |
ð |
|
36 |
zh |
ʒ |
|
37 |
h |
h |
|
38 |
r |
ɹ |
|
39 |
l |
l |
|
40 |
m |
m |
|
41 |
n |
n |
|
42 |
ng |
ŋ |
|
43 |
y |
j |
|
44 |
w |
w |