Introduction to the speech synthesis timestamp feature

Updated at:

Speech synthesis timestamps mark the start and end of each Chinese character or English word in synthesized audio. Use them to synchronize video subtitles, text highlighting, and virtual character lip movements. Enable timestamps in the synthesis request to receive timing data alongside the audio stream.

Limitations

  • Word-level timestamps are available only for voices that support word-level phoneme boundaries.

  • The short-text speech synthesis RESTful API does not return timestamps. Use the WebSocket API or a corresponding SDK to receive them.

  • Subtitle text follows pronunciation and may not match the original text character by character. For example, in Chinese speech, 12 can be represented by two subtitle units: 十 and 二. Display the original text and use the returned timing data to determine sentence boundaries or highlighting times. Do not treat subtitle indexes as character offsets into the original string.

Code example

This Java example uses nls-sdk-tts 2.1.6. It reads the project AppKey and a valid NLS token from environment variables, prints subtitle information, and saves the audio to tts_test.wav. An existing file with that name is overwritten.

Add dependencies and configure credentials

Add the following dependency to the dependencies section of the Maven project:

<dependency>
    <groupId>com.alibaba.nls</groupId>
    <artifactId>nls-sdk-tts</artifactId>
    <version>2.1.6</version>
</dependency>

Set NLS_APP_KEY and NLS_TOKEN in the runtime environment to the project AppKey and NLS token. The token must be valid and belong to the same account as the project. Do not substitute an STS token or an API key for another product.

Synthesize speech and receive timestamps

import com.alibaba.fastjson.JSONArray;
import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizer;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerListener;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerResponse;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;

public class SpeechSynthesizerDemo {
    private static String requiredEnv(String name) {
        String value = System.getenv(name);
        if (value == null || value.trim().isEmpty()) {
            throw new IllegalArgumentException("Missing environment variable: " + name);
        }
        return value;
    }

    public static void main(String[] args) throws Exception {
        String appKey = requiredEnv("NLS_APP_KEY");
        String token = requiredEnv("NLS_TOKEN");
        String endpoint = "wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1";
        NlsClient client = new NlsClient(endpoint, token);

        try (OutputStream audio = Files.newOutputStream(Paths.get("tts_test.wav"))) {
            AtomicReference<String> failure = new AtomicReference<>();
            AtomicBoolean completed = new AtomicBoolean(false);
            AtomicBoolean firstAudio = new AtomicBoolean(true);
            final long start = System.nanoTime();
            SpeechSynthesizerListener listener = new SpeechSynthesizerListener() {
                @Override
                public void onMessage(ByteBuffer message) {
                    if (firstAudio.compareAndSet(true, false)) {
                        System.out.println("First audio latency (ms): "
                                + (System.nanoTime() - start) / 1_000_000);
                    }
                    byte[] bytes = new byte[message.remaining()];
                    message.get(bytes);
                    try {
                        audio.write(bytes);
                    } catch (IOException e) {
                        failure.compareAndSet(null, "Failed to write audio: " + e.getMessage());
                    }
                }

                @Override
                public void onMetaInfo(SpeechSynthesizerResponse response) {
                    JSONArray subtitles = (JSONArray) response.getObject("subtitles");
                    if (subtitles != null) {
                        System.out.println("MetaInfo: " + subtitles.toJSONString());
                    }
                }

                @Override
                public void onComplete(SpeechSynthesizerResponse response) {
                    completed.set(true);
                    System.out.println("SynthesisCompleted: " + response.getTaskId());
                }

                @Override
                public void onFail(SpeechSynthesizerResponse response) {
                    failure.set("task_id=" + response.getTaskId()
                            + ", status=" + response.getStatus()
                            + ", status_text=" + response.getStatusText());
                }
            };

            SpeechSynthesizer synthesizer = new SpeechSynthesizer(client, listener);
            try {
                synthesizer.setAppKey(appKey);
                synthesizer.setFormat(OutputFormatEnum.WAV);
                synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_16K);
                synthesizer.setVoice("siyue");
                synthesizer.setPitchRate(100);
                synthesizer.setSpeechRate(100);
                synthesizer.setText("Hello world. I have 12 apples.");
                synthesizer.addCustomedParam("enable_subtitle", true);
                synthesizer.start();
                synthesizer.waitForComplete();
                if (failure.get() != null) {
                    throw new IllegalStateException(failure.get());
                }
                if (!completed.get()) {
                    throw new IllegalStateException("Synthesis did not complete");
                }
                System.out.println("Audio saved to tts_test.wav");
            } finally {
                synthesizer.close();
            }
        } finally {
            client.shutdown();
        }
    }
}

On success, the program prints MetaInfo subtitle arrays, SynthesisCompleted, and an audio-file confirmation. First-audio latency is measured from before synthesis starts until the first audio chunk arrives. It differs from the time needed to receive all audio.

Note

The example saves audio to a file. For latency-sensitive playback, play audio chunks as they arrive in onMessage instead of waiting for synthesis to finish.

Note

For internal network access from an ECS instance in the China (Shanghai) region, use the following endpoint when creating NlsClient:

NlsClient client = new NlsClient(
        "ws://nls-gateway.cn-shanghai-internal.aliyuncs.com/ws/v1", token);

Word-level timestamps

Request parameters

Before starting synthesis, set enable_subtitle to true. Timestamps are disabled by default. In the Java SDK, use:

synthesizer.addCustomedParam("enable_subtitle", true);

Server response

The service returns timestamps in MetaInfo events. Each event contains a subtitle array in payload.subtitles. The Java SDK delivers these events through the onMetaInfo callback.

Fields in each subtitles element

Field

Type

Description

text

String

A pronunciation-based text unit, such as a Chinese character or an English word.

begin_time

Integer

The start time of the text unit in the synthesized audio, in milliseconds.

end_time

Integer

The end time of the text unit in the synthesized audio, in milliseconds.

phoneme

String

When only word-level timestamps are enabled, this field contains the string "null", not JSON null.

begin_index

Integer

The zero-based start index of the text unit. This is not a character offset into the original string.

end_index

Integer

The exclusive end index of the text unit.

For example, the two words in Hello world! occupy index ranges [0, 1) and [1, 2). In the Chinese input 你好,世界。, the character 世 has a start index of 2.

Note

A synthesis request can produce multiple MetaInfo events. Subtitle arrays can repeat or include previously returned entries. Update existing data by index and timestamp instead of appending each entire array.

Sample response

The following sample payload is returned for Hello world!. Timing values vary with the voice, speech rate, and text.

{
  "subtitles": [
    {
      "text": "Hello",
      "phoneme": "null",
      "begin_index": 0,
      "end_index": 1,
      "begin_time": 0,
      "end_time": 328
    },
    {
      "text": "world",
      "phoneme": "null",
      "begin_index": 1,
      "end_index": 2,
      "begin_time": 328,
      "end_time": 775
    }
  ]
}

Phoneme-level timestamps

Phoneme-level timestamps provide the start and end times of individual phonemes within each text unit for more precise lip synchronization. Availability depends on the voice. For example, aida, aiqi, aiyue, ailun, ava, and emily return phoneme_list. With the feature enabled, siyue and aicheng return phoneme strings but not phoneme timestamp arrays.

Note

Phoneme-level timestamps are not currently supported on mobile devices.

Request parameters

Before starting synthesis, set both enable_subtitle and enable_phoneme_timestamp to true. For example, to synthesize Chinese text with aida:

synthesizer.setVoice("aida");
synthesizer.addCustomedParam("enable_subtitle", true);
synthesizer.addCustomedParam("enable_phoneme_timestamp", true);

Server response

Phoneme information is returned in payload.subtitles in MetaInfo events. The text, timing, and index fields have the same meanings as for word-level timestamps. The phoneme fields are:

Field

Type

Description

phoneme

String

The space-separated phoneme sequence for the text unit. The string "null" indicates that no phoneme information is available.

phoneme_list

List

The phoneme timestamp array. Availability depends on the voice. A populated phoneme field alone does not indicate that phoneme timestamps are available.

Fields in each phoneme_list element

Field

Type

Description

index

Integer

The zero-based index of the phoneme within the current text unit.

begin_time

Integer

The start time of the phoneme in the synthesized audio, in milliseconds.

end_time

Integer

The end time of the phoneme in the synthesized audio, in milliseconds.

phoneme

String

The phoneme name. For mappings, see Phoneme mappings to Pinyin and IPA.

tone

String

The tone or stress marker. For English, "0", "1", and "2" indicate no stress, primary stress, and secondary stress, respectively. For Pinyin, "1" through "5" indicate the first, second, third, fourth, and neutral tones.

Sample response

The following excerpt from payload shows the Chinese characters 今 and 天 when aida synthesizes 今天阳光很好。Hello world..

{
  "subtitles": [
    {
      "text": "今",
      "phoneme": "j_c in_c",
      "begin_index": 0,
      "end_index": 1,
      "begin_time": 0,
      "end_time": 200,
      "phoneme_list": [
        {
          "index": 0,
          "begin_time": 0,
          "end_time": 104,
          "phoneme": "j_c",
          "tone": "1"
        },
        {
          "index": 1,
          "begin_time": 104,
          "end_time": 200,
          "phoneme": "in_c",
          "tone": "1"
        }
      ]
    },
    {
      "text": "天",
      "phoneme": "t_c ian_c",
      "begin_index": 1,
      "end_index": 2,
      "begin_time": 200,
      "end_time": 370,
      "phoneme_list": [
        {
          "index": 0,
          "begin_time": 200,
          "end_time": 280,
          "phoneme": "t_c",
          "tone": "1"
        },
        {
          "index": 1,
          "begin_time": 280,
          "end_time": 370,
          "phoneme": "ian_c",
          "tone": "1"
        }
      ]
    }
  ]
}

Phoneme mappings to Pinyin and IPA

Chinese phonemes use the symbols in the mapping table. For example, 天 maps to t_c ian_c. Some Pinyin syllables use special symbols, so appending _c to Pinyin letters does not always produce the correct phonemes. For the complete mapping, see the Pinyin-to-phoneme mapping table.

The following tables map English phonemes to International Phonetic Alphabet (IPA) symbols. Some phoneme names include a numeric suffix. For example, ah1 is the stressed form of ah, and ah0 is the unstressed form. Suffixes 0, 1, and 2 have the same meanings as the tone values.

American English

No.

American English phoneme

IPA symbol

1

aa

ɑ

2

ae

æ

3

ah1

ʌ

4

ah2

ə

5

ah0

ə

6

ao

ɔ

7

aw

aʊ

8

ay

aɪ

9

b

b

10

ch

tʃ

11

d

d

12

dh

ð

13

eh

ɛ

14

er

ɝ

15

ey

eɪ

16

f

f

17

g

g

18

hh

h

19

ih

ɪ

20

iy

i

21

jh

dʒ

22

k

k

23

l

l

24

m

m

25

n

n

26

ng

ŋ

27

ow

oʊ

28

oy

ɔɪ

29

p

p

30

r

ɹ

31

s

s

32

sh

ʃ

33

t

t

34

th

θ

35

uh

ʊ

36

uw

u

37

v

v

38

w

w

39

y

j

40

z

z

41

zh

ʒ

British English

No.

British English phoneme

IPA symbol

1

@

ə

2

@@

ɜː

3

a

æ

4

aa

ɑː

5

e

ɛ

6

i

ɪ

7

ii

iː

8

o

ɒ

9

oo

ɔː

10

u

ʊ

11

uh

ʌ

12

uu

uː

13

ai

aɪ

14

au

aʊ

15

e@

ɛə

16

ei

eɪ

17

i@

ɪə

18

oi

ɔɪ

19

ou

əʊ

20

u@

ʊə

21

p

p

22

b

b

23

t

t

24

d

d

25

k

k

26

g

g

27

ch

tʃ

28

jh

dʒ

29

s

s

30

z

z

31

v

v

32

f

f

33

sh

ʃ

34

th

θ

35

dh

ð

36

zh

ʒ

37

h

h

38

r

ɹ

39

l

l

40

m

m

41

n

n

42

ng

ŋ

43

y

j

44

w

w