语音合成时间戳功能介绍

添加依赖和配置凭证

在 Maven 项目的 dependencies 中添加以下依赖：

<dependency>
    <groupId>com.alibaba.nls</groupId>
    <artifactId>nls-sdk-tts</artifactId>
    <version>2.1.6</version>
</dependency>

在运行环境中设置 NLS_APP_KEY 和 NLS_TOKEN，分别填写项目 AppKey 和 NLS Token。Token 必须有效且与项目所属账号匹配，不使用 STS Token 或其他产品的 API Key 代替。

合成并接收时间戳

import com.alibaba.fastjson.JSONArray;
import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizer;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerListener;
import com.alibaba.nls.client.protocol.tts.SpeechSynthesizerResponse;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;

public class SpeechSynthesizerDemo {
    private static String requiredEnv(String name) {
        String value = System.getenv(name);
        if (value == null || value.trim().isEmpty()) {
            throw new IllegalArgumentException("Missing environment variable: " + name);
        }
        return value;
    }

    public static void main(String[] args) throws Exception {
        String appKey = requiredEnv("NLS_APP_KEY");
        String token = requiredEnv("NLS_TOKEN");
        String endpoint = "wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1";
        NlsClient client = new NlsClient(endpoint, token);

        try (OutputStream audio = Files.newOutputStream(Paths.get("tts_test.wav"))) {
            AtomicReference<String> failure = new AtomicReference<>();
            AtomicBoolean completed = new AtomicBoolean(false);
            AtomicBoolean firstAudio = new AtomicBoolean(true);
            final long start = System.nanoTime();
            SpeechSynthesizerListener listener = new SpeechSynthesizerListener() {
                @Override
                public void onMessage(ByteBuffer message) {
                    if (firstAudio.compareAndSet(true, false)) {
                        System.out.println("First audio latency (ms): "
                                + (System.nanoTime() - start) / 1_000_000);
                    }
                    byte[] bytes = new byte[message.remaining()];
                    message.get(bytes);
                    try {
                        audio.write(bytes);
                    } catch (IOException e) {
                        failure.compareAndSet(null, "Failed to write audio: " + e.getMessage());
                    }
                }

                @Override
                public void onMetaInfo(SpeechSynthesizerResponse response) {
                    JSONArray subtitles = (JSONArray) response.getObject("subtitles");
                    if (subtitles != null) {
                        System.out.println("MetaInfo: " + subtitles.toJSONString());
                    }
                }

                @Override
                public void onComplete(SpeechSynthesizerResponse response) {
                    completed.set(true);
                    System.out.println("SynthesisCompleted: " + response.getTaskId());
                }

                @Override
                public void onFail(SpeechSynthesizerResponse response) {
                    failure.set("task_id=" + response.getTaskId()
                            + ", status=" + response.getStatus()
                            + ", status_text=" + response.getStatusText());
                }
            };

            SpeechSynthesizer synthesizer = new SpeechSynthesizer(client, listener);
            try {
                synthesizer.setAppKey(appKey);
                synthesizer.setFormat(OutputFormatEnum.WAV);
                synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_16K);
                synthesizer.setVoice("siyue");
                synthesizer.setPitchRate(100);
                synthesizer.setSpeechRate(100);
                synthesizer.setText("Hello world. I have 12 apples.");
                synthesizer.addCustomedParam("enable_subtitle", true);
                synthesizer.start();
                synthesizer.waitForComplete();
                if (failure.get() != null) {
                    throw new IllegalStateException(failure.get());
                }
                if (!completed.get()) {
                    throw new IllegalStateException("Synthesis did not complete");
                }
                System.out.println("Audio saved to tts_test.wav");
            } finally {
                synthesizer.close();
            }
        } finally {
            client.shutdown();
        }
    }
}

合成成功后，程序输出 MetaInfo 字幕数组、SynthesisCompleted 及音频文件保存提示。首包延迟从发起合成前开始计时，到收到第一包音频结束；它不同于接收完整音频所需的时间。

NlsClient client = new NlsClient(
        "ws://nls-gateway.cn-shanghai-internal.aliyuncs.com/ws/v1", token);

参数设置

发起合成请求前，将 enable_subtitle 设置为 true。该功能默认关闭。Java SDK 设置方式如下：

synthesizer.addCustomedParam("enable_subtitle", true);

服务端响应

服务端通过 MetaInfo 事件返回时间戳，payload.subtitles 为字幕数组。Java SDK 通过 onMetaInfo 回调接收该事件。

subtitles 元素字段

例如，Hello world! 的两个单词分别对应索引区间 [0, 1) 和 [1, 2)；你好，世界。 中“世”的起始索引为 2。

返回示例

以下为合成 Hello world! 时返回的 payload 示例，时间值随音色、语速和文本变化。

{
  "subtitles": [
    {
      "text": "Hello",
      "phoneme": "null",
      "begin_index": 0,
      "end_index": 1,
      "begin_time": 0,
      "end_time": 328
    },
    {
      "text": "world",
      "phoneme": "null",
      "begin_index": 1,
      "end_index": 2,
      "begin_time": 328,
      "end_time": 775
    }
  ]
}

参数设置

发起合成请求前，同时将 enable_subtitle 和 enable_phoneme_timestamp 设置为 true。例如，使用 aida 合成中文：

synthesizer.setVoice("aida");
synthesizer.addCustomedParam("enable_subtitle", true);
synthesizer.addCustomedParam("enable_phoneme_timestamp", true);

服务端响应

音素信息仍通过 MetaInfo 事件的 payload.subtitles 返回，text、时间和索引字段与字级时间戳含义相同。音素相关字段如下：

phoneme_list 元素字段

返回示例

以下为 aida 合成“今天阳光很好。Hello world.”时返回的 payload 节选，仅保留“今”和“天”。

{
  "subtitles": [
    {
      "text": "今",
      "phoneme": "j_c in_c",
      "begin_index": 0,
      "end_index": 1,
      "begin_time": 0,
      "end_time": 200,
      "phoneme_list": [
        {
          "index": 0,
          "begin_time": 0,
          "end_time": 104,
          "phoneme": "j_c",
          "tone": "1"
        },
        {
          "index": 1,
          "begin_time": 104,
          "end_time": 200,
          "phoneme": "in_c",
          "tone": "1"
        }
      ]
    },
    {
      "text": "天",
      "phoneme": "t_c ian_c",
      "begin_index": 1,
      "end_index": 2,
      "begin_time": 200,
      "end_time": 370,
      "phoneme_list": [
        {
          "index": 0,
          "begin_time": 200,
          "end_time": 280,
          "phoneme": "t_c",
          "tone": "1"
        },
        {
          "index": 1,
          "begin_time": 280,
          "end_time": 370,
          "phoneme": "ian_c",
          "tone": "1"
        }
      ]
    }
  ]
}

音素与拼音、国际音标的映射

中文音素使用映射表中的标记，例如“天”对应 t_c ian_c。部分拼音使用特殊标记，不能一律将拼音字母直接加上 _c。完整映射请参见拼音与音素映射表。

英文音素与国际音标的映射如下。部分音素名称附带数字，例如 ah1 表示 ah 的重音形式，ah0 表示轻音形式；数字 0、1、2 与 tone 的含义对应。