Java SDK

Prerequisites

• Obtain the AppKey and token required for authentication. For more information, see Manage projects and Obtain a token using an SDK.

• If you connect using the SDK, download and install it. For more information, see Getting Started.

Download and install

Download the latest version of the SDK from the Maven repository.

<dependency>
    <groupId>com.alibaba.nls</groupId>
    <artifactId>nls-sdk-tts</artifactId>
    <version>2.2.19</version>
</dependency>
<dependency>
    <groupId>com.alibaba.nls</groupId>
    <artifactId>nls-sdk-common</artifactId>
    <version>2.2.19</version>
</dependency>

SDK usage notes

• The long-text speech synthesis interface is supported in SDK versions 2.2.18 and later.

• The NlsClient uses the Netty framework. Creating an `NlsClient` object is resource-intensive, but the object can be reused. We recommend that you align the creation and shutdown of the NlsClient with the lifecycle of your program.

• StreamInputTts objects cannot be reused. Each speech synthesis task requires a unique StreamInputTts object. For example, to perform N speech synthesis tasks for N texts, create N StreamInputTts objects.

• StreamInputTtsListener objects have a one-to-one correspondence with StreamInputTts objects. Do not assign one StreamInputTtsListener object to multiple StreamInputTts objects. Otherwise, the speech synthesis tasks cannot be distinguished.

• The Java SDK depends on the Netty network library. If your application also depends on Netty, update Netty to version 4.1.17.Final or later.

Key interfaces

StreamInputTts is the main class for the CosyVoice large speech synthesis model. It provides the following key interfaces:

• startTts: Establishes a WebSocket connection with the server and configures callbacks and parameters.

  /**
   * Starts a long-text synthesis task and synchronously receives confirmation from the server.
   * @param text The text to be synthesized. SSML is supported.
   * @param milliSeconds The timeout period for the server response.
   * @throws Exception
   */
  public void startTts(String text, long milliSeconds)

• waitForComplete: Blocks until speech synthesis completes and then disconnects the WebSocket connection from the server.

  /**
   * Waits for the speech synthesis to complete.
   */
  public void waitForComplete()

• addCustomedParam: Sets custom request parameters. Use this interface to configure advanced properties or new features for the Voice Service.

  public void addCustomedParam(String key, Object value)

  The parameters that must be set using this method are:

  • enable_aigc_tag

  • aigc_propagator

  • aigc_propagate_id

StreamInputTtsListener is a callback class that includes the following callback functions:

• Event callback functions: Respond to callback events.

  /**
   * The server detects the beginning of a sentence.
   * @param response
   */
  abstract public void onSentenceBegin(
      StreamInputSpeechSynthesizerResponse response);
  
  /**
   * The server detects the end of a sentence and returns the start and end positions and all timestamps for the sentence.
   * @param response
   */
  abstract public void onSentenceEnd(
      StreamInputSpeechSynthesizerResponse response);
  /**
   * Synthesis is complete.
   * @param response
   */
  abstract public void onSynthesisComplete(
      StreamInputSpeechSynthesizerResponse response);
  /**
   * Handles failures.
   * @param response
   */
  abstract public void onFail(StreamInputSpeechSynthesizerResponse response);
  /**
   * Returns timestamps incrementally in response=>payload.
   * @param response
   */
  abstract public void onSentenceSynthesis(
      StreamInputSpeechSynthesizerResponse response);

• Data callback function: Returns synthesized audio data.

  /**
  * Receives the audio data stream from speech synthesis.
  * @param message Binary audio data.
  */
  abstract public void onAudioData(ByteBuffer message);

Example

The following Java code example requests speech synthesis with SSML text input, plays the audio through a speaker, and saves the audio file.

package org.example;

import com.alibaba.nls.client.protocol.NlsClient;
import com.alibaba.nls.client.protocol.OutputFormatEnum;
import com.alibaba.nls.client.protocol.SampleRateEnum;
import com.alibaba.nls.client.protocol.tts.StreamInputTts;
import com.alibaba.nls.client.protocol.tts.StreamInputTtsListener;
import com.alibaba.nls.client.protocol.tts.StreamInputTtsResponse;

import javax.sound.sampled.*;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicBoolean;


class PlaybackRunnable implements Runnable {
    // Set the audio format. Configure the format based on your device, synthesis parameters, and platform.
    // In this example, 24k, 16-bit, and mono-channel are used. Select other sample rates and formats based on the model's sample rate and your device's compatibility.
    private AudioFormat af;

    private DataLine.Info info;

    private SourceDataLine targetSource;

    private AtomicBoolean runFlag;

    private ConcurrentLinkedQueue<ByteBuffer> queue;

    public PlaybackRunnable(int sample_rate) throws FileNotFoundException {
        af = new AudioFormat(sample_rate, 16, 1, true, false);
        info = new DataLine.Info(SourceDataLine.class, af);
        targetSource = null;
        runFlag = new AtomicBoolean(true);
        queue = new ConcurrentLinkedQueue<>();
    }

    // Prepare the player.
    public void prepare() throws LineUnavailableException {
        targetSource = (SourceDataLine) AudioSystem.getLine(info);
        targetSource.open(af, 4096);
        targetSource.start();
    }

    public void put(ByteBuffer buffer) {
        queue.add(buffer);
    }

    // Stop playback.
    public void stop() {
        runFlag.set(false);
    }

    @Override
    public void run() {
        if (targetSource == null) {
            return;
        }
        while (runFlag.get()) {
            if (queue.isEmpty()) {
                try {
                    Thread.sleep(100);
                } catch (InterruptedException e) {
                }
                continue;
            }
            ByteBuffer buffer = queue.poll();
            if (buffer == null) {
                continue;
            }
            byte[] data = buffer.array();
            targetSource.write(data, 0, data.length);
        }
        // Play all cached data.
        if (!queue.isEmpty()) {
            ByteBuffer buffer = null;
            while ((buffer = queue.poll()) != null) {
                byte[] data = buffer.array();
                targetSource.write(data, 0, data.length);
            }
        }
        // Release the player.
        targetSource.drain();
        targetSource.stop();
        targetSource.close();
    }
}


public class StreamInputTtsPlayableDemo {
    private static long startTime;
    NlsClient client;
    private String appKey;

    public StreamInputTtsPlayableDemo(String appKey, String token, String url) {
        this.appKey = appKey;
        // Create an NlsClient instance. One instance is sufficient for the entire application. The lifecycle of the instance can be the same as that of the application. By default, the endpoint is the public endpoint of Alibaba Cloud.
        if (url.isEmpty()) {
            client = new NlsClient(token);
        } else {
            client = new NlsClient(url, token);
        }
    }

    private static StreamInputTtsListener getSynthesizerListener(final PlaybackRunnable audioPlayer) {
        StreamInputTtsListener listener = null;
        try {
            listener = new StreamInputTtsListener() {
                private boolean firstRecvBinary = true;
                File f=new File("ssml_test.pcm");
                FileOutputStream fout = new FileOutputStream(f);


                // The streaming text-to-speech synthesis starts.
                @Override
                public void onSynthesisStart(StreamInputTtsResponse response) {
                    System.out.println("name: " + response.getName() +
                            ", status: " + response.getStatus());
                }

                // The server detects the beginning of a sentence.
                @Override
                public void onSentenceBegin(StreamInputTtsResponse response) {
                    System.out.println("name: " + response.getName() +
                            ", status: " + response.getStatus());
                    System.out.println("Sentence Begin");
                }

                // The server detects the end of a sentence and obtains the start and end positions and all timestamps for the sentence.
                @Override
                public void onSentenceEnd(StreamInputTtsResponse response) {
                    System.out.println("name: " + response.getName() +
                            ", status: " + response.getStatus() + ", subtitles: " + response.getObject("subtitles"));

                }

                // The streaming text-to-speech synthesis is complete.
                @Override
                public void onSynthesisComplete(StreamInputTtsResponse response) {
                    // When onSynthesisComplete is called, it indicates that all TTS data has been received, and all text has been synthesized into audio and returned.
                    System.out.println("name: " + response.getName() + ", status: " + response.getStatus());
                    audioPlayer.stop();
                }

                // Receives the binary audio data from speech synthesis.
                @Override
                public void onAudioData(ByteBuffer message) {
                    if (firstRecvBinary) {
                        // Calculate the latency of the first audio packet. Start audio playback upon receiving the first audio packet to improve response speed, especially in real-time interaction scenarios.
                        firstRecvBinary = false;
                        long now = System.currentTimeMillis();
                        System.out.println("tts first latency : " + (now - StreamInputTtsPlayableDemo.startTime) + " ms");
                    }
                    byte[] bytesArray = new byte[message.remaining()];
                    message.get(bytesArray, 0, bytesArray.length);
                    System.out.println("recv audio bytes:" + bytesArray.length);
                    audioPlayer.put(ByteBuffer.wrap(bytesArray));
                    try {
                        fout.write(bytesArray);
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }

                // Receives the incremental audio timestamps from speech synthesis.
                @Override
                public void onSentenceSynthesis(StreamInputTtsResponse response) {
                    System.out.println("name: " + response.getName() +
                            ", status: " + response.getStatus() + ", subtitles: " + response.getObject("subtitles"));
                }

                @Override
                public void onFail(StreamInputTtsResponse response) {
                    // The task_id is the unique identifier for communication between the client and the server. Provide this task_id when you report an issue.
                    System.out.println(
                            "session_id: " + getStreamInputTts().getCurrentSessionId() +
                                    ", task_id: " + response.getTaskId() +
                                    // Status code
                                    ", status: " + response.getStatus() +
                                    // Error message
                                    ", status_text: " + response.getStatusText());
                    audioPlayer.stop();
                }
            };
        } catch (Exception e) {
            e.printStackTrace();
        }
        return listener;
    }

    public static void main(String[] args) throws Exception {
        String appKey = "your-appkey";
        String token = "your-token";
        // Use the default URL.
        String url = "wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1";
        String text = "<speak bgm=\"http://nls.alicdn.com/bgm/2.wav\">How is the weather today</speak>";
        StreamInputTtsPlayableDemo demo = new StreamInputTtsPlayableDemo(appKey, token, url);
        demo.process(text);
        demo.shutdown();
    }

    public void process(String text) throws InterruptedException, FileNotFoundException {
        StreamInputTts synthesizer = null;
        PlaybackRunnable playbackRunnable = new PlaybackRunnable(8000);
        try {
            playbackRunnable.prepare();
        } catch (LineUnavailableException e) {
            throw new RuntimeException(e);
        }
        Thread playbackThread = new Thread(playbackRunnable);
        // Start the playback thread.
        playbackThread.start();
        try {
            // Create an instance and establish a connection.
            synthesizer = new StreamInputTts(client, getSynthesizerListener(playbackRunnable));
            synthesizer.setAppKey(appKey);
            // Set the encoding format of the returned audio. Optional values: PCM, WAV, MP3, and OPUS.
            synthesizer.setFormat(OutputFormatEnum.PCM);
            // Set the sample rate of the returned audio.
            synthesizer.setSampleRate(SampleRateEnum.SAMPLE_RATE_8K);
            synthesizer.setVoice("longxiaochun_v2");
            // The volume. Valid values: 0 to 100. This parameter is optional. The default value is 50.
            synthesizer.setVolume(50);
            // The pitch. Valid values: -500 to 500. This parameter is optional. The default value is 0.
            synthesizer.setPitchRate(0);
            // The speech rate. Valid values: -500 to 500. The default value is 0.
            synthesizer.setSpeechRate(0);
            // This method serializes the preceding parameters into a JSON object, sends the object to the server, and waits for confirmation from the server.
            // synthesizer.setBitRate(64); // The bitrate. This parameter is valid only for OPUS encoding.
            long start = System.currentTimeMillis();
            synthesizer.startTts(text, 1000);
            synthesizer.waitForComplete();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            // Close the connection.
            if (null != synthesizer) {
                synthesizer.close();
                playbackThread.join();
            }
        }
    }

    public void shutdown() {
        client.shutdown();
    }
}

Common SDK error codes

For more information about error codes, see Error code reference.