Streaming text-to-speech Android SDK-Intelligent Speech Interaction(ISI)-阿里云帮助中心

This topic describes how to use the Android software development kit (SDK) for Alibaba Cloud Intelligent Speech Interaction. It covers downloading and installing the SDK, key interfaces, and code examples.

Prerequisites

Read the API reference. For more information, see API reference.
Obtain a project AppKey. For more information, see Create a project.
Obtain an access token. For more information, see Overview of obtaining a token.

Download and installation

Select and download the mobile SDK.
Important
After you download the SDK, replace the placeholders in the sample initialization code with your Alibaba Cloud account information, Appkey, and Token to run the sample.
Unzip the package. The SDK package is an AAR file in the app/libs folder. Add the AAR package as a dependency to your project. For C++ development on Android, you can find the dynamic libraries and header files in the `android_libs` and `android_include` folders within the unzipped package.
Open the project in Android Studio to view the sample code. The sample code for speech synthesis is in the StreamInputTtsBasicActivity.java file. Replace the placeholder values for appkey and token to run the sample.

Key interfaces

startStreamInputTts: Starts streaming text-to-speech (TTS).

/**
 * Starts streaming TTS. Do not call this method on the UI thread, as it may cause blocking.
 * @param callback The event listener callback. See the specific callbacks below.
 * @param ticket The initialization parameters in a JSON string. See the description below.
 * @param parameters The initialization parameters in a JSON string. See the description below.
 * @param session_id The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a unique 32-character ID.
 * @param level The log printing level. A smaller value prints more logs.
 * @param save_log Specifies whether to save logs to a file. The storage path is the value of the debug_path field in the ticket. Note: Log files have no size limit. Continuous storage may fill up the disk.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int startStreamInputTts(INativeStreamInputTtsCallback callback,
                                            String ticket, String parameters,
                                            String session_id, int log_level,
                                            boolean save_log);

The INativeStreamInputTtsCallback type includes the following callbacks.

onStreamInputTtsEventCallback: The callback for SDK events.

/**
 * Event callback
 * @param event The callback event. See the event list below.
 * @param task_id The task ID for the entire real-time speech synthesis session. This must be a unique 32-character ID and remain consistent throughout the request.
 * @param session_id The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a unique 32-character ID.
 * @param ret_code The error code. This is valid when a STREAM_INPUT_TTS_EVENT_TASK_FAILED event occurs. For more information, see https://help.aliyun.com/document_detail/459864.html.
 * @param error_msg The detailed error message. This is valid when a STREAM_INPUT_TTS_EVENT_TASK_FAILED event occurs.
 * @param timestamp The timestamp information in the synthesis result.
 * @param all_response The complete response message in a JSON string. You can parse this string to get the required information.
 */
void onStreamInputTtsEventCallback(StreamInputTtsEvent event,
                                   String task_id, String session_id,
                                   int ret_code, String error_msg,
                                   String timestamp, String all_response);

Event list:

Name	Description
STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED	Speech synthesis has started and is ready for playback.
STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN	The server-side has detected the beginning of a sentence.
STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS	Returns the incremental speech synthesis result, including the latest audio and timestamps. The result is a full sentence within a sentence and incremental between sentences.
STREAM_INPUT_TTS_EVENT_SENTENCE_END	The server-side has detected the end of a sentence and returns the full timestamp for that sentence.
STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE	The server-side detects the end of a sentence and returns the full timestamp.
STREAM_INPUT_TTS_EVENT_TASK_FAILED	An error occurred during speech synthesis. For details, see ret_code and error_msg.

onStreamInputTtsDataCallback: The callback for synthetic data.

/**
 * Synthetic data callback.
 * @param data The synthesized audio data. Write this data to the player.
 */
void onStreamInputTtsDataCallback(byte[] data);

The following table describes the initialization parameters for the ticket. For an example of how to generate the ticket, see the code sample below.

Parameter	Type	Required	Description
url	String	No	The endpoint. The Beijing service is used by default.
app_key	String	Yes	The AppKey of the project created in the console.
token	String	Yes	Make sure the token is valid and has not expired.
debug_path	String	No	If save_log is true, operational logs are stored in this path.
complete_waiting_ms	Integer	No	The timeout period in milliseconds for waiting for the STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE event after calling stop. Default: 10,000 ms (10 seconds).

The following table describes the task parameters for `parameters`. For an example of how to generate the parameters, see the code sample below.

Parameter	Type	Required	Description
voice	String	Yes	The speaker's voice.
format	String	No	The audio coding format, such as pcm, wav, or mp3.
sample_rate	Integer	No	The audio sampling rate. Valid values: 8000, 16000, 24000, and 48000. Default: 24000.
volume	Integer	No	The playback volume. Range: 0 to 100. Default: 50.
speech_rate	Integer	No	The speech rate. Range: -500 to 500. Default: 0.
pitch_rate	Integer	No	The pitch. Range: -500 to 500. Default: 0.
enable_subtitle	Boolean	No	Enables word-level timestamps. For more information about how to use this feature, see Timestamp feature introduction.

playStreamInputTts: Synchronously synthesizes a single long text sentence. Each call performs one synthesis operation. The input text supports Speech Synthesis Markup Language (SSML).

/**
 * Synchronously synthesizes a single long sentence of text, returning after synthesis is complete. Do not call this method on the UI thread, as it may cause blocking.
 * @param callback The event listener callback. See the specific callbacks below.
 * @param ticket The initialization parameters in a JSON string. See the description below.
 * @param parameters The initialization parameters in a JSON string. See the description below.
 * @param text The single sentence of text to be synthesized. SSML is supported. Only UTF-8 encoded text is supported. A single synthesis request should contain fewer than 5,000 characters. The total cannot exceed 100,000 characters. One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.
 * @param session_id The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a unique 32-character ID.
 * @param level The log printing level. A smaller value prints more logs.
 * @param save_log Specifies whether to save logs to a file. The storage path is the value of the debug_path field in the ticket. Note: Log files have no size limit. Continuous storage may fill up the disk.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int playStreamInputTts(INativeStreamInputTtsCallback callback,
                                           String ticket, String parameters,
                                           String text, String session_id,
                                           int log_level, boolean save_log);

asyncPlayStreamInputTts: Asynchronously synthesizes a single long text sentence. Each call performs one synthesis operation. The input text supports SSML.

/**
 * Asynchronously synthesizes a single long sentence of text, returning immediately after the call. Do not call this method on the UI thread, as it may cause blocking.
 * @param callback The event listener callback. See the specific callbacks below.
 * @param ticket The initialization parameters in a JSON string. See the description below.
 * @param parameters The initialization parameters in a JSON string. See the description below.
 * @param text The single sentence of text to be synthesized. SSML is supported. Only UTF-8 encoded text is supported. A single synthesis request should contain fewer than 5,000 characters. The total cannot exceed 100,000 characters. One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.
 * @param session_id The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a unique 32-character ID.
 * @param level The log printing level. A smaller value prints more logs.
 * @param save_log Specifies whether to save logs to a file. The storage path is the value of the debug_path field in the ticket. Note: Log files have no size limit. Continuous storage may fill up the disk.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int asyncPlayStreamInputTts(INativeStreamInputTtsCallback callback,
                                                String ticket, String parameters,
                                                String text, String session_id,
                                                int log_level, boolean save_log);

stopStreamInputTts: Stops speech synthesis.

/**
 * Stops speech synthesis and waits to receive all synthesized data until the STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE event occurs.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int stopStreamInputTts();

asyncStopStreamInputTts: Asynchronously stops speech synthesis. The call returns immediately.

/**
 * Asynchronously stops speech synthesis. The call returns immediately without waiting to receive all synthesized data or the STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE event.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int asyncStopStreamInputTts();

cancelStreamInputTts: Immediately stops speech synthesis.

/**
 * Immediately stops speech synthesis. No callbacks are returned after this call.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int cancelStreamInputTts();

sendStreamInputTts: Sends the text to be synthesized.

/**
 * Sends the text to be synthesized.
 * @param text The text to be synthesized. Only UTF-8 encoded text is supported. A single synthesis request should contain fewer than 5,000 characters. The total cannot exceed 100,000 characters. One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.
 * @return For information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public synchronized int sendStreamInputTts(String text);

Procedure

Initialize the SDK and the player component.
Set parameters as needed.
Call startStreamInputTts to start streaming text-to-speech.
Call sendStreamInputTts to continuously send the text to be synthesized. In the synthetic data callback, stream the data to the player for playback.
Call stopStreamInputTts to indicate that all text has been sent and to wait for the synthesis to complete.
Handle the callback that indicates that speech synthesis is complete.

Code examples

Start speech synthesis

// Initialize the SDK.
NativeNui stream_input_tts_instance = new NativeNui(Constants.ModeType.MODE_STREAM_INPUT_TTS);
int ret = stream_input_tts_instance.startStreamInputTts(
        new INativeStreamInputTtsCallback() {},
        genTicket(), genParameters(), "",
        Constants.LogLevel.toInt(Constants.LogLevel.LOG_LEVEL_VERBOSE), false);

The genTicket method generates a JSON string that contains user information. This string includes the following fields. For more information about how to obtain the values for these fields, see the API reference.

/**
 * Sample for generating a ticket. For details, see the code sample in the demo project.
 */
private String genTicket() {
    String str = "";
    try {
        // Important:
        //  Before you use the interactive voice response service, you must prepare an account and enable the related services. For specific steps, see:
        //    https://help.aliyun.com/zh/isi/getting-started/start-here
        //
        // Primary account:
        //  Account (RAM user) information mainly includes an AccessKey ID (ak_id) and an AccessKey secret (ak_secret).
        //  Do not store this account information in the app code or on the client to prevent information leaks and financial loss.
        //
        // STS temporary credentials:
        //  Because sending account information to the client poses a risk of leaks, Alibaba Cloud provides a temporary access management service, Security Token Service (STS).
        //  STS uses the ak_id and ak_secret to generate temporary credentials: sts_ak_id, sts_ak_secret, and sts_token.
        //  (The prefix sts_ is used to distinguish these temporary credentials from the primary account information).
        // What is STS: https://help.aliyun.com/zh/ram/product-overview/what-is-sts
        // STS SDK overview: https://help.aliyun.com/zh/ram/developer-reference/sts-sdk-overview
        // STS Python SDK call example: https://help.aliyun.com/zh/ram/developer-reference/use-the-sts-openapi-example
        //
        // Account requirements:
        //  To use offline features (such as offline speech synthesis and wake-up words), you must provide an app_key, ak_id, and ak_secret, or an app_key, sts_ak_id, sts_ak_secret, and sts_token.
        //  To use online features (such as speech synthesis, real-time transcription, short sentence recognition, and audio file transcription), you only need to provide an app_key and a token.
        JSONObject object = Auth.getTicket(Auth.GetTicketMethod.GET_TOKEN_FROM_SERVER_FOR_ONLINE_FEATURES);
        if (!object.containsKey("token")) {
            Log.e(TAG, "Cannot get token!!!");
        }
        object.put("url", "wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1"); // Default, required.
        str = object.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    Log.i(TAG, "UserContext:" + str);
    return str;
}

The genParameters method generates a JSON string that contains task parameters. For more information, see the API reference.

/**
 * Sample for generating parameters. For details, see the code sample in the demo project.
 */
private String genParameters() {
    String str = "";
    try {
        JSONObject object = new JSONObject();
        object.put("enable_subtitle", true);
        object.put("voice", "zhixiaoxia");
        object.put("format", "pcm");
        object.put("sample_rate", 16000);
        object.put("volume", 50);
        object.put("speech_rate", 0);
        object.put("pitch_rate", 0);
        str = object.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    Log.i(TAG, "user parameters:" + str);
    return str;
}

Send text for synthesis in a stream

String sentence = "How do you make scrambled eggs with tomatoes?";
stream_input_tts_instance.sendStreamInputTts(sentence);

Stop speech synthesis

stream_input_tts_instance.stopStreamInputTts();

Handle callbacks

onStreamInputTtsEventCallback: The event callback for streaming text-to-speech. You can use this callback to control the player based on the synthesis status.

public void onStreamInputTtsEventCallback(
                    INativeStreamInputTtsCallback.StreamInputTtsEvent event, String task_id,
                    String session_id, int ret_code, String error_msg,
                    String timestamp, String all_response) {
    Log.i(TAG, "stream input tts event:" + event + " session id " + session_id + " session id " + task_id + " ret " + ret_code);
    if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED) {
        Log.i(TAG, "STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED");
        mAudioTrack.play();
        Log.i(TAG, "start play");
    } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS) {
        Log.i(TAG, "STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS:" + timestamp);
    } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE || event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_TASK_FAILED) {
        /*
         * Note: The STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE event indicates that TTS has finished synthesizing and has returned all audio data through the callback. It does not mean that the player has finished playing all the audio data.
         */
        Log.i(TAG, "play end");

        // Indicates that all data has been pushed. When the player finishes playback, a playOver callback is triggered.
        mAudioTrack.isFinishSend(true);

        if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_TASK_FAILED) {
            Log.e(TAG, "STREAM_INPUT_TTS_EVENT_TASK_FAILED error_code:" + ret_code + " errmsg:" + error_msg);
        }
    } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN) {
        Log.i(TAG, "STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN:" + all_response);
    } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_END) {
        Log.i(TAG, "STREAM_INPUT_TTS_EVENT_SENTENCE_END:" + all_response);
    }
}

onStreamInputTtsDataCallback: The callback for speech synthesis data. In this callback, you can write the synthesized data to the player for playback.

public void onStreamInputTtsDataCallback(byte[] data) {
      if (data.length > 0) {
          if (mEncodeType.equals("pcm")) {
              mAudioTrack.setAudioData(data);
          }
      }
  }