Streaming text-to-speech HarmonyOS Next SDK-Intelligent Speech Interaction(ISI)-阿里云帮助中心

This topic describes how to use the HarmonyOS Next NUI software development kit (SDK) from Alibaba Cloud Voice Service. It covers downloading and installing the SDK, its key interfaces, and code examples.

Prerequisites

Familiarize yourself with the interface description. For more information, see Interface description.
Obtain a project AppKey. For more information, see Create a project.
Obtain an Access Token. For more information, see Overview of obtaining a token.

Download and install

Download harmony_neonui_sdk.tar.gz.

Note

After downloading, you must replace the placeholder values for your Alibaba Cloud account information, Appkey, and Token in the sample initialization code to run the project.

Category	Compatibility
System	Supports HarmonyOS Next 5.0, API LEVEL 12, and DevEco Studio 5.0.3.403
Architecture	arm64-v8a

This SDK also includes the following features. If you need a feature that is not supported, you must obtain the corresponding SDK from its documentation.

Feature	Supported
Short sentence recognition	Yes
Real-time speech recognition	Yes
Speech synthesis	Yes
Real-time long-form text-to-speech	Yes
Streaming text-to-speech	Yes
Offline speech synthesis	No
Rapid audio file recognition	Yes
Wake words and command words	No
Tingwu real-time stream ingest	Yes

Integrate the SDK as an arkts HAR package. Decompress the package. The `entry/libs/neonui.har` file is the HAR package generated by the SDK. Import and call this file in your project. For HarmonyOS Next C++ integration, the dynamic libraries and header files are located in the `native/libs` and `native/include` directories.
Open the project in DevEco Studio. The sample code for streaming text-to-speech (TTS) is located in the `StreamTTSPage.ets` file. To run the project, replace the AppKey and Token in the `UserKeyStreamTTS` class within `UserKey.ets`. The test flow for the streaming TTS project is: `Index.ets` -> `StreamTTSPage.ets` -> `StreamTTS.ets`.

Key interfaces

startStreamInputTts: Starts a streaming TTS task.

/**
 * Starts streaming TTS. Do not call this function in the UI thread, as it may cause blocking.
 * @param callback: The event listener callback. See the specific callbacks below.
 * @param ticket: The initialization parameters in a JSON string format. See the description below.
 * @param parameters: The initialization parameters in a JSON string format. See the description below.
 * @param session_id: The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a unique 32-character ID.
 * @param level: The log printing level. A smaller value prints more logs.
 * @param save_log: Specifies whether to save logs to a file. The storage directory is the value of the `debug_path` field in the ticket. Note: Log files have no size limit. Continuous storage may fill up the disk.
 * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public startStreamInputTts( callback:INativeStreamInputTtsCallback,
				ticket:string, parameters:string,
				session_id:string, log_level:number,
				save_log:boolean):number

The INativeStreamInputTtsCallback type includes the following callbacks.

onStreamInputTtsEventCallback: This is the SDK event callback.

/**
 * Event callback
 * @param event: The callback event. See the event list below.
 * @param task_id: The task ID for the entire real-time speech synthesis session. This 32-character unique ID must remain consistent throughout the request.
 * @param session_id: The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a unique 32-character ID.
 * @param ret_code: The error code. This is valid when a `STREAM_INPUT_TTS_EVENT_TASK_FAILED` event occurs. For more information, see https://help.aliyun.com/document_detail/459864.html.
 * @param error_msg: The detailed error message. This is valid when a `STREAM_INPUT_TTS_EVENT_TASK_FAILED` event occurs.
 * @param timestamp: The timestamp information in the synthesis result.
 * @param all_response: The complete response message in a JSON string format. You can parse this string to get the required information.
 */
onStreamInputTtsEventCallback(event:StreamInputTtsEvent, task_id:string, session_id:string,
				ret_code:number, error_msg:string, timestamp:string,
				all_response:string):void;

The following table lists the events:

Name	Description
STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED	Speech synthesis has started and is ready for playback.
STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN	The server detected the start of a sentence.
STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS	Incrementally returns the speech synthesis result, including the latest audio and timestamps. The data is complete within a sentence and incremental between sentences.
STREAM_INPUT_TTS_EVENT_SENTENCE_END	The server detected the end of a sentence and returns the complete timestamp for that sentence.
STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE	The server detected the end of a sentence and returns the complete timestamp for that sentence.
STREAM_INPUT_TTS_EVENT_TASK_FAILED	An error occurred during speech synthesis. For details, see `ret_code` and `error_msg`.

onStreamInputTtsDataCallback: This callback receives the synthesized audio data.

/**
 * Synthetic data callback.
 * @param data: The synthesized audio data to be written to the player.
 */
onStreamInputTtsDataCallback(data:ArrayBuffer|null):void

The following table describes the initialization parameters for the ticket. For an example of how this is generated, see the code sample below.

Parameter	Type	Required	Description
url	String	No	The service endpoint. By default, the service in Beijing is used.
app_key	String	Yes	The AppKey of the project created in the console.
token	String	Yes	Make sure the token is valid and has not expired.
debug_path	String	No	If `save_log` is set to true, operational logs are stored in this path.
complete_waiting_ms	Integer	No	The timeout period in milliseconds (ms) to wait for the `STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE` event after `stop` is called. Default value: 10s.

The following table describes the task parameters. For an example of how this is generated, see the code sample below.

Parameter	Type	Required	Description
voice	String	Yes	The speaker's voice.
format	String	No	The audio encoding format, such as `pcm`, `wav`, or `mp3`.
sample_rate	Integer	No	The audio sampling rate. Valid values: 8000, 16000, 24000, and 48000. Default value: 24000.
volume	Integer	No	The playback volume. The range is 0 to 100. The default value is 50.
speech_rate	Integer	No	The speech rate. The range is -500 to 500. The default value is 0.
pitch_rate	Integer	No	The pitch rate. The range is -500 to 500. The default value is 0.
enable_subtitle	Boolean	No	Enables word-level timestamps. For more information about how to use this feature, see Timestamp feature introduction.

stopStreamInputTts: Stops the speech synthesis task.

/**
 * Stops speech synthesis and waits to receive all synthesized data until the `STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE` event.
 * @param flag_async: Specifies whether to use an asynchronous process for the stop operation. If set to true, the `stopStreamInputTts` interface call is asynchronous, the function returns quickly, and the synthesis `COMPLETE` event is returned asynchronously. If set to false, the interface call blocks for a specified time until a `COMPLETE` or `FAILED` event occurs. The default value is true to prevent user thread blocking and timeouts.
 * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public stopStreamInputTts(flag_async:boolean=true):number

sendStreamInputTts: Sends text for synthesis.

/**
 * Sends text for synthesis.
 * @param text: The text to be synthesized. Only text encoded in UTF-8 is supported. A single synthesis request should contain fewer than 5,000 characters, and the total must not exceed 100,000 characters. One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.
 * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
public sendStreamInputTts(text:string):number

Procedure

Create an instance of the SDK class object.
Initialize the SDK and the player component.
Set the required parameters.
Call `startStreamInputTts` to start streaming text-to-speech.
Continuously call `sendStreamInputTts` to send text for synthesis. In the synthetic data callback, write the received data to the player for playback. We recommend using streaming playback.
Call `stopStreamInputTts` to signal that all text has been sent and to wait for the synthesis process to complete.
Handle the callback that indicates the end of speech synthesis or a synthesis error.

Code examples

Start speech synthesis

// Initialize the SDK
// Import the required SDK modules
import {Constants, NativeNui, INativeStreamInputTtsCallback, StreamInputTtsEvent} from 'neonui'

// Implement the callback handler function. For the specific implementation, see the callback code section at the end of this topic.
const g_ttscallback_instance:INativeStreamInputTtsCallback = {
  onStreamInputTtsEventCallback: cb_tts_event_callback,
  onStreamInputTtsDataCallback: cb_tts_user_data_callback
};

// Create an instance of the SDK class object
NativeNui stream_input_tts_instance = new NativeNui(Constants.ModeType.MODE_STREAM_INPUT_TTS);
// Call startStreamInputTts to start streaming text-to-speech.
let ret:number = this.stream_input_tts_instance.startStreamInputTts(
  g_ttscallback_instance,
  genTicketTTS(), 
  genParameters(this.fontname), "", 
  Constants.LogLevel.toInt(Constants.LogLevel.LOG_LEVEL_VERBOSE), false )

The `genTicketTTS` function generates a JSON string that contains user information. This string includes the following fields. For information about how to obtain these values, see the interface description document.

/**
 * Ticket generation example. For details, see the code sample in the demo project.
 */
function genTicketTTS():string {
  // Important:
  //  Before you use the interactive voice response service, prepare an account and activate the related services. For specific steps, see:
  //    https://help.aliyun.com/zh/isi/getting-started/start-here
  //
  // Primary account:
  //  Account (RAM user) information mainly includes an AccessKey ID (ak_id) and an AccessKey secret (ak_secret).
  //  Do not store this account information in the app code or on the client to prevent information leaks and financial losses.
  //
  // STS temporary credentials:
  //  Because sending account information to the client poses a risk of leaks, Alibaba Cloud provides a temporary access control service, Security Token Service (STS).
  //  STS uses the ak_id and ak_secret of an account to generate temporary credentials: sts_ak_id, sts_ak_secret, and sts_token.
  //  (The prefix sts_ is used to distinguish STS temporary credentials from primary account information.)
  // What is STS: https://help.aliyun.com/zh/ram/product-overview/what-is-sts
  // STS SDK overview: https://help.aliyun.com/zh/ram/developer-reference/sts-sdk-overview
  // STS Python SDK call example: https://help.aliyun.com/zh/ram/developer-reference/use-the-sts-openapi-example
  //
  // Account requirements:
  //  To use offline features (offline speech synthesis, wake words), you must provide an app_key, ak_id, and ak_secret, or an app_key, sts_ak_id, sts_ak_secret, and sts_token.
  //  To use online features (speech synthesis, real-time transcription, short sentence recognition, audio file transcription, etc.), you only need an app_key and a token.
  let str:string = "";
  // Method to obtain a token:
  let object:object = Object({
    "appkey" : UserKeyStreamTTS.app_key,
    "token" : UserKeyStreamTTS.token,
    "url" : UserKeyStreamTTS.url
  })
  str = JSON.stringify(object);
  console.info("in genTicketTTS, UserContext:" + str);
  return str;
}

The `genParameters` function generates a JSON string that contains parameter information. For more information, see the interface description document.

function genParameters(voice:string):string {
  let str:string = "";
  let object:object = Object({
    "enable_subtitle" : "1",
    "voice" : "zhixiaoxia",
    "format" : "pcm",
    "sample_rate" : "16000",
    "volume" : "50",
    "speech_rate" : "0",
    "pitch_rate" : "0"
  })
  str = JSON.stringify(object);
  console.info("in genParameters, UserContext:" + str);
  return str;
}

Send text for synthesis in a stream

let ttstext1:string = "How do you make scrambled eggs with tomatoes?";
let ttstext2:string = "It has to be a recipe that looks, smells, and tastes great.";
console.log(`womx ttstext1 is ${ttstext1}`);
this.stream_input_tts_instance.sendStreamInputTts(ttstext1)
console.log(`womx ttstext2 is ${ttstext2}`);
this.stream_input_tts_instance.sendStreamInputTts(ttstext2)

End speech synthesis

this.stream_input_tts_instance.stopStreamInputTts()

Handle callbacks

onStreamInputTtsEventCallback: This is the event callback for streaming text-to-speech. You can use it to control the player based on the speech synthesis status.

// Implement the callback handler function
let countNumber:number = 0
function cb_tts_event_callback(event:StreamInputTtsEvent, task_id:string, session_id:string,
  ret_code:number, error_msg:string, timestamp:string,
  all_response:string):void{
  //console.info("womx cb_tts_event_callback  %d %s %d", event, task_id, code);

  console.info( "stream input tts event:" + event + " session id " + session_id + " session id " + task_id + " ret " + ret_code);
  if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED) {
    console.info("STREAM_INPUT_TTS_EVENT_SYNTHESIS_STARTED");
    //waitinginit()  // When the synthesis start event is returned, initialize the player to prevent status exceptions caused by premature initialization.

    countNumber=0
    console.info("start play");
  } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS) {
    console.info("STREAM_INPUT_TTS_EVENT_SENTENCE_SYNTHESIS:" + timestamp);
  } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE || event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_TASK_FAILED) {
    /*
     * Note: The STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE event indicates that TTS has finished synthesizing and has returned all audio data through the callback. It does not mean that the player has finished playing all the audio data.
     */

    // Synthesis is complete or an error occurred. Respond accordingly.
    console.info("play end");

    // Indicates that data push is complete
    //AudioRenderer.voiceEnd() // Synthesis is finished. Notify the player that the data has ended and wait for playback to complete.
    is_started = false;

    if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_TASK_FAILED) {
      console.info("STREAM_INPUT_TTS_EVENT_TASK_FAILED error_code:" + ret_code + " errmsg:" + error_msg);
      //AudioRenderer.voiceStop(true)  // A synthesis error occurred. Stop the player playback directly.
    }
  } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN) {
    console.info("STREAM_INPUT_TTS_EVENT_SENTENCE_BEGIN:" + all_response);
  } else if (event == StreamInputTtsEvent.STREAM_INPUT_TTS_EVENT_SENTENCE_END) {
    console.info("STREAM_INPUT_TTS_EVENT_SENTENCE_END:" + all_response);
  }
}

onStreamInputTtsDataCallback: This is the speech synthesis data callback. Write the synthesized data from this callback to the player for playback.

function cb_tts_user_data_callback(buffer:ArrayBuffer|null):void{
  console.log(`womx cb_tts_user_data_callback uid[${process.uid}] pid[${process.pid}] tid[${process.tid}]`);
  countNumber+=1
  if (1==countNumber) {
    //waitingstart()  // The first frame of synthesized data is returned. Start the player at this time.
  }
  if (buffer){
    console.info("womx cb_tts_user_data_callback %d. times=%d", buffer.byteLength, countNumber);
    if (buffer.byteLength > 0) {
      // The synthesized data is sent to the player module for real-time playback. Note: A large amount of data is synthesized in a short period. You need to manage the balance between the data cache and the player's streaming playback progress.
      /*if (gFlagAudioRendererUseCallback) {
        AudioRenderer.setVoiceArrayBuffer(buffer as ArrayBuffer)
      } else {
        AudioRenderer.writePlayerData(buffer)
      }*/
    }
  } else {
    console.info("womx cb_tts_user_data_callback undefined");
  }
}

FAQ

How do I handle application unresponsiveness or termination caused by interface call timeouts?

You can refer to the implementation of asynchronous calls in `OneSentenceAsrWorker.ets` in the sample code.