HarmonyOS Next SDK

Prerequisites

• For more information, see the API description.

• You have obtained an Appkey for your project. For more information, see Create a project.

• You have obtained an Access Token. For more information, see Overview of obtaining a token.

Download and install

1. Download harmony_neonui_sdk.tar.gz.

   Important

   After you download the SDK, you must replace the placeholder Alibaba Cloud account information, Appkey, and Token in the sample initialization code before you run the code.

   Category	Compatibility
   System	Supports HarmonyOS Next version 5.0, API LEVEL 12, and DevEco Studio version 5.0.3.403
   Architecture	arm64-v8a

   This SDK also includes other features. If a feature you need is not supported, go to the corresponding documentation to obtain the required SDK.

   Feature	Support
   One-sentence recognition	Yes
   Real-time speech recognition	Yes
   Speech synthesis	Yes
   Real-time long-text speech synthesis	Yes
   Streaming text-to-speech synthesis	Yes
   Offline speech synthesis	No
   Express audio file recognition	Yes
   Wake-up and command words	No
   Tingwu real-time stream ingest	Yes

2. Integrate the SDK as an ArkTS HAR package. After you decompress the archive, find the SDK-generated HAR file at entry/libs/neonui.har and import it into your project. To use the HarmonyOS Next C++ integration method, obtain the dynamic libraries and header files from the native/libs and native/include directories in the archive.

3. Open the project directory in DevEco Studio. The speech synthesis sample code is located in the TTSPage.ets file. Replace the Appkey and Token in the UserKey class within UserKey.ets, and then run the project.

Key SDK interfaces

• tts_initialize: Initializes the SDK.

  /**
   * Initialize the SDK. The SDK is a singleton. Release it before reinitializing.
   * Do not call this method on the UI thread, as it may cause blocking.
   * @param callback: Event listener callback. See specific callbacks below.
   * @param ticket: Initialization parameters in JSON string format. See the description below or the API documentation: https://help.aliyun.com/document_detail/173642.html.
   * @param level: Log printing level. A lower value means more logs are printed.
   * @param save_log: Whether to save logs to a file. The storage directory is specified by the debug_path field in the ticket.
   * Note: Log files have no size limit. Be cautious about continuous storage filling up the disk.
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public tts_initialize(callback:NuiTtsSdkListener ,
                              ticket:string ,
                              level:number ,
                              save_log:boolean ):number

  The NuiTtsSdkListener type includes the following callbacks.

  • onTtsEventCallback: The SDK event callback.

    /**
     * Event callback
     * @param event: Callback event. See the event list below.
     * @param task_id: Task ID of the request.
     * @param ret_code: Error code. Valid only when the TTS_EVENT_ERROR event occurs. See https://help.aliyun.com/document_detail/459864.html.
     */
    onTtsEventCallback: (event:NuiSdkTtsEvent, taskid:string, ret_code:number) => void;

    Event list:

    Name	Description
    TTS_EVENT_START	Speech synthesis starts. Prepare for playback.
    TTS_EVENT_END	Speech synthesis ends. All synthesized data has been delivered, but playback may not be complete.
    TTS_EVENT_CANCEL	Cancel speech synthesis.
    TTS_EVENT_PAUSE	Pause speech synthesis.
    TTS_EVENT_RESUME	Resume speech synthesis.
    TTS_EVENT_ERROR	An error occurred during speech synthesis. Call getparamTts("error_msg") to get detailed error messages.

  • onTtsDataCallback: The synthesized data callback.

    /**
     * Synthesized data callback. When enable_subtitle is enabled, info and data are returned alternately.
     * @param info: Returns timestamp results in JSON format when the timestamp feature is used.
     * @param info_len: Data length of the info field. Not currently used.
     * @param data: Synthesized audio data. Write to the player.
     */
    onTtsDataCallback: (info:string, info_len:number , buffer:ArrayBuffer|null) => void;

    The following describes the parameters in the ticket. For more information about how to generate the ticket, see the code example.

    Parameter	Type	Required	Description
    workspace	String	Yes	Working directory path. The SDK reads configuration files from this path. Read and write permission is required.
    app_key	String	Yes	Appkey created for your project in the console.
    token	String	Yes	Ensure this Token is valid and within its validity period. You can set the Token during initialization or update it later using parameter settings.
    device_id	String	Yes	User-level account ID. Ensure it is unique.
    mode_type	String	Yes	Set to online speech synthesis mode. Speech synthesis must be set to 2. Otherwise, the application will fail to run.
    tts_version	String	Yes	Set the speech synthesis mode. 1: Long-text speech synthesis (over 300 characters) 0: Short-text speech synthesis (up to 300 characters)
    custom_params	String	No	To configure parameters that are supported by the interaction protocol but not documented in the API reference, use this universal setting interface. custom_params is the key, and the value is a JSON string. See the code example for specific usage.

• setparamTts: Sets the TTS parameters.

  /**
   * Set parameters as key-value pairs. See API documentation: https://help.aliyun.com/document_detail/173642.html
   * @param param: Parameter name. See API documentation.
   * @param value: Parameter value. See API documentation.
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public setparamTts(param:string, value:string):number

• getparamTts: Retrieves a parameter.

  /**
   * Get a parameter value
   * @param param: Parameter name. See API documentation: https://help.aliyun.com/document_detail/173642.html.
   * @return: Parameter value.
   */
  public getparamTts(param:string):string

• startTts: Starts playback.

  /**
   * Start a synthesis task
   * @param priority: Task priority. Use "1".
   * @param taskid: Task ID. Pass a 32-byte UUID or leave empty for the SDK to generate one.
   * @param text: Text to synthesize.
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public startTts(priority:string, taskid:string, text:string):number

• cancelTts: Cancels playback.

  /**
   * Cancel a synthesis task
   * @param taskid: Task ID to stop. If empty, cancel all tasks.
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public cancelTts(taskid:string):number

• pauseTts: Pauses playback.

  /**
   * Pause a synthesis task
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public pauseTts():number

• resumeTts: Resumes playback.

  /**
   * Resume a paused task
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public resumeTts():number

• tts_release: Releases the SDK resources.

  /**
   * Release the SDK
   * @return: See error codes: https://help.aliyun.com/document_detail/459864.html.
   */
  public tts_release():number

Procedure

1. Create an instance of the SDK class object.

2. Initialize the SDK and the playback component.

3. Set the parameters as needed.

4. Call startTts to start playback.

5. In the synthesized data callback, write the data to the player for streaming playback.

6. Receive the callback that indicates that the speech synthesis is complete.

Code examples

Speech synthesis initialization

// Get the resource path, which is the working directory. Use the fixed path context.resourceDir+"/resources_cloud"
ttsLongText:boolean = false;  // Default: non-long-text TTS configuration
ticket: string = "this is ticket string"

resourceDir:string = ""
let context = getContext(this) as common.UIAbilityContext;
this.resourceDir = context.resourceDir;

// SDK initialization
g_ttsinstance:NativeNui = new NativeNui(Constants.ModeType.MODE_TTS, "")

let path:string = this.resourceDir+"/resources_cloud"
this.ticket = genTicketTTS(path)
console.info("ticket is %s", this.ticket);

let level:number= 1
let save_log:boolean = false
let retcode:number = this.g_ttsinstance.tts_initialize(g_ttscallback_instance,this.ticket,level, save_log)

The genTicket method generates a JSON string that contains the resource directory and user information. The user information includes the following fields. For more information about how to obtain the fields, see the API documentation.

/**
 * Ticket generation example. See the demo project for details.
 */
function genTicketTTS(workpath:string):string {
  let str:string = "";
  // Note:
  //  Prepare your account and activate the required services before using Voice Service.
  //  For steps, see:
  //    https://help.aliyun.com/zh/isi/getting-started/start-here
  //
  // Primary account:
  //  Account (RAM user) information includes AccessKey ID (referred to as ak_id below) and AccessKey Secret (referred to as ak_secret below).
  //  Never store this account information in app code or on mobile devices to prevent leaks and financial loss.
  //
  // STS temporary credentials:
  //  Because distributing account information to clients risks exposure, Alibaba Cloud provides STS (Security Token Service) for temporary access management.
  //  STS uses ak_id and ak_secret to generate temporary sts_ak_id, sts_ak_secret, and sts_token.
  //  (To distinguish primary account information from STS temporary credentials, the prefix sts_ indicates STS-generated temporary credentials.)
  // What is STS: https://help.aliyun.com/zh/ram/product-overview/what-is-sts
  // STS SDK overview: https://help.aliyun.com/zh/ram/developer-reference/sts-sdk-overview
  // STS Python SDK example: https://help.aliyun.com/zh/ram/developer-reference/use-the-sts-openapi-example
  //
  // Account requirements:
  //  For offline features (offline speech synthesis, wake-up), you must provide app_key, ak_id, and ak_secret, or app_key, sts_ak_id, sts_ak_secret, and sts_token.
  //  For online features (speech synthesis, real-time transcription, one-sentence recognition, audio file transcription, etc.), you only need app_key and token.
  
  // Set "mode_type" to online synthesis:
  //  Local = 0,
  //  Mix = 1,  // init local and cloud
  //  Cloud = 2,

  let object:object = Object({
    "app_key" : UserKeyTTS.app_key,
    "token" : UserKeyTTS.token,
    "url" : UserKeyTTS.url,
    "device_id" : "empty_device_id_womx",  // Required. Use a unique ID for easier troubleshooting.
    "mode_type" : "2",
    "workspace" : workpath    // Required. Working directory path. The SDK reads configuration files from this path and requires read and write permission.
  })
  str = JSON.stringify(object);
  console.info("UserContext:" + str);
  return str;
}

Set parameters as needed

// For detailed parameters, see: https://help.aliyun.com/document_detail/173642.html

//  For online speech synthesis voices, refer to the Alibaba Cloud website:
//  https://help.aliyun.com/document_detail/84435.html
this.g_ttsinstance.setparamTts("font_name", "xiaoyun");

//  Set the sample rate for the selected voice. Also set the corresponding sample rate in your player; otherwise, audio playback will fail.
this.g_ttsinstance.setparamTts("sample_rate", "16000");

//  Enable word-level phoneme boundary detection. This parameter only works for voices that support word-level phoneme boundaries. "1" enables it; "0" disables it.
this.g_ttsinstance.setparamTts("enable_subtitle", "1");

//  Set parameters not documented in the API reference. Use "custom_params" as the key and a JSON string as the value.
// this.g_ttsinstance.setparamTts("custom_params",{\"enable_phoneme_timestamp\":true}");

//  Adjust speech rate
// this.g_ttsinstance.setparamTts("speed_level", "1");
//  Adjust pitch
// this.g_ttsinstance.setparamTts("pitch_level", "0");
//  Adjust volume
// this.g_ttsinstance.setparamTts("volume", "1.0");

//  Supports synthesizing text up to 300 characters. Each Chinese character, English letter, or punctuation mark counts as one character.
//  Text longer than 300 characters will be truncated. Ensure your input text is under 300 characters (excluding SSML formatting).
//  Short-text and long-text synthesis have different pricing. Activate long-text synthesis separately if needed.
//  Skip the following if you do not require long-text synthesis.
let charNum: number = this.g_ttsinstance.getUtf8CharsNum(ttsText);
console.info("chars:" + charNum + " of text:" + ttsText);
if (this.ttsLongText || charNum > 300) { // ttsLongText controls whether long-text synthesis is enabled
  console.info(`tts text with config ${this.ttsLongText} ,and chars number ${charNum}.`);
  // For text over 300 characters, enable long-text synthesis mode
  this.g_ttsinstance.setparamTts("tts_version", "1");
} else {
  // For text under 300 characters, use short-text synthesis mode
  this.g_ttsinstance.setparamTts("tts_version", "0");
}

Start speech synthesis

// One task per instance. To handle multiple tasks simultaneously, create multiple instances.
this.g_ttsinstance.startTts("1", "", ttsText);

Callback handling

• onTtsEventCallback: The speech synthesis event callback. You can control the player based on the synthesis status.

  function cb_tts_event_callback(event:NuiSdkTtsEvent, task_id:string, code:number):void{
    console.info("womx cb_tts_event_callback  %d %s %d", event, task_id, code);
  
    console.log(`womx cb_tts_event_callback uid[${process.uid}] pid[${process.pid}] tid[${process.tid}]`);
    if (event==NuiSdkTtsEvent.TTS_EVENT_START){
      waitinginit() // Initialize AudioRenderer and start playback
    } else if (event==NuiSdkTtsEvent.TTS_EVENT_END){
      /*
       * Note: TTS_EVENT_END means all synthesized audio data has been delivered via callbacks, not that playback has finished.
       */
      console.info("womx call voiceEnd()");
      AudioRenderer.voiceEnd() // Notify AudioRenderer that data delivery is complete. Play remaining data and stop.
    } else if (event == TtsEvent.TTS_EVENT_PAUSE) {
        console.info("womx call pause()");
    } else if (event == TtsEvent.TTS_EVENT_RESUME) {
        console.info("womx call resume()");
    } else if(event==NuiSdkTtsEvent.TTS_EVENT_CANCEL || event==NuiSdkTtsEvent.TTS_EVENT_ERROR ){
      console.info("womx call stop()");
      AudioRenderer.voiceStop(true) // Call await AudioRenderer.stop(true)
    }
  }

• onTtsDataCallback: The speech synthesis data callback. Write the synthesized data from the callback to the player for playback.

  function cb_tts_user_data_callback(info:string, info_len:number , buffer:ArrayBuffer|null):void{
    if (buffer){
      console.info("womx cb_tts_user_data_callback %s %d %d. times=%d", info, info_len, buffer.byteLength, countNumber);
      if (buffer.byteLength > 0) {
        AudioRenderer.writePlayerData(buffer)
      }
    } else {
      console.info("womx cb_tts_user_data_callback %s %d undefined", info, info_len);
    }
  }

Cancel speech synthesis

this.g_ttsinstance.cancelTts("")

Exit speech synthesis

this.g_ttsinstance.tts_release()