HarmonyOS Next SDK

Prerequisites

• Before you use the SDK, review the API reference. For more information, see API reference.

• Obtain your project's AppKey. For more information, see Create a project.

• You have obtained an access token. For more information, see Overview of obtaining an access token.

Download and installation

1. Download harmony_neonui_sdk.tar.gz.

2. To integrate the SDK as an arkts HAR package, decompress the package. The `entry/libs/neonui.har` file is the HAR package file generated by the SDK. Import and call this file in your project. To connect using HarmonyOS Next C++, find the dynamic library and header files in the `native/libs` and `native/include` directories of the decompressed package.

3. Open the project in DevEco Studio. You can find the sample code for audio file recognition in the `FileTranscriberPage.ets` file. In the `UserKey.ets` file, replace the AppKey and token in the `UserKey` class, and then run the project.

Key SDK interfaces

• initialize: Initializes the SDK.

  /**
   * Initializes the SDK. The SDK is a singleton. Release the SDK before you initialize it again. 
   * Do not call this method in the UI thread. It may cause blocking.
   * @param callback: The event listener callback. For more information, see the callback description that follows.
   * @param parameters: The initialization parameters. For more information, see the API reference.
   * @param level: The log printing level. A smaller value indicates that more logs are printed.
   * @param save_log: Specifies whether to save log information. The default value is false.
   * @return: For more information, see Error codes.
   */
  public initializeFileTrans(callback:INativeFileTransCallback ,
  				parameters:string ,
  				level:number ,
  				save_log:boolean=false):number 

  You must implement the `onFileTransEventCallback` callback for the `INativeFileTransCallback` type.

  onFileTransEventCallback: The event callback for file recognition.

  /**
   * The main event callback of the SDK.
   * @param event: The callback event. For more information, see the following event list.
   * @param resultCode: For more information, see Error codes. This parameter is valid when an EVENT_ASR_ERROR event occurs.
   * @param arg2: A reserved parameter.
   * @param asrResult: The speech recognition result.
   * @param taskId: The transcription task ID.
   */
  onFileTransEventCallback(event:Constants.NuiEvent, resultCode:number, arg2:number,
  				asrResult:AsrResult, taskId:string):void

  The following table lists the events:

  Name	Description
  EVENT_FILE_TRANS_CONNECTED	Successfully connected to the file recognition service.
  EVENT_FILE_TRANS_UPLOADED	The file is successfully uploaded.
  EVENT_FILE_TRANS_RESULT	The final recognition result.
  EVENT_ASR_ERROR	Determine the cause of the error based on the error code.

• setParams: Sets SDK parameters in JSON format.

  /**
   * Sets parameters in JSON format.
   * @param params: For more information, see the API reference.
   * @return: For more information, see Error codes.
   */
  public setParams(params:string):number

• startFileTranscriber: Starts file recognition.

  /**
   * Starts recognition.
   * @param params: The recognition parameters. For more information, see the API reference.
   * @param taskId: The ID of the transcription task to start. The SDK generates a random string.
   * @return: For more information, see Error codes.
   */
  public startFileTranscriber(params:string, task_id:ArrayBuffer):number

• cancelFileTranscriber: Stops recognition.

  /**
   * Stops recognition.
   * @return: For more information, see Error codes.
   */
  public cancelFileTranscriber(task_id:string):number

• release: Releases SDK resources.

  /**
   * Releases SDK resources.
   * @return: For more information, see Error codes.
   */
  public release():number

Procedure

1. Create an instance of the SDK class object.

2. Initialize the SDK.

3. Set parameters as needed.

4. Call `startFileTranscriber` to start recognition.

5. Obtain the final recognition result from the `EVENT_FILE_TRANS_RESULT` event.

6. After the call is complete, call the `release` method to release SDK resources.

Code examples

NUI SDK initialization

// Initialize the SDK. Note that you must enter the relevant ID information in Auth.getAliYunTicket to use the SDK.
    let ret:number = this.g_asrinstance.initializeFileTrans(this.cbhandle, this.genInitParams(asset_path,debug_path), Constants.LogLevel.LOG_LEVEL_VERBOSE, false);
    console.info("result = " + ret);
    if (ret == Constants.NuiResultCode.SUCCESS) {
    } else {
      // Throw an error exception message.
    }

    // Set the relevant recognition parameters. For more information, see the API reference.
    this.g_asrinstance.setParams(this.genParams());

`genInitParams` generates a JSON string that contains the resource directory and user information. The user information includes the following fields. `genParams` generates parameters in JSON string format. You must set these parameters after you call `initializeFileTrans()` and before you call `start`.

  genInitParams(workpath:string, debugpath:string):string {
    let str:string = "";
    // Method to obtain a token:

    let object:Map<string, string|number|boolean|object> = new Map();

    // Account and project creation
    // For information about how to obtain an ak_id, ak_secret, and app_key, see https://help.aliyun.com/document_detail/72138.html
    object.set("app_key", UserKey.app_key); // Required

    // Method 1:
    // For information about how to obtain an ak_id, ak_secret, and app_key, see https://help.aliyun.com/document_detail/72138.html
    // Then, see https://help.aliyun.com/document_detail/466615.html and use Method 1 to obtain temporary credentials.
    // Description: The remote server generates temporary credentials with a validity period and sends them to the mobile client. This ensures that the account information (ak_id and ak_secret) is not leaked.
    // Method to obtain a token (runs on the app server): https://help.aliyun.com/document_detail/450255.html?spm=a2c4g.72153.0.0.79176297EyBj4k
    object.set("token", UserKey.token); // Required

    // Method 2:
    // Using Security Token Service (STS) to obtain temporary credentials is not supported.

    // Method 3: (Not recommended. This method poses a risk of leaking your Alibaba Cloud account credentials.)
    // Refer to the implementation of the Auth class to access the Alibaba Cloud Token service on the client and obtain the SDK. Do not store your AccessKey pair (AK) in the local or client-side environment.
    // Advantage: The client obtains the token without requiring an app server.
    // Disadvantage: The client obtains the AK information, which can be easily leaked.
    //            JSONObject object = Auth.getAliYunTicket();

    object.set("device_id", "***"/*Utils.getDeviceId()*/); // Required. Enter a unique ID to help locate issues.
    object.set("url", UserKey.url_file_trans); // Default
    object.set("workspace", workpath); // Required. Read and write permissions are required.

    // This parameter takes effect when the save_log parameter is set to true during SDK initialization. It specifies whether to save audio for debugging. The data is saved in the debug directory. Make sure that debug_path is valid and writable.
    //            object.put("save_wav", "true");
    // The debug directory. When the save_log parameter is set to true during SDK initialization, this directory is used to save intermediate audio files.
    object.set("debug_path", debugpath);

    // FullMix = 0   // Select this mode to enable local features and register for authentication.
    // FullCloud = 1
    // FullLocal = 2 // Select this mode to enable local features and register for authentication.
    // AsrMix = 3    // Select this mode to enable local features and register for authentication.
    // AsrCloud = 4
    // AsrLocal = 5  // Select this mode to enable local features and register for authentication.
    object.set("service_mode", Constants.ModeFullCloud); // Required
    str = MapToJson(object)

    console.info("configinfo genInitParams:" + str);
    return str;
  }

  genParams():string {
    let params:string = "";
    let nls_config:Map<string, string|number|boolean|object> = new Map();
    nls_config.set("enable_intermediate_result", true);

    let parameters:Map<string, string|number|boolean|object> = new Map();
    parameters.set("nls_config", Object( JSON.parse(MapToJson(nls_config)) ) );

    params = MapToJson(parameters);//parameters.toString();
    console.log("configinfo genParams" + params)
    return params;
  }

Start recognition

Call the `startFileTranscriber` method to start recognition.

let task_id:ArrayBuffer = new ArrayBuffer(32);
let ret:number = this.g_asrinstance.startFileTranscriber(this.genDialogParams(), task_id);


genDialogParams():string {
    let params:string = "";
    let dialog_param:Map<string, string|number|boolean|object> = new Map();
    // To switch the app_key at runtime
    //dialog_param.put("app_key", "");
    let debug_path:string = this.resourceDir
    dialog_param.set("file_path", debug_path + "/2.wav"); // Use the resources/resfile/2.wav file in the DevEco Studio project directory as the input file for transcription.

    
    let nls_config:Map<string, string|number|boolean|object> = new Map();
    nls_config.set("format", "wav"); // Matches the audio file encoding format of the "file_path" field.


    dialog_param.set("nls_config", Object( JSON.parse(MapToJson(nls_config)) ) );

    params = MapToJson(dialog_param);

    console.info("configinfo dialog params: " + params);
    return params;
  }

Handle callbacks

onFileTransEventCallback: The NUI SDK event callback. Do not call SDK methods from within the event callback because this may cause a deadlock.

class NativeNuiCallbackHandle implements  INativeFileTransCallback{
  asrmessage:string;
  message:string;
  task_result: Map<string, string> = new Map();  // Caches the recognition results of multiple files.
  task_result_all:string="";                     // A summary string of the recognition results for all files.
  constructor() {
    this.asrmessage=""
    this.message=""
  }

  onFileTransEventCallback(event:Constants.NuiEvent, resultCode:number, finish:number,
				asrResult:AsrResult, taskId:string):void{

    let asrinfo:string = ""
    console.info("event=" + event + " task_id " + taskId);
    if (event == Constants.NuiEvent.EVENT_FILE_TRANS_UPLOADED) {
      this.message = "EVENT_FILE_TRANS_UPLOADED"
      showToast("Upload complete. Transcribing...");
    } else if (event == Constants.NuiEvent.EVENT_FILE_TRANS_RESULT) {
      this.message = "EVENT_FILE_TRANS_RESULT"
      showToast("Transcription complete.");
    } else if (event == Constants.NuiEvent.EVENT_ASR_ERROR) {
      this.message = "EVENT_ASR_ERROR"
      console.info("error happened: " + resultCode);
      showToast("error happened: " + resultCode);
    }

    if (asrResult) {
      asrinfo = asrResult.asrResult
      console.log(`asrinfo is ${asrinfo}`)
      if (!asrinfo){
        console.log(`asrinfo is null, with event ${event}`)
        return
      }
      let asrresult_json:object|null = JSON.parse(asrResult.asrResult)
      console.log(JSON.stringify(asrresult_json))
      if (asrresult_json) {
        let payload:object|null = asrresult_json["flash_result"];
        if (payload) {
          //console.log(JSON.stringify(payload))
          let filetransresultarray:Array<object> = payload["sentences"]
          let result_temp:string=""
          for (let i=0;i<filetransresultarray.length;i++){
            result_temp = result_temp +filetransresultarray[i]["text"]
          }
          this.asrmessage=result_temp

          this.task_result.set(taskId, this.asrmessage)
        }
      }
    }
    this.task_result_all=""
    let index:number = 0
    this.task_result.forEach((value:string, key:string)=>{
      this.task_result_all += `${index}==>${key}:${value};;\n`
      index += 1
    })
  }
}

Cancel recognition

FAQ