Stream ingest on HarmonyOS mobile clients

Prerequisites

• Create a real-time recording and obtain an ingest URL.

• Install the HarmonyOS mobile Push SDK.

SDK key interfaces

• initialize: Initializes the SDK.

  /**
  * Initializes the SDK. The SDK supports multiple instances. Release the existing instance before initializing a new one. Do not call this in the UI thread, as it may cause blocking.
  * @param callback: The event listener callback. For more information, see the specific callbacks described later in this topic.
  * @param parameters: The initialization parameters in a JSON string. For more information, see the description below or the API reference at https://help.aliyun.com/document_detail/173298.html.
  * @param level: The log printing level. A smaller value means more logs are printed.
  * @param save_log: Specifies whether to save logs to a file. The storage directory is the value of the debug_path field in the ticket. Note: Log files have no size limit. Continuous storage may fill up the disk.
  * @return: For more information, see Error codes at https://help.aliyun.com/document_detail/459864.html.
  */
  public initialize(callback:INativeNuiCallback ,
  			parameters:string ,
  			level:number ,
  			save_log:boolean=false ):number

  The parameters are as follows:

  Parameter	Type	Required	Description
  workspace	String	Yes	The path of the working directory. The SDK reads configuration files from this path.
  app_key	String	Yes	Must be set to "default".
  token	String	Yes	Must be set to "default".
  url	String	Yes	The MeetingJoinUrl returned when you create a Tingwu real-time recording task. This URL is used as the ingest URL to push the audio stream for real-time recognition.
  service_mode	String	Yes	Must be set to "1", which enables the online feature.
  device_id	String	Yes	The device ID. It uniquely identifies a device, such as a MAC address, serial number (SN), or UniquePsuedoID.
  debug_path	String	No	The debug directory. If the save_log parameter is set to true during SDK initialization, this directory is used to save log files.
  save_wav	String	No	This parameter takes effect when the save_log parameter is set to true during SDK initialization. It specifies whether to save debug audio. The data is saved in the debug directory. Ensure that debug_path is valid and writable. Note: Audio files have no size limit. Continuous storage may fill up the disk.

  The INativeNuiCallback type includes the following callbacks.

  • onNuiAudioStateChanged: Enables or disables the recording feature based on the audio state.

    /**
     * When interfaces such as start, stop, or cancel are called, the SDK uses this callback to notify the application to enable or disable recording.
     * @param state: The required state for recording (start/stop/close).
     */
    onNuiAudioStateChanged:(state:Constants.AudioState)=>void

  • onNuiNeedAudioData: Provides audio data in the callback.

    /**
     * When recognition starts, this callback is invoked continuously. The application needs to fill in the voice data in the callback.
     * @param buffer: The storage area to fill with voice data.
     * @return: The number of bytes actually filled.
     */
    
    onNuiNeedAudioData:(buffer:ArrayBuffer)=>number;

  • onNuiEventCallback: SDK event callback.

    /**
     * The main event callback for the SDK.
     * @param event: The callback event. For more information, see the event list below.
     * @param resultCode: For more information, see Error codes. This is valid when an EVENT_ASR_ERROR event occurs.
     * @param arg2: Reserved parameter.
     * @param kwsResult: Keyword spotting feature (not currently supported).
     * @param asrResult: The speech recognition result.
     */
    onNuiEventCallback:(event:Constants.NuiEvent, resultCode:number, arg2:number, kwsResult:KwsResult,
    							asrResult:AsrResult)=>void;

    Event list:

    Name	Description
    EVENT_VAD_START	Detects the start of human speech.
    EVENT_VAD_END	Detects the end of human speech.
    EVENT_ASR_PARTIAL_RESULT	Intermediate speech recognition result.
    EVENT_ASR_ERROR	Determines the cause of the error based on the error code information.
    EVENT_MIC_ERROR	Recording error. This indicates that the SDK has not received any audio for 2 consecutive seconds. Check if the recording system is working correctly.
    EVENT_SENTENCE_START	Real-time speech recognition event. Indicates that the start of a sentence is detected.
    EVENT_SENTENCE_END	Real-time speech recognition event. Indicates that the end of a sentence is detected and a complete result is returned.
    EVENT_SENTENCE_SEMANTICS	Not in use.
    EVENT_RESULT_TRANSLATED	Translation result.
    EVENT_TRANSCRIBER_COMPLETE	The final event after speech recognition stops.

  • onNuiAudioRMSChanged: Audio root mean square (RMS) value callback.

    /**
     * Audio RMS value callback.
     * @param val: The audio data RMS value callback. The range is from -160 to 0. It is generally used to display voice animations in the UI.
     */
    onNuiAudioRMSChanged:(val:number)=>number;

• setParams: Sets SDK parameters in JSON format.

  /**
   * Sets parameters in JSON format.
   * @param params: For more information, see the API reference at https://help.aliyun.com/document_detail/173298.html.
   * @return: For more information, see Error codes at https://help.aliyun.com/document_detail/459864.html.
   */
  public setParams(params:string):number

  Parameters:

  Parameter	Type	Required	Description
  service_type	Integer	Yes	Must be set to "4". This is the type of voice service to request. The value for Tingwu real-time stream ingest is "4".
  nls_config	JsonObject	Yes	Parameter settings for accessing the Voice Service. For more information, see the details below.
  nls_config.sr_format	String	Yes	Must be set to "pcm". When you create a Tingwu task by calling the CreateTask operation, also set the encoding format of the audio stream data to pcm.
  nls_config.sample_rate	Integer	Yes	The audio sampling rate. Default value: 16000 Hz. When you create a Tingwu task by calling the CreateTask operation, also specify the sampling rate of the audio stream data. The supported values are 8000 and 16000.

• startDialog: Starts recognition.

  /**
   * Starts recognition.
   * @param vad_mode: Multiple modes are available. For recognition scenarios, use P2T.
   * @param dialog_params: The dialog parameters in a JSON string. For more information, see the API reference at https://help.aliyun.com/document_detail/173298.html.
   * @return: For more information, see Error codes at https://help.aliyun.com/document_detail/459864.html.
   */
  public startDialog(vad_mode:Constants.VadMode, dialog_params:string):number

• stopDialog: Stops recognition. After this interface is called, the server-side returns the final recognition result and ends the task.

  /**
   * Stops the recognition process. After you call this method, the server-side returns the final recognition result and ends the task.
   * @return The error code. For more information, see https://help.aliyun.com/document_detail/459864.html.
   */
  public stopDialog():number

• cancelDialog: Stops recognition immediately.

  /**
   * Stops recognition immediately. After this interface is called, the task ends immediately without waiting for the server-side to return the final recognition result.
   * @return: For more information, see Error codes at https://help.aliyun.com/document_detail/459864.html.
   */
  public cancelDialog():number

• release: Releases the SDK.

  /**
   * Releases SDK resources.
   * @return: For more information, see Error codes at https://help.aliyun.com/document_detail/459864.html.
   */
  public release():number

• GetVersion: Retrieves the current SDK version information.

  /**
   * Gets the current SDK version information.
   * @return: The SDK version information as a string.
   */
  public GetVersion():string

Procedure

1. Initialize the SDK and the recording instance.

2. Set parameters as needed.

3. Call startDialog to start recognition.

4. Use the onNuiAudioStateChanged audio state callback to turn on the audio recorder.

5. Provide recording data in the onNuiNeedAudioData callback.

6. The EVENT_SENTENCE_START event callback indicates that the recognition of a sentence has started. You can then retrieve the intermediate recognition result from the EVENT_ASR_PARTIAL_RESULT event callback, the complete recognition result and related information for the sentence from the EVENT_SENTENCE_END event callback, and the translation result from the EVENT_RESULT_TRANSLATED event callback.

7. Call stopDialog to stop recognition. Confirm that recognition has stopped in the EVENT_TRANSCRIBER_COMPLETE event callback.

8. After the task is complete, call the release interface to release SDK resources.

Code examples

NUI SDK initialization

// Define the class NativeNuiCallbackHandle to implement the INativeNuiCallback interface.
class NativeNuiCallbackHandle implements INativeNuiCallback{
  // Implement the five interface functions in INativeNuiCallback internally.
  // Omitted here.
}

let context = getContext(this) as common.UIAbilityContext;
this.filesDir = context.filesDir;
this.resourceDir = context.resourceDir;

// Get the resource path here. Because the resource files are stored in the resfiles directory of the project, use the resfiles directory under the sandbox path.
let asset_path:string = this.resourceDir+"/resources_cloud"
// Because users cannot directly operate the device directory, set the debug path to the public directory filesDir under the sandbox path where the application is located.
let debug_path:string = this.filesDir

// Initialize the SDK. Note that users need to fill in the relevant ID information in genInitParams to use it.
cbhandle:NativeNuiCallbackHandle = new NativeNuiCallbackHandle()
g_asrinstance:NativeNui = new NativeNui(Constants.ModeType.MODE_DIALOG, "asr")
let ret:number = this.g_asrinstance.initialize(this.cbhandle, this.genInitParams(asset_path,debug_path), Constants.LogLevel.LOG_LEVEL_VERBOSE, false);
console.info("result = " + ret);
if (ret == Constants.NuiResultCode.SUCCESS) {
  console.error(`call g_asrinstance.initialize() return success`);
} else {
  // Throw an error exception message.
  console.error(`call g_asrinstance.initialize() return error:${ret}`);
}

The genInitParams function generates a JSON string that contains the resource directory and user information. The user information includes the following fields.

genInitParams(workpath:string, debugpath:string):string {
  let str:string = "";

  let object:Map<string, string|number|boolean|object> = new Map();

  // Account and project creation.
  //  To learn how to obtain an AccessKey ID, AccessKey secret, and app_key, see https://help.aliyun.com/document_detail/72138.html.
  object.set("app_key","default"); // Required
  object.set("token", "default"); // Required

  object.set("device_id", "meta60protestdevice"/*Utils.getDeviceId()*/); // Required. We recommend that you enter a unique ID to help locate problems.
  // Fill in the url with the generated MeetingJoinUrl.
  // Because the MeetingJoinUrl generation process involves an AccessKey ID and AccessKey secret, account information cannot be stored on the mobile client. Therefore, it needs to be generated on the server-side and sent to the mobile client.
  // For more information, see: https://help.aliyun.com/zh/tingwu/api-tingwu-2023-09-30-createtask?spm=a2c4g.11186623.0.i32
  object.set("url", "wss://tingwu-realtime-cn-hangzhou-pre.aliyuncs.com/api/ws/v1?XXXX"); // Required

  object.set("workspace", workpath); // Required, and read and write permissions are needed.

  // This parameter takes effect when the save_log parameter is set to true during SDK initialization. It specifies whether to save debug audio. This data is saved in the debug directory. Ensure that debug_path is valid and writable.
  //            object.put("save_wav", "true");
  // The debug directory. When the save_log parameter is set to true during SDK initialization, this directory is used to save intermediate audio files.
  object.set("debug_path", debugpath);

  object.set("service_mode", Constants.ModeFullCloud); // Required
  str = MapToJson(object)

  console.info("configinfo genInitParams:" + str);
  return str;
}

Parameter settings

Set the parameters as a JSON string.

import {MapToJson} from 'neonui'

// Set relevant recognition parameters. For more information, see the API documentation.
//  Call this after initialize() and before startDialog().
this.g_asrinstance.setParams(this.genParams());

genParams():string {
  let params:string = "";
  let nls_config:Map<string, string|number|boolean|object> = new Map();

  nls_config.set("sample_rate", 16000);
  nls_config.set("sr_format", "pcm");

  let parameters:Map<string, string|number|boolean|object> = new Map();
  parameters.set("nls_config", Object( JSON.parse(MapToJson(nls_config)) ) );
  parameters.set("service_type", Constants.kServiceTypeSpeechTranscriber); // Required

  params = MapToJson(parameters);
  console.log("configinfo genParams" + params)
  return params;
}

Start recognition

Use the startDialog interface to start the listener.

import {MapToJson} from 'neonui'

let ret:number = this.g_asrinstance.startDialog(Constants.VadMode.TYPE_P2T,
  this.genDialogParams());
console.info("start done . ret = ", ret);
if (ret != 0) {
  console.info("call startDialog return error. ", ret);
}
nui_instance.startDialog(Constants.VadMode.TYPE_P2T, genDialogParams());

genDialogParams():string {
  let params:string = "";
  let dialog_param:Map<string, string|number|boolean|object> = new Map();
  params = MapToJson(dialog_param);
  console.info("configinfo dialog params: " + params);
  return params;
}

Push recorded data

• updateAudio: In the callback function registered in the AudioCapturer's on('readData') method, call the updateAudio interface to send the recorded data to the SDK.

  //g_asrinstance.updateAudio(buffer,false)
  /*The 'readData' interface registered in AudioCapturer is AudioCapturer.readDataCallback.
   *AudioCapturer.audioCapturer.on('readData', AudioCapturer.readDataCallback);
  */
  class AudioCapturer{
    static  readDataCallback = (buffer: ArrayBuffer) => {
      console.log(`${TAG} read data byte length is ${buffer.byteLength}. uid[${process.uid}] pid[${process.pid}] tid[${process.tid}]`);
      AudioCapturer.g_asrinstance.updateAudio(buffer,false)
    }
  }

Callback handling

• onNuiAudioStateChanged: Recording state callback. The SDK internally maintains the recording state. Use this callback to control the audio recorder.

  /*
  For HarmonyOS development environment IDE versions earlier than 5.0.3.403, if the AudioCapturer module uses the registered callback [on("readData",)] to read recorded data, the callback may not be triggered if start is called directly after AudioCapturer.stop. In this case, the flow must be (stop, release) followed by (createAudioCapturer, start) to work correctly. After upgrading to IDE version 5.0.3.403, this issue is resolved. Therefore, the calls to the create/release interfaces are commented out in the following sample code.
  */
  onNuiAudioStateChanged(state:Constants.AudioState):void {
      console.info(`womx onUsrNuiAudioStateChanged(${state})`)
      if (state === Constants.AudioState.STATE_OPEN){
        console.info(`womx onUsrNuiAudioStateChanged(${state}) audio recorder start`)
        //AudioCapturer.init(g_asrinstance)
        AudioCapturer.start()
        console.info(`womx onUsrNuiAudioStateChanged(${state}) audio recorder start done`)
      } else if (state === Constants.AudioState.STATE_CLOSE){
        console.info(`womx onUsrNuiAudioStateChanged(${state}) audio recorder close`)
        AudioCapturer.stop()
        //AudioCapturer.release()
        console.info(`womx onUsrNuiAudioStateChanged(${state}) audio recorder close done`)
      } else if (state === Constants.AudioState.STATE_PAUSE){
        console.info(`womx onUsrNuiAudioStateChanged(${state}) audio recorder pause`)
        AudioCapturer.stop()
        //AudioCapturer.release()
        console.info(`womx onUsrNuiAudioStateChanged(${state}) audio recorder pause done`)
      }
    }

• onNuiNeedAudioData: Recorded data callback. Provide the recorded data in this callback.

  public int onNuiNeedAudioData(byte[] buffer, int len) {
    console.info(`warning, this callback should not be called in HarmonyOS Next`)
    return 0;
  }

• onNuiEventCallback: NUI SDK event callback. Do not call SDK interfaces in the event callback, because this may cause a deadlock.

  // The following variables are used during the event callback process to aggregate real-time recognition results, historical recognition results, real-time translation results, and historical translation results.
  asrspeechrealtimeResultOld:string=""
  asrmessage:string="";
  message:string="";    // Saves the event information of the current callback.
  asr_message_id:string="";
  asrspeechrealtimeResultTranslatedOld:string=""
  asrmessageTranslated:string="";
  
  onNuiEventCallback(event:Constants.NuiEvent, resultCode:number, arg2:number, kwsResult:KwsResult,
    asrResult:AsrResult):void {
    let asrinfo:string = ""
    console.log("onUsrNuiEventCallback last this callback handle.asrmessage is " + this.asrmessage)
    console.log("onUsrNuiEventCallback new this callback handle is " + JSON.stringify(this))
    console.log("onUsrNuiEventCallback event is " + event)
  
    if (event === Constants.NuiEvent.EVENT_TRANSCRIBER_COMPLETE){
      // Real-time recognition ends.
      this.message = "EVENT_TRANSCRIBER_COMPLETE"
      this.asrspeechrealtimeResultOld=""
      this.asrspeechrealtimeResultTranslatedOld=""
    } else if (event === Constants.NuiEvent.EVENT_SENTENCE_START){
      this.message = "EVENT_SENTENCE_START"
    } else if (event === Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT || event === Constants.NuiEvent.EVENT_SENTENCE_END){
      if (event === Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT ) {
        // For example, display the intermediate recognition result of the current sentence.
        this.message = "EVENT_ASR_PARTIAL_RESULT"
      } else if(event === Constants.NuiEvent.EVENT_SENTENCE_END){
        // For example, display the complete recognition result of the current sentence.
        this.message = "EVENT_SENTENCE_END"
      }
    } else if (event === Constants.NuiEvent.EVENT_RESULT_TRANSLATED){
      // For example, display the translation result of the current sentence.
      this.message = "EVENT_RESULT_TRANSLATED"
    } else if (event === Constants.NuiEvent.EVENT_ASR_ERROR){
      // In EVENT_ASR_ERROR, asrResult contains error information. Using it with the error code resultCode and the task_id makes it easier to troubleshoot problems. Please record and save this information.
      this.message = "EVENT_ASR_ERROR"
    } else if (event === Constants.NuiEvent.EVENT_MIC_ERROR){
      // EVENT_MIC_ERROR indicates that no audio data has been received for 2 seconds. Check the recording-related code, permissions, or whether the recording module is occupied by another application.
      this.message = "EVENT_MIC_ERROR"
    } else if (event === Constants.NuiEvent.EVENT_DIALOG_EX){ /* unused */
      // You can ignore this event.
      this.message = "EVENT_DIALOG_EX"
    }
    if (asrResult) {
      asrinfo = asrResult.asrResult
      console.log(`asrinfo asrResult string is ${asrinfo}. all response is [${asrResult.allResponse}]`)
      if (asrinfo) {
        try {
          let asrresult_json:object|null = JSON.parse(asrinfo)
          //console.log(JSON.stringify(asrresult_json))
          if (asrresult_json) {
            let payload:object|null = asrresult_json["payload"];
            if (payload) {
              //console.log(JSON.stringify(payload))
  
              if (event === Constants.NuiEvent.EVENT_RESULT_TRANSLATED) {
                // When a translation result event occurs, update the translation result in real time.
                
                if (payload["translate_result"]) {
                  // Update the current translation result in real time. 
                  console.log(`translate_result is [${payload["translate_result"][0]["text"]}], with oldinfo [${this.asrspeechrealtimeResultTranslatedOld}]`)
                  
                  // this.asrmessageTranslated saves the translation results of all sentences after the start. This includes the aggregated translation results of historical sentences + the real-time translation result of the current sentence.
                  this.asrmessageTranslated = this.asrspeechrealtimeResultTranslatedOld + payload["translate_result"][0]["text"];
                  
                }
                if (this.asr_message_id == asrresult_json["header"]["source_message_id"]){
                  // When the source_message_id corresponding to the translation result is the message_id of EVENT_SENTENCE_END, it means the translation of the sentence corresponding to EVENT_SENTENCE_END is finished. At this point, you can update the total translation result of historical sentences.
                  this.asrspeechrealtimeResultTranslatedOld = this.asrmessageTranslated
                }
              } else {
                // When a recognition result event occurs, update the recognition result in real time.
                if (payload["result"]) {
                  // this.asrmessage saves the recognition results of all sentences after the start. This includes the aggregated recognition results of historical sentences + the real-time recognition result of the current sentence.
                  this.asrmessage = this.asrspeechrealtimeResultOld + payload["result"];
                }
              }
  
              if(event === Constants.NuiEvent.EVENT_SENTENCE_END){
                // When an EVENT_SENTENCE_END event occurs, aggregate the historical recognition results and temporarily store the message_id of the last recognition result as a filter for the translation result of that sentence.
                
                // Aggregate the historical recognition results.
                this.asrspeechrealtimeResultOld = this.asrmessage
  
                // Temporarily store the message_id of the last recognition result as a filter for the translation result of that sentence.
                this.asr_message_id = asrresult_json["header"]["message_id"]
              }
            }
          }
        } catch (e){
          console.error("got asrinfo asrResult not json, so do not fresh asrinfo." + JSON.stringify(e))
        }
      }
    }
    console.info(`womx onUsrNuiEventCallback(${event}, ${resultCode},${arg2}, kwsResult:${kwsResult},asrResult:"${this.asrmessage}") done`)
  }

Stop recognition

nui_instance.stopDialog();

Release the SDK

nui_instance.release();