HarmonyOS Next SDK

Initialize the SDK

Import the SDK types. The callback parameter of createRecognizer is the callback implementation shown below. The application provides the other parameters.

import { NativeNui, Constants, INativeNuiCallback, KwsResult, AsrResult } from 'neonui';

function createRecognizer(callback: INativeNuiCallback, resourceDir: string,
  filesDir: string, appkey: string, token: string, deviceId: string): NativeNui {
  const nui = new NativeNui(Constants.ModeType.MODE_DIALOG, 'asr');
  const parameters: Record<string, string> = {
    app_key: appkey,
    token: token,
    device_id: deviceId,
    url: 'wss://nls-gateway.cn-shanghai.aliyuncs.com/ws/v1',
    workspace: resourceDir + '/resources_cloud',
    debug_path: filesDir,
    service_mode: Constants.ModeAsrCloud
  };
  const ret = nui.initialize(callback, JSON.stringify(parameters),
    Constants.LogLevel.LOG_LEVEL_VERBOSE, false);
  if (ret !== Constants.NuiResultCode.SUCCESS) {
    throw new Error('initialize failed: ' + ret);
  }
  return nui;
}

Provide app_key, token, a unique device_id, and a readable and writable workspace. debug_path specifies the debug file directory. Enable save_log only when needed. Log files have no size limit and must be cleaned up. To save debug audio with save_wav, enable save_log and provide a valid, writable debug directory.

Configure recognition

Set recognition parameters after initialization succeeds and before starting recognition.

function configureRecognizer(nui: NativeNui, vadMode: boolean): number {
  const nlsConfig: Record<string, Object> = {
    enable_intermediate_result: true,
    enable_punctuation_prediction: true,
    enable_inverse_text_normalization: true,
    enable_voice_detection: vadMode
  };
  if (vadMode) {
    nlsConfig['max_start_silence'] = 10000;
    nlsConfig['max_end_silence'] = 800;
  }
  const parameters: Record<string, Object> = {
    nls_config: nlsConfig,
    service_type: Constants.kServiceTypeASR
  };
  return nui.setParams(JSON.stringify(parameters));
}

You can configure customization_id, vocabulary_id, sample_rate (for example, 16000), and sr_format (for example, opus) in nls_config as needed. For parameter conditions, see the API reference. For audio encoding limits, see the FAQ below.

Start recognition

function startRecognizer(nui: NativeNui, token: string): number {
  const dialogParams: Record<string, string> = { token: token };
  return nui.startDialog(Constants.VadMode.TYPE_P2T, JSON.stringify(dialogParams));
}

Use Constants.VadMode.TYPE_P2T. To update the token or Appkey for a recognition session, pass token or app_key in dialog_params. If a subsequent session does not specify these temporary parameters, the initialization parameters are used.

The online SDK does not include a local VAD module. To detect the start and end of speech, enable cloud VAD with enable_voice_detection as shown above. Do not use TYPE_VAD, which requires an SDK with offline capabilities.

Send recorded audio

Call the following function from the recorder’s readData callback and pass audio in capture order. The SDK package provides the AudioRecorder wrapper.

function pushAudio(nui: NativeNui, buffer: ArrayBuffer): number {
  return nui.updateAudio(buffer, false);
}

Handle callbacks

The RecordingController interface below represents application recording controls and must be connected to the recorder: start starts recording and stop stops recording. Use onNuiAudioStateChanged to control capture and send the recorded data to the same SDK instance.

interface RecordingController {
  start(): void;
  stop(): void;
}

interface RecognitionPayload {
  result: string;
}

interface RecognitionResponse {
  payload: RecognitionPayload;
}

class NativeNuiCallbackHandle implements INativeNuiCallback {
  private recorder: RecordingController;
  constructor(recorder: RecordingController) {
    this.recorder = recorder;
  }

  onNuiAudioStateChanged(state: Constants.AudioState): void {
    if (state === Constants.AudioState.STATE_OPEN) {
      this.recorder.start();
    } else if (state === Constants.AudioState.STATE_CLOSE ||
      state === Constants.AudioState.STATE_PAUSE) {
      this.recorder.stop();
    }
  }

  onNuiNeedAudioData(buffer: ArrayBuffer): number {
    return 0;
  }

  onNuiEventCallback(event: Constants.NuiEvent, resultCode: number, arg2: number,
    kwsResult: KwsResult, asrResult: AsrResult): void {
    if (event === Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT ||
      event === Constants.NuiEvent.EVENT_ASR_RESULT) {
      if (asrResult && asrResult.asrResult) {
        try {
          const response = JSON.parse(asrResult.asrResult) as RecognitionResponse;
          if (response.payload) {
            console.info(response.payload.result);
          }
        } catch (e) {
          console.error('Invalid recognition response');
        }
      }
    } else if (event === Constants.NuiEvent.EVENT_ASR_ERROR ||
      event === Constants.NuiEvent.EVENT_MIC_ERROR) {
      console.error('recognition error: ' + resultCode);
    }
  }

  onNuiAudioRMSChanged(val: number): number {
    return 0;
  }

  onNuiLogTrackCallback(level: Constants.LogLevel, log: string): void {
  }
}

The example sends audio proactively, so onNuiNeedAudioData returns 0. asrResult can contain a task_id for troubleshooting. A nonempty allResponse contains the complete JSON response. For EVENT_ASR_ERROR, check the result code, error details, and task ID. For EVENT_MIC_ERROR, check recording permissions, recording code, and whether another application is using the recorder. EVENT_DIALOG_EX requires no handling.

Stop recognition

function stopRecognizer(nui: NativeNui): number {
  return nui.stopDialog();
}

To cancel immediately without waiting for the final result, call cancelDialog() instead.

Release resources

function releaseRecognizer(nui: NativeNui): number {
  return nui.release();
}

Initialization and configuration

initialize(callback: INativeNuiCallback, parameters: string,
  level: number, save_log: boolean = false): number
setParams(params: string): number
GetVersion(): string

parameters and params are JSON strings. A lower level produces more logs. save_log is disabled by default; when enabled, debug_path specifies the log directory. GetVersion returns the SDK version string.

Recognition and resource management

startDialog(vad_mode: Constants.VadMode, dialog_params: string): number
updateAudio(data: ArrayBuffer, first_pack: boolean): number
stopDialog(): number
cancelDialog(): number
release(): number

startDialog starts recognition with TYPE_P2T; dialog_params contains temporary parameters for the session. updateAudio sends audio to the SDK. stopDialog requests the final result and ends recognition. cancelDialog ends the task immediately without waiting for the final result. release releases SDK resources.

Is OPUS audio supported?

The SDK supports OPUS data for short sentence recognition and flash recognition. Real-time speech recognition supports only PCM audio with 16-bit samples and one channel. For details, see the API reference.

Can audio be supplied through callbacks?

Yes. If shared data and locking ensure that audio is produced and consumed in order, set nui_config.enable_callback_recording to true in the nui.json resource file. Supply recorded audio in onNuiNeedAudioData instead of proactively sending it with updateAudio.

How do real-time and short sentence recognition calls differ?

How do I handle an unresponsive application caused by SDK call timeouts?

Avoid synchronous SDK calls on the UI thread. For thread scheduling, see OneSentenceAsrWorker.ets in the SDK package.