HarmonyOS Next SDK

Updated at:

This topic describes how to use the HarmonyOS Next NUI SDK for flash recognition, including SDK integration, the call sequence, code examples, APIs, and callbacks.

Prerequisites

Important

Obtain the NLS token on your application server and send it to the client. Do not embed or store an AccessKey ID or AccessKey secret on the client, or log tokens or initialization parameters that contain a token.

Download and integrate the SDK

  1. Download harmony_neonui_sdk.tar.gz.

  2. Extract the package and import entry/libs/neonui.har into your application. The package also provides C++ dynamic libraries and header files in native/libs and native/include, respectively.

  3. Open the sample project in DevEco Studio. Set the Appkey and token in entry/src/main/ets/Utils/UserKey.ets, then run entry/src/main/ets/pages/FileTranscriberPage.ets.

Call sequence

  1. Create an SDK instance and implement INativeFileTransCallback.

  2. Call initializeFileTrans, then call setParams as needed.

  3. Specify the audio file path and encoding format, then call startFileTranscriber.

  4. Receive the final result in the EVENT_FILE_TRANS_RESULT callback.

  5. Call release when the SDK is no longer needed.

Important

SDK APIs are synchronous. Do not call them on the UI thread, where they can block the interface. To avoid deadlocks, do not call SDK APIs directly from SDK event callbacks. Schedule subsequent SDK operations outside the callback.

Code examples

The following snippets show SDK calls, not a standalone application page. For the complete page, see FileTranscriberPage.ets in the SDK package. The application wrapper is entry/src/main/ets/neonui/FileTranscriber.ets. Use resourceDir + "/resources_cloud" from the application context as the resource directory and a writable filesDir as the debug directory.

The examples create an instance with the constructor. You can also obtain the singleton by calling NativeNui.GetInstance(). Release an instance before initializing it again.

Initialize the SDK

Import the SDK types. The callback parameter is the file recognition callback implementation below. The application provides the other parameters.

import { NativeNui, Constants, INativeFileTransCallback, AsrResult } from 'neonui';

function createFileRecognizer(callback: INativeFileTransCallback, resourceDir: string,
  filesDir: string, appkey: string, token: string, deviceId: string): NativeNui {
  const nui = new NativeNui(Constants.ModeType.MODE_DIALOG, 'file');
  const parameters: Record<string, string> = {
    app_key: appkey,
    token: token,
    device_id: deviceId,
    url: 'https://nls-gateway.cn-shanghai.aliyuncs.com/stream/v1/FlashRecognizer',
    workspace: resourceDir + '/resources_cloud',
    debug_path: filesDir,
    service_mode: Constants.ModeFullCloud
  };
  const ret = nui.initializeFileTrans(callback, JSON.stringify(parameters),
    Constants.LogLevel.LOG_LEVEL_VERBOSE, false);
  if (ret !== Constants.NuiResultCode.SUCCESS) {
    throw new Error('initializeFileTrans failed: ' + ret);
  }
  return nui;
}

Provide app_key, token, a unique device_id, and a readable and writable workspace. debug_path specifies the debug file directory. To save debug audio, configure save_wav, enable save_log, and ensure that the debug directory is valid and writable.

Configure recognition

function configureFileRecognizer(nui: NativeNui): number {
  const nlsConfig: Record<string, boolean> = { enable_intermediate_result: true };
  const parameters: Record<string, Object> = { nls_config: nlsConfig };
  return nui.setParams(JSON.stringify(parameters));
}

Start recognition

The example uses entry/src/main/resources/resfile/2.wav from the project. Pass resourceDir from the application context to the function. format must match the actual file encoding. The task ID buffer must contain at least 33 bytes.

function startFileRecognition(nui: NativeNui, resourceDir: string): ArrayBuffer {
  const nlsConfig: Record<string, string> = { format: 'wav' };
  const parameters: Record<string, Object> = {
    file_path: resourceDir + '/2.wav',
    nls_config: nlsConfig
  };
  const taskId = new ArrayBuffer(33);
  const ret = nui.startFileTranscriber(JSON.stringify(parameters), taskId);
  if (ret !== Constants.NuiResultCode.SUCCESS) {
    throw new Error('startFileTranscriber failed: ' + ret);
  }
  return taskId;
}

To switch projects at runtime, set app_key in the task parameters.

Handle callbacks

Recognition results are in flash_result.sentences, with each sentence’s text in text. The example below caches results for multiple files by task ID.

interface RecognizedSentence {
  text: string;
}

interface FlashResult {
  sentences: RecognizedSentence[];
}

interface FileRecognitionResponse {
  flash_result: FlashResult;
}

class FileCallbackHandle implements INativeFileTransCallback {
  taskResults: Map<string, string> = new Map();
  combinedResults: string = '';

  onFileTransEventCallback(event: Constants.NuiEvent, resultCode: number,
    finish: number, asrResult: AsrResult, taskId: string): void {
    if (event === Constants.NuiEvent.EVENT_FILE_TRANS_UPLOADED) {
      console.info('file uploaded: ' + taskId);
    } else if (event === Constants.NuiEvent.EVENT_ASR_ERROR) {
      console.error('recognition error: ' + resultCode + ', task: ' + taskId);
    } else if (event === Constants.NuiEvent.EVENT_FILE_TRANS_RESULT &&
      asrResult && asrResult.asrResult) {
      const response = JSON.parse(asrResult.asrResult) as FileRecognitionResponse;
      if (response.flash_result) {
        let text = '';
        response.flash_result.sentences.forEach((sentence: RecognizedSentence) => {
          text += sentence.text;
        });
        this.taskResults.set(taskId, text);
        this.combinedResults = '';
        this.taskResults.forEach((value: string, key: string) => {
          this.combinedResults += key + ': ' + value + '\n';
        });
      }
    }
  }
}

Release resources

function releaseFileRecognizer(nui: NativeNui): number {
  return nui.release();
}

Key APIs and parameters

For parameter descriptions and conditions, see the API reference. For return codes and troubleshooting, see the Error code reference.

Initialization and configuration

initializeFileTrans(callback: INativeFileTransCallback, parameters: string,
  level: number, save_log: boolean = false): number
setParams(params: string): number

parameters and params are JSON strings. A lower level produces more logs. save_log is disabled by default. Call setParams after initialization succeeds and before starting a task.

Recognition and resource management

startFileTranscriber(params: string, task_id: ArrayBuffer): number
cancelFileTranscriber(task_id: string): number
release(): number

The params argument of startFileTranscriber contains the audio file path and other parameters. task_id receives the task ID generated by the SDK. Allocate at least 33 bytes for this buffer. cancelFileTranscriber takes the task ID as a string. release releases SDK resources.

Callbacks and events

onFileTransEventCallback(event: Constants.NuiEvent, resultCode: number,
  finish: number, asrResult: AsrResult, taskId: string): void

event is the task event. resultCode provides the error code for EVENT_ASR_ERROR. The third parameter is reserved. asrResult contains the recognition result, and taskId is the task ID.

Event

Description

EVENT_FILE_TRANS_CONNECTED

Connected to the recognition service.

EVENT_FILE_TRANS_UPLOADED

File uploaded.

EVENT_FILE_TRANS_RESULT

Final recognition result returned.

EVENT_ASR_ERROR

Recognition error. Check the error code.

FAQ

How do I query task status by task ID?

The SDK does not provide an API to query status by task ID. Use onFileTransEventCallback to receive connection, upload, result, and error events for each task.

Is OPUS audio supported?

The SDK supports OPUS data for short sentence recognition and flash recognition. Real-time speech recognition supports only PCM audio with 16-bit samples and one channel. For details, see the API reference.

How do I handle an unresponsive application caused by SDK call timeouts?

Avoid synchronous SDK calls on the UI thread. For thread scheduling, see OneSentenceAsrWorker.ets in the SDK package.