HarmonyOS Next SDK
This topic describes how to use the HarmonyOS Next NUI SDK for flash recognition, including SDK integration, the call sequence, code examples, APIs, and callbacks.
Prerequisites
Review the parameters and limits for flash recognition. For more information, see Flash recognition mobile SDK reference.
Obtain a project Appkey. For more information, see Manage projects.
Obtain a valid NLS token (Access Token). For more information, see Obtain a token.
Obtain the NLS token on your application server and send it to the client. Do not embed or store an AccessKey ID or AccessKey secret on the client, or log tokens or initialization parameters that contain a token.
Download and integrate the SDK
Download harmony_neonui_sdk.tar.gz.
Extract the package and import
entry/libs/neonui.harinto your application. The package also provides C++ dynamic libraries and header files innative/libsandnative/include, respectively.Open the sample project in DevEco Studio. Set the Appkey and token in
entry/src/main/ets/Utils/UserKey.ets, then runentry/src/main/ets/pages/FileTranscriberPage.ets.
Call sequence
Create an SDK instance and implement
INativeFileTransCallback.Call
initializeFileTrans, then callsetParamsas needed.Specify the audio file path and encoding format, then call
startFileTranscriber.Receive the final result in the
EVENT_FILE_TRANS_RESULTcallback.Call
releasewhen the SDK is no longer needed.
SDK APIs are synchronous. Do not call them on the UI thread, where they can block the interface. To avoid deadlocks, do not call SDK APIs directly from SDK event callbacks. Schedule subsequent SDK operations outside the callback.
Code examples
The following snippets show SDK calls, not a standalone application page. For the complete page, see FileTranscriberPage.ets in the SDK package. The application wrapper is entry/src/main/ets/neonui/FileTranscriber.ets. Use resourceDir + "/resources_cloud" from the application context as the resource directory and a writable filesDir as the debug directory.
The examples create an instance with the constructor. You can also obtain the singleton by calling NativeNui.GetInstance(). Release an instance before initializing it again.
Initialize the SDK
Import the SDK types. The callback parameter is the file recognition callback implementation below. The application provides the other parameters.
import { NativeNui, Constants, INativeFileTransCallback, AsrResult } from 'neonui';
function createFileRecognizer(callback: INativeFileTransCallback, resourceDir: string,
filesDir: string, appkey: string, token: string, deviceId: string): NativeNui {
const nui = new NativeNui(Constants.ModeType.MODE_DIALOG, 'file');
const parameters: Record<string, string> = {
app_key: appkey,
token: token,
device_id: deviceId,
url: 'https://nls-gateway.cn-shanghai.aliyuncs.com/stream/v1/FlashRecognizer',
workspace: resourceDir + '/resources_cloud',
debug_path: filesDir,
service_mode: Constants.ModeFullCloud
};
const ret = nui.initializeFileTrans(callback, JSON.stringify(parameters),
Constants.LogLevel.LOG_LEVEL_VERBOSE, false);
if (ret !== Constants.NuiResultCode.SUCCESS) {
throw new Error('initializeFileTrans failed: ' + ret);
}
return nui;
}
Provide app_key, token, a unique device_id, and a readable and writable workspace. debug_path specifies the debug file directory. To save debug audio, configure save_wav, enable save_log, and ensure that the debug directory is valid and writable.
Configure recognition
function configureFileRecognizer(nui: NativeNui): number {
const nlsConfig: Record<string, boolean> = { enable_intermediate_result: true };
const parameters: Record<string, Object> = { nls_config: nlsConfig };
return nui.setParams(JSON.stringify(parameters));
}
Start recognition
The example uses entry/src/main/resources/resfile/2.wav from the project. Pass resourceDir from the application context to the function. format must match the actual file encoding. The task ID buffer must contain at least 33 bytes.
function startFileRecognition(nui: NativeNui, resourceDir: string): ArrayBuffer {
const nlsConfig: Record<string, string> = { format: 'wav' };
const parameters: Record<string, Object> = {
file_path: resourceDir + '/2.wav',
nls_config: nlsConfig
};
const taskId = new ArrayBuffer(33);
const ret = nui.startFileTranscriber(JSON.stringify(parameters), taskId);
if (ret !== Constants.NuiResultCode.SUCCESS) {
throw new Error('startFileTranscriber failed: ' + ret);
}
return taskId;
}
To switch projects at runtime, set app_key in the task parameters.
Handle callbacks
Recognition results are in flash_result.sentences, with each sentence’s text in text. The example below caches results for multiple files by task ID.
interface RecognizedSentence {
text: string;
}
interface FlashResult {
sentences: RecognizedSentence[];
}
interface FileRecognitionResponse {
flash_result: FlashResult;
}
class FileCallbackHandle implements INativeFileTransCallback {
taskResults: Map<string, string> = new Map();
combinedResults: string = '';
onFileTransEventCallback(event: Constants.NuiEvent, resultCode: number,
finish: number, asrResult: AsrResult, taskId: string): void {
if (event === Constants.NuiEvent.EVENT_FILE_TRANS_UPLOADED) {
console.info('file uploaded: ' + taskId);
} else if (event === Constants.NuiEvent.EVENT_ASR_ERROR) {
console.error('recognition error: ' + resultCode + ', task: ' + taskId);
} else if (event === Constants.NuiEvent.EVENT_FILE_TRANS_RESULT &&
asrResult && asrResult.asrResult) {
const response = JSON.parse(asrResult.asrResult) as FileRecognitionResponse;
if (response.flash_result) {
let text = '';
response.flash_result.sentences.forEach((sentence: RecognizedSentence) => {
text += sentence.text;
});
this.taskResults.set(taskId, text);
this.combinedResults = '';
this.taskResults.forEach((value: string, key: string) => {
this.combinedResults += key + ': ' + value + '\n';
});
}
}
}
}
Release resources
function releaseFileRecognizer(nui: NativeNui): number {
return nui.release();
}
Key APIs and parameters
For parameter descriptions and conditions, see the API reference. For return codes and troubleshooting, see the Error code reference.
Initialization and configuration
initializeFileTrans(callback: INativeFileTransCallback, parameters: string,
level: number, save_log: boolean = false): number
setParams(params: string): number
parameters and params are JSON strings. A lower level produces more logs. save_log is disabled by default. Call setParams after initialization succeeds and before starting a task.
Recognition and resource management
startFileTranscriber(params: string, task_id: ArrayBuffer): number
cancelFileTranscriber(task_id: string): number
release(): number
The params argument of startFileTranscriber contains the audio file path and other parameters. task_id receives the task ID generated by the SDK. Allocate at least 33 bytes for this buffer. cancelFileTranscriber takes the task ID as a string. release releases SDK resources.
Callbacks and events
onFileTransEventCallback(event: Constants.NuiEvent, resultCode: number,
finish: number, asrResult: AsrResult, taskId: string): void
event is the task event. resultCode provides the error code for EVENT_ASR_ERROR. The third parameter is reserved. asrResult contains the recognition result, and taskId is the task ID.
|
Event |
Description |
|
EVENT_FILE_TRANS_CONNECTED |
Connected to the recognition service. |
|
EVENT_FILE_TRANS_UPLOADED |
File uploaded. |
|
EVENT_FILE_TRANS_RESULT |
Final recognition result returned. |
|
EVENT_ASR_ERROR |
Recognition error. Check the error code. |
FAQ
How do I query task status by task ID?
The SDK does not provide an API to query status by task ID. Use onFileTransEventCallback to receive connection, upload, result, and error events for each task.
Is OPUS audio supported?
The SDK supports OPUS data for short sentence recognition and flash recognition. Real-time speech recognition supports only PCM audio with 16-bit samples and one channel. For details, see the API reference.
How do I handle an unresponsive application caused by SDK call timeouts?
Avoid synchronous SDK calls on the UI thread. For thread scheduling, see OneSentenceAsrWorker.ets in the SDK package.