Stream ingest on Android mobile clients

Prerequisites

• Create a real-time recording and obtain an ingest URL

• Install the Android Push SDK

SDK key interfaces

• initialize: Initializes the SDK.

  /**
   * Initializes the SDK. The SDK is a singleton. Release the SDK before you initialize it again. Do not call this method in the UI thread because it may cause blocking.
   * @param callback The event listener callback. For more information, see the callbacks described later in this topic.
   * @param parameters The initialization parameters in a JSON string. For more information, see the following table.
   * @param level The log level. The smaller the value, the more logs are printed.
   * @param save_log Specifies whether to save logs to a file. The log file is saved to the directory specified by the debug_path field in the ticket. Note: Log files have no size limit. Continuous storage may fill up the disk.
   * @return For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int initialize(final INativeNuiCallback callback,
                                     String parameters,
                                     final Constants.LogLevel level,
                                     final boolean save_log)

  The parameters are described as follows:

  Parameter	Type	Required	Description
  workspace	String	Yes	The path of the working directory. The SDK reads configuration files from this path.
  app_key	String	Yes	Set this parameter to "default".
  token	String	Yes	Set this parameter to "default".
  url	String	Yes	The MeetingJoinUrl returned when you create a Tingwu real-time recording task. This URL is used as the ingest URL for the audio stream during real-time audio stream recognition.
  service_mode	String	Yes	Set this parameter to "1" to enable the online feature.
  device_id	String	Yes	The device ID. This ID uniquely identifies a device, such as a MAC address, serial number (SN), or UniquePseudoID.
  debug_path	String	No	The debug directory. If the save_log parameter is set to true during SDK initialization, this directory is used to save log files.
  save_wav	String	No	This parameter takes effect only when the save_log parameter is set to true during SDK initialization. It specifies whether to save audio for debugging. The data is saved in the debug directory. Make sure the path specified by debug_path is valid and writable. Note: Audio files have no size limit. Continuous storage may fill up the disk.

  The INativeNuiCallback type includes the following callbacks.

  • onNuiAudioStateChanged: Enables or disables the recording feature based on the audio state.

    /**
     * When you call methods such as start, stop, or cancel, the SDK uses this callback to notify the application to start or stop recording.
     * @param state The required state for recording (on/off).
     */
    void onNuiAudioStateChanged(AudioState state);

  • onNuiNeedAudioData: Provides audio data in the callback.

    /**
     * When recognition starts, this callback is invoked repeatedly. The application must fill in the audio data in the callback.
     * @param buffer The storage area for filling in audio.
     * @param len The number of bytes of audio to fill.
     * @return The number of bytes actually filled.
     */
    int onNuiNeedAudioData(byte[] buffer, int len);

  • onNuiEventCallback: The SDK event callback.

    /**
     * The main event callback of the SDK.
     * @param event The callback event. For more information, see the following event list.
     * @param resultCode For more information about error codes, see the error codes. This is valid when an EVENT_ASR_ERROR event occurs.
     * @param arg2 A reserved parameter.
     * @param kwsResult The voice wakeup feature (not currently supported).
     * @param asrResult The speech recognition result.
     */
    void onNuiEventCallback(NuiEvent event, final int resultCode, final int arg2, KwsResult kwsResult, AsrResult asrResult);

    Event list:

    Name	Description
    EVENT_VAD_START	The start of a human voice is detected.
    EVENT_VAD_END	The end of a human voice is detected.
    EVENT_ASR_PARTIAL_RESULT	The intermediate result of speech recognition.
    EVENT_ASR_ERROR	Determine the cause of the error based on the error code information.
    EVENT_MIC_ERROR	A recording error occurred. This indicates that the SDK has not received any audio for 2 consecutive seconds. Check if the recording system is working correctly.
    EVENT_SENTENCE_START	A real-time speech recognition event. This indicates that the start of a sentence is detected.
    EVENT_SENTENCE_END	A real-time speech recognition event. This indicates that the end of a sentence is detected and a complete result is returned.
    EVENT_SENTENCE_SEMANTICS	Not in use.
    EVENT_RESULT_TRANSLATED	The translation result.
    EVENT_TRANSCRIBER_COMPLETE	The final event after speech recognition is stopped.

  • onNuiAudioRMSChanged: The audio energy value callback.

    /**
     * Receives the real-time audio energy value during operation.
     * @param val The audio data energy value callback. The value ranges from -160 to 0. This is typically used for displaying voice animations in the UI.
     */
    public void onNuiAudioRMSChanged(float val);

• setParams: Sets SDK parameters in JSON format.

  /**
   * Sets parameters in JSON format. This method must be called after initialize and before startDialog.
   * @param params For more information about the parameters, see the following table.
   * @return For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int setParams(String params);

  The following table describes the parameters.

  Parameter	Type	Required	Description
  service_type	Integer	Yes	Set this parameter to "4". This specifies the type of voice service to request. The value for Tingwu real-time stream ingest is "4".
  nls_config	JsonObject	Yes	The parameter settings for accessing the Voice Service. For more information, see the following parameters.
  nls_config.sr_format	String	Yes	Set this parameter to "pcm". When you create a Tingwu task, also set the encoding format of the audio stream data to pcm. For more information, see CreateTask.
  nls_config.sample_rate	Integer	Yes	The audio sample rate. Default value: 16000 Hz. When you create a Tingwu task, also specify the sample rate of the audio stream data. Supported values are 8000 and 16000. For more information, see CreateTask.

• startDialog: Starts recognition.

  /**
   * Starts recognition.
   * @param vad_mode Multiple modes are available. For recognition scenarios, use P2T.
   * @param dialog_params The dialog parameters in a JSON string. You can leave this empty by passing an empty JSON string.
   * @return For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int startDialog(VadMode vad_mode, String dialog_params);

• stopDialog: Stops recognition.

  /**
   * Stops recognition. After you call this method, the server returns the final recognition result and ends the task.
   * @return For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int stopDialog();

• cancelDialog: Immediately stops recognition.

  /**
   * Immediately stops recognition. After you call this method, the task ends immediately without waiting for the server to return the final recognition result.
   * @return For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int cancelDialog();

• release: Releases the SDK.

  /**
   * Releases SDK resources.
   * @return For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int release();

• GetVersion: Retrieves the current SDK version information.

  /**
   * Gets the current SDK version information.
   * @return The SDK version information as a string.
   */
  public synchronized String GetVersion();

Calling steps

1. Initialize the SDK and the recording instance.

2. Set parameters as needed.

3. Call startDialog to start recognition.

4. Enable the recorder based on the onNuiAudioStateChanged audio state callback.

5. Provide recording data in the onNuiNeedAudioData callback.

6. The EVENT_SENTENCE_START event callback indicates that the recognition of a sentence has started. You can obtain the intermediate recognition result from the EVENT_ASR_PARTIAL_RESULT event callback, the complete recognition result and related information for the sentence from the EVENT_SENTENCE_END event callback, and the translation result from the EVENT_RESULT_TRANSLATED event callback.

7. Call stopDialog to stop recognition. You can confirm that recognition has stopped using the EVENT_TRANSCRIBER_COMPLETE event callback.

8. After the call ends, use the release method to release SDK resources.

Proguard configuration

If your code uses obfuscation, add the following configuration to proguard-rules.pro:

-keep class com.alibaba.idst.nui.*{*;}

Code example

NUI SDK initialization

// Obtain the resource path, which is the working directory.
//  The working directory is created internally using context.getApplicationContext().getFilesDir().toString() + "/asr_my".
//  For example, /data/user/0/mit.alibaba.nuidemo/files/asr_my.
String workspace = CommonUtils.getModelPath(this);

// Create the debug path.
String debug_path = getExternalCacheDir().getAbsolutePath() + "/debug_" + System.currentTimeMillis();
Utils.createDir(debug_path);

// Copy asset resources from nuisdk.aar to the workspace.
CommonUtils.copyAssetsData(this);

// Initialize the SDK. You must enter the relevant ID information in genInitParams before you can use it.
NativeNui nui_instance = new NativeNui();
int ret = nui_instance.initialize(this, genInitParams(asset_path,debug_path), Constants.LogLevel.LOG_LEVEL_VERBOSE, true);

The genInitParams method generates a JSON string that contains the resource directory and user information. The user information includes the following fields.

private String genInitParams(String workpath, String debugpath) {
    String str = "";
    try{
        // 1. Interfaces and implementation: https://help.aliyun.com/zh/tingwu/interface-and-implementation
        //    Follow the steps in the document to first create an AccessKey and a project.
        //    Then, you need to call the CreateTask operation on your server to create a real-time recording and obtain the MeetingJoinUrl.
        //    This MeetingJoinUrl is the URL below.
        JSONObject object = new JSONObject();

        // Account and project creation
        //  For information about how to obtain an AccessKey ID and AccessKey secret, see https://help.aliyun.com/document_detail/72138.html.
        object.put("app_key", "default"); // Required
        object.put("token", "default"); // Required

        // Fill in the url with the generated MeetingJoinUrl.
        // Because the MeetingJoinUrl generation process involves an AccessKey ID and AccessKey secret, account information cannot be stored on the mobile client. Therefore, it must be generated on the server and sent to the mobile client.
        // For more information, see https://help.aliyun.com/zh/tingwu/api-tingwu-2023-09-30-createtask.
        object.put("url", "wss://tingwu-realtime-cn-hangzhou-pre.aliyuncs.com/api/ws/v1?xxxx"); // Required

        // The path of the working directory. The SDK reads configuration files from this path.
        object.put("workspace", workpath); // Required, and read and write permissions are needed.
        object.put("device_id", Utils.getDeviceId()); // Required. We recommend that you enter a unique ID to facilitate troubleshooting.
        // This parameter takes effect only when the save_log parameter is set to true during SDK initialization. It specifies whether to save audio for debugging. The data is saved in the debug directory. Make sure the path specified by debug_path is valid and writable.
//        object.put("save_wav", "true");
        // The debug directory. If the save_log parameter is set to true during SDK initialization, this directory is used to save intermediate audio files.
        object.put("debug_path", debugpath);

        object.put("service_mode", Constants.ModeFullCloud); // Required

        str = object.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }

    // Note: The str string contains sensitive information such as ak_id, ak_secret, token, and app_key. Do not output this information in logs in actual products.
    Log.i(TAG, "InsideUserContext:" + str);
    return str;
}

Parameter settings

Set the parameters in a JSON string.

// Set recognition-related parameters. For more information, see the API documentation.
//  Call this after initialize() and before startDialog().
nui_instance.setParams(genParams());

private String genParams() {
    String params = "";
    try {
        JSONObject nls_config = new JSONObject();
        nls_config.put("sample_rate", 16000);
        nls_config.put("sr_format", "pcm");
        JSONObject tmp = new JSONObject();
        tmp.put("nls_config", nls_config);
        tmp.put("service_type", Constants.kServiceTypeSpeechTranscriber); // Required
        params = tmp.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    return params;
}

Start recognition

Use the startDialog method to start the listener.

nui_instance.startDialog(Constants.VadMode.TYPE_P2T, genDialogParams());

private String genDialogParams() {
    String params = "";
    try {
        JSONObject dialog_param = new JSONObject();
        params = dialog_param.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    return params;
}

Callback handling

• onNuiAudioStateChanged: The recording state callback. The SDK internally maintains the recording state. You can use the state from this callback to control the recorder.

  public void onNuiAudioStateChanged(Constants.AudioState state) {
      Log.i(TAG, "onNuiAudioStateChanged");
      if (state == Constants.AudioState.STATE_OPEN) {
          Log.i(TAG, "audio recorder start");
          mAudioRecorder.startRecording();
      } else if (state == Constants.AudioState.STATE_CLOSE) {
          Log.i(TAG, "audio recorder close");
          mAudioRecorder.release();
      } else if (state == Constants.AudioState.STATE_PAUSE) {
          Log.i(TAG, "audio recorder pause");
          mAudioRecorder.stop();
      }
  }

• onNuiNeedAudioData: The recording data callback. You can fill in the recording data in this callback.

  public int onNuiNeedAudioData(byte[] buffer, int len) {
      int ret = 0;
      if (mAudioRecorder.getState() != AudioRecord.STATE_INITIALIZED) {
          Log.e(TAG, "audio recorder not init");
          return -1;
      }
      ret = mAudioRecorder.read(buffer, 0, len);
      // The return value tells the SDK how much data was read.
      // If a value less than 0 is returned, an error occurred.
      // If 0 is returned, it means there is no recording data. If 0 is returned for 2 consecutive seconds, the EVENT_MIC_ERROR event is triggered.
      return ret;
  }

• onNuiEventCallback: The NUI SDK event callback. Do not call SDK methods in the event callback to prevent deadlocks.

  public void onNuiEventCallback(Constants.NuiEvent event, final int resultCode, final int arg2, KwsResult kwsResult, AsrResult asrResult) {
      Log.i(TAG, "event=" + event + " resultCode=" + resultCode);
      // asrResult contains a task_id. The task_id helps with troubleshooting. Record and save it.
      //
      // The new version adds asrResult.allResponse. If it is not null and not empty, it provides complete information in a JSON string.
      if (event == Constants.NuiEvent.EVENT_TRANSCRIBER_COMPLETE) {
        // Real-time recognition ends.
      } else if (event == Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT) {
        // For example, display the intermediate recognition result of the current sentence.
        showText(asrView, asrResult.asrResult);
      } else if (event == Constants.NuiEvent.EVENT_SENTENCE_END) {
        // For example, display the complete recognition result of the current sentence.
        showText(asrView, asrResult.asrResult);
      } else if (event == Constants.NuiEvent.EVENT_RESULT_TRANSLATED) {
        // For example, display the translation result of the current sentence.
        showText(asrView, asrResult.asrResult);
      } else if (event == Constants.NuiEvent.EVENT_ASR_ERROR) {
        // In EVENT_ASR_ERROR, asrResult contains error information. Using it with the resultCode error code and the task_id makes troubleshooting easier. Record and save this information.
      } else if (event == Constants.NuiEvent.EVENT_MIC_ERROR) {
        // EVENT_MIC_ERROR indicates that no audio data has been received for 2 seconds. Check the recording-related code and permissions, or check whether the recording module is used by another application.
      } else if (event == Constants.NuiEvent.EVENT_DIALOG_EX) { /* unused */
        // You can ignore this event.
    }
  }

End recognition

nui_instance.stopDialog();

Release the SDK

nui_instance.release();