Android SDK

Prerequisites

• Before using the SDK, read the API Reference.

• Obtain a project AppKey. For details, see Create a project.

• Obtain an access token.

Download and installation

1. Select and download mobile SDKs.

   Important

   To run the sample, replace the placeholder Alibaba Cloud account information, AppKey, and token in the initialization code.

2. Unzip the ZIP package. The app/libs directory contains the SDK package in AAR format. Integrate this AAR package as a dependency in your project. If you need to use the Android C++ interface, the dynamic library and header files are in the android_libs and android_include directories of the ZIP package.

3. Open the project in Android Studio to view the reference implementation. The code sample for Short Speech Recognition is in the SpeechRecognizerActivity.java file. You can run it directly after replacing the AppKey and token.

Key SDK interfaces

• initialize: Initializes the SDK.

  /**
   * Initializes the SDK. The SDK uses a singleton pattern. You must release the previous instance before re-initializing.
   * Do not call this method on the UI thread, because it may cause the thread to block.
   * @param callback The event listener callback. See the callbacks below for details.
   * @param parameters Initialization parameters in a JSON string format. See the description below or the API Reference: https://help.aliyun.com/document_detail/173298.html.
   * @param level The logging level. A lower value results in more detailed logs.
   * @param save_log Specifies whether to save logs to a file. The log directory is specified by the debug_path field in the ticket. Note that log files have no size limit, so be aware that continuous storage can fill up the disk.
   * @return See Error Codes for details: https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int initialize(final INativeNuiCallback callback,
                                     String parameters,
                                     final Constants.LogLevel level,
                                     final boolean save_log)

  The INativeNuiCallback type includes the following callbacks.

  • onNuiAudioStateChanged: Toggles the recording function based on the audio state.

    /**
     * When interfaces like start, stop, or cancel are called, the SDK notifies the app to start or stop recording through this callback.
     * @param state The required recording state (open/close).
     */
    void onNuiAudioStateChanged(AudioState state);

  • onNuiNeedAudioData: Called repeatedly to request audio data from the app.

    /**
     * When recognition starts, this callback is called repeatedly. The app needs to provide speech data in this callback.
     * @param buffer The buffer to fill with speech data.
     * @param len The number of bytes of speech data to fill.
     * @return The actual number of bytes filled.
     */
    int onNuiNeedAudioData(byte[] buffer, int len);

  • onNuiEventCallback: SDK event callback.

    /**
     * The main event callback for the SDK.
     * @param event The callback event. See the event list below.
     * @param resultCode See the error codes. This is valid when an EVENT_ASR_ERROR event occurs.
     * @param arg2 Reserved parameter.
     * @param kwsResult Wake word result. This feature is not currently supported.
     * @param asrResult Speech recognition result.
     */
    void onNuiEventCallback(NuiEvent event, final int resultCode, final int arg2, KwsResult kwsResult, AsrResult asrResult);

  • onNuiAudioRMSChanged: Audio energy value callback.

    /**
     * Audio energy value callback.
     * @param val The energy level of the audio data, ranging from -160 to 0. This is typically used to display voice activity animations in the UI.
     */
    public void onNuiAudioRMSChanged(float val);

    Event list:

    Parameter	Description
    EVENT_VAD_START	The start of speech is detected.
    EVENT_VAD_END	The end of speech is detected.
    EVENT_ASR_PARTIAL_RESULT	A partial result is available.
    EVENT_ASR_RESULT	The final result is available.
    EVENT_ASR_ERROR	An error occurred during recognition. Check the error code for details.
    EVENT_MIC_ERROR	Microphone error. This event is triggered when the SDK has not received audio data for two consecutive seconds. Check if your recording system is functioning correctly.

• setParams: Sets SDK parameters in JSON format.

  /**
   * Sets parameters in JSON format.
   * @param params For details, see the API Reference: https://help.aliyun.com/document_detail/173298.html.
   * @return See Error Codes for details: https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int setParams(String params);

• startDialog: Starts recognition.

  /**
   * Starts recognition.
   * @param vad_mode Multiple modes are available. For recognition scenarios, use P2T.
   * @param dialog_params Dialog parameters in a JSON string format. For details, see the API Reference: https://help.aliyun.com/document_detail/173298.html.
   * @return See Error Codes for details: https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int startDialog(VadMode vad_mode, String dialog_params);

• stopDialog: Ends recognition.

  /**
   * Ends recognition. After calling this interface, the server returns the final recognition result and ends the task.
   * @return See Error Codes for details: https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int stopDialog();

• cancelDialog: Ends recognition immediately.

  /**
   * Ends recognition immediately. After calling this interface, the task ends without waiting for the server to return the final recognition result.
   * @return See Error Codes for details: https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int cancelDialog();

• release: Releases the SDK.

  /**
   * Releases SDK resources.
   * @return See Error Codes for details: https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int release();

• GetVersion: Gets the current SDK version information.

  /**
   * Gets the current SDK version information.
   * @return The SDK version information as a string.
   */
  public synchronized String GetVersion();

Procedure

1. Initialize the SDK and the audio recorder instance.

2. Set parameters based on your business requirements.

3. Call startDialog to start recognition.

4. Implement the onNuiAudioStateChanged callback to start or stop the audio recorder based on the received state.

5. Provide audio data in the onNuiNeedAudioData callback.

6. Get partial results from the EVENT_ASR_PARTIAL_RESULT event callback.

7. Call stopDialog to end recognition. The final result is delivered in the EVENT_ASR_RESULT event callback.

8. When you are finished, call the release() method to free SDK resources.

Proguard configuration

If your code uses obfuscation, add the following line to your proguard-rules.pro file:

-keep class com.alibaba.idst.nui.*{*;}

Code samples

NUI SDK initialization

// Get the resource path, which is the working directory.
//  The working directory is created internally with context.getApplicationContext().getFilesDir().toString() + "/asr_my".
//  For example: /data/user/0/mit.alibaba.nuidemo/files/asr_my
String asset_path = CommonUtils.getModelPath(this);
// Create the debug path.
String debug_path = getExternalCacheDir().getAbsolutePath() + "/debug_" + System.currentTimeMillis();
Utils.createDir(debug_path);
// Copy assets from nuisdk.aar to the workspace.
CommonUtils.copyAssetsData(this);
// Initialize the SDK. You must fill in your ID information in genInitParams to use the SDK.
NativeNui nui_instance = new NativeNui();
int ret = nui_instance.initialize(this, genInitParams(asset_path, debug_path), Constants.LogLevel.LOG_LEVEL_VERBOSE, true);

The genInitParams method generates a JSON string that contains the resource directory and user information. The user information includes the following fields.

private String genInitParams(String workpath, String debugpath) {
    String str = "";
    try{
        // Get account credentials.
        // The getTicket method in the sample project provides several possible approaches. Choose a secure method that fits your business needs.
        //
        // Note:
        // Before using the Intelligent Speech Interaction service, you must have an account and enable the required services.
        // For detailed steps, see: https://help.aliyun.com/zh/isi/getting-started/start-here
        //
        // Primary Account:
        // Your account credentials include an AccessKey ID (ak_id) and an AccessKey Secret (ak_secret).
        // To prevent security breaches and financial losses, never store these credentials in your app's code or on the client side.
        //
        // STS temporary credential:
        // Because distributing account credentials to a client is risky, Alibaba Cloud provides the Security Token Service (STS).
        // STS generates a temporary sts_ak_id, sts_ak_secret, and sts_token from your original AccessKey ID and AccessKey Secret.
        // The 'sts_' prefix is used to distinguish the temporary credential from the original account credentials.
        // What is STS: https://help.aliyun.com/zh/ram/product-overview/what-is-sts
        // STS SDK Overview: https://help.aliyun.com/zh/ram/developer-reference/sts-sdk-overview
        // STS Python SDK call example: https://help.aliyun.com/zh/ram/developer-reference/use-the-sts-openapi-example
        //
        // Credential Requirements:
        // To use an offline feature (such as offline speech synthesis or wake word), you must provide app_key, ak_id, and ak_secret, or app_key, sts_ak_id, sts_ak_secret, and sts_token.
        // To use an online feature (such as speech synthesis, real-time speech recognition, Short Speech Recognition, or Audio File Transcription), you only need to provide an app_key and a token.
        JSONObject object = Auth.getTicket(Auth.GetTicketMethod.GET_TOKEN_FROM_SERVER_FOR_ONLINE_FEATURES);
        if (!object.containsKey("token")) {
            Log.e(TAG, "Cannot get token!!!");
        }
        object.put("device_id", Utils.getDeviceId()); // Required. We recommend using a unique ID to help with troubleshooting.
        object.put("url", "wss://nls-gateway.cn-shanghai.aliyuncs.com:443/ws/v1"); // Default URL for the China (Shanghai) region.
        object.put("workspace", workpath); // Required. Must have read and write permissions.
        object.put("sample_rate", "16000");
        object.put("format", "opus");
        // This parameter takes effect when the save_log parameter in initialize() is true. It specifies whether to save debug audio. This data is saved in the debug directory, so ensure debug_path is valid and writable.
        //object.put("save_wav", "true");
        // The debug directory. When the save_log parameter in initialize() is true, this directory is used to save intermediate audio files.
        object.put("debug_path", debugpath);
      // AsrCloud = 4  // You can select this for online Short Speech Recognition.
        object.put("service_mode", Constants.ModeAsrCloud); // Required.
        str = object.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    Log.i(TAG, "InsideUserContext:" + str);
    return str;
}

Parameter settings

Set the parameters as a JSON string.

// Set recognition parameters. For details, see the API documentation.
// Call this method after initialize() and before startDialog().
nui_instance.setParams(genParams());
private String genParams() {
    String params = "";
    try {
        JSONObject nls_config = new JSONObject();
        nls_config.put("enable_intermediate_result", true);
        // Parameters can be configured based on your business needs.
        // For interface details, see: https://help.aliyun.com/document_detail/173298.html
        // See section "2. Start recognition".
        // The public SDK (version 01B) does not include a local VAD module. Only the SDK with the wake word feature (version 029) has a VAD module.
        // To use VAD mode, you must enable the online VAD mode by setting the nls_config parameter, as shown in genParams().
        //
        // Mode description:
        // To use Push-to-Talk (P2T) mode, where you press to start speaking and release to stop, do not enable enable_voice_detection.
        // To use VAD mode, which automatically detects when the user has finished speaking, enable enable_voice_detection.
        if (vadMode.get()) {
            nls_config.put("enable_voice_detection", true);
            nls_config.put("max_start_silence", 10000);
            nls_config.put("max_end_silence", 800);
        }
        //nls_config.put("enable_punctuation_prediction", true);
        //nls_config.put("enable_inverse_text_normalization", true);
        //nls_config.put("enable_voice_detection", true);
        //nls_config.put("customization_id", "test_id");
        //nls_config.put("vocabulary_id", "test_id");
        //nls_config.put("max_start_silence", 10000);
        //nls_config.put("max_end_silence", 800);
        //nls_config.put("sample_rate", 16000);
        //nls_config.put("sr_format", "opus");
        /* If a parameter is supported by the feature but not listed in the documentation, you can use the following method to set it. */
        //JSONObject extend_config = new JSONObject();
        //extend_config.put("custom_test", true);
        //nls_config.put("extend_config", extend_config);
        JSONObject parameters = new JSONObject();
        parameters.put("nls_config", nls_config);
        parameters.put("service_type", Constants.kServiceTypeASR); // Required.
        // If you have HttpDns, you can set it here.
        //parameters.put("direct_ip", Utils.getDirectIp());
        params = parameters.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    return params;
}

Starting recognition

Call the startDialog() method to start listening.

// By default, Constants.VadMode.TYPE_P2T is used.
// Constants.VadMode.TYPE_VAD is only supported in SDKs with offline features. To enable VAD, set the enable_voice_detection parameter.
NativeNui.GetInstance().startDialog(Constants.VadMode.TYPE_P2T, genDialogParams());
private String genDialogParams() {
    String params = "";
    try {
        JSONObject dialog_param = new JSONObject();
        // You can update parameters, especially an expired token, by calling startDialog during runtime.
        //dialog_param.put("token", "");
        params = dialog_param.toString();
    } catch (JSONException e) {
        e.printStackTrace();
    }
    return params;
}

Callback handling

• onNuiAudioStateChanged: Recording state callback. The SDK maintains the recording state internally. Start or stop the recorder based on this callback.

  public void onNuiAudioStateChanged(Constants.AudioState state) {
      Log.i(TAG, "onNuiAudioStateChanged");
      if (state == Constants.AudioState.STATE_OPEN) {
          Log.i(TAG, "audio recorder start");
          mAudioRecorder.startRecording();
      } else if (state == Constants.AudioState.STATE_CLOSE) {
          Log.i(TAG, "audio recorder close");
          mAudioRecorder.release();
      } else if (state == Constants.AudioState.STATE_PAUSE) {
          Log.i(TAG, "audio recorder pause");
          mAudioRecorder.stop();
      }
  }

• onNuiNeedAudioData: Recording data callback. Provide the recording data in this callback.

  public int onNuiNeedAudioData(byte[] buffer, int len) {
      int ret = 0;
      if (mAudioRecorder.getState() != AudioRecord.STATE_INITIALIZED) {
          Log.e(TAG, "audio recorder not init");
          return -1;
      }
      ret = mAudioRecorder.read(buffer, 0, len);
      // The return value tells the SDK how much data was read.
      // A value less than 0 indicates an error.
      // A value of 0 indicates no audio data. Returning 0 for two consecutive seconds triggers the EVENT_MIC_ERROR event.
      return ret;
  }

• onNuiEventCallback: NUI SDK event callback. Do not call SDK methods within an event callback, as this may cause a deadlock.

  public void onNuiEventCallback(Constants.NuiEvent event, final int resultCode, final int arg2, KwsResult kwsResult, AsrResult asrResult) {
      Log.i(TAG, "event=" + event + " resultCode=" + resultCode);
      // The asrResult contains a task_id. We recommend logging this ID to help with troubleshooting.
      //
      // The asrResult.allResponse field, if not null or empty, contains the complete server response as a JSON string.
      if (event == Constants.NuiEvent.EVENT_ASR_RESULT) {
          // For example, display the recognition result.
          showText(asrView, asrResult.asrResult);
      } else if (event == Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT) {
          // For example, display the partial recognition result.
          showText(asrView, asrResult.asrResult);
      } else if (event == Constants.NuiEvent.EVENT_ASR_ERROR) {
          // In EVENT_ASR_ERROR, asrResult contains the error message. Using it with the resultCode and task_id makes troubleshooting easier. Please log this information.
      } else if (event == Constants.NuiEvent.EVENT_MIC_ERROR) {
          // EVENT_MIC_ERROR indicates that no audio data has been received for 2 seconds. Check your recording-related code and permissions, or see if the recording module is being used by another application.
      } else if (event == Constants.NuiEvent.EVENT_DIALOG_EX) { /* unused */
          // You can ignore this event.
      }
  }

Ending recognition

nui_instance.stopDialog();

Releasing the SDK

nui_instance.release();

if (CommonUtils.copyAssetsData(this)) {
Log.i(TAG, "copy assets data done");
} else {
Log.i(TAG, "copy assets failed");
return;
}