Android SDK

Recording and network permissions

Declare network and recording permissions in AndroidManifest.xml. Obtain runtime recording permission before recording audio.

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />

ProGuard configuration

If code obfuscation is enabled, keep the NUI classes in proguard-rules.pro:

-keep class com.alibaba.idst.nui.* { *; }

Call flow

1. Initialize the SDK and recorder, and configure recognition parameters.

2. Call startDialog to start recognition. Start recording in onNuiAudioStateChanged when the state requires it.

3. Supply PCM audio in onNuiNeedAudioData and receive intermediate and sentence results in the event callback.

4. Call stopDialog to stop recognition and wait for EVENT_TRANSCRIBER_COMPLETE to confirm completion. Call cancelDialog if you do not need to wait for the final result.

5. Call release when the SDK is no longer needed.

Initialization

Import the SDK, recording, and JSON classes, and declare the fields used by the subsequent methods. The example uses 16 kHz, 16-bit mono PCM audio.

import android.media.AudioFormat;
import android.media.AudioRecord;
import android.media.MediaRecorder;
import android.util.Log;
import com.alibaba.idst.nui.AsrResult;
import com.alibaba.idst.nui.Constants;
import com.alibaba.idst.nui.INativeNuiCallback;
import com.alibaba.idst.nui.KwsResult;
import com.alibaba.idst.nui.NativeNui;
import org.json.JSONException;
import org.json.JSONObject;

private static final String TAG = "RealtimeRecognizer";
private static final int SAMPLE_RATE = 16000;
private final NativeNui nui_instance = new NativeNui();
private AudioRecord mAudioRecorder;

Set app_key to the project Appkey and token to the NLS token delivered by your server. Use Constants.ModeFullCloud for cloud recognition. In SDK 2.7.4, this mode does not require the local resource directory workspace.

private String genInitParams(String appkey, String token) throws JSONException {
    JSONObject parameters = new JSONObject();
    parameters.put("app_key", appkey);
    parameters.put("token", token);
    parameters.put("service_mode", Constants.ModeFullCloud);
    parameters.put("url", gatewayUrl());
    return parameters.toString();
}

private String gatewayUrl() {
    return "wss://nls-gateway.cn-shanghai.aliyuncs.com/ws/v1";
}

The following method creates the recorder and initializes the SDK. Obtain recording permission before calling it. Configure parameters and start a task only after it returns 0. You can add a unique device_id to the initialization parameters for troubleshooting.

private int initializeRecognizer(String appkey, String token) throws JSONException {
    int minBufferSize = AudioRecord.getMinBufferSize(SAMPLE_RATE,
            AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
    if (minBufferSize <= 0) {
        throw new IllegalStateException("Invalid recording configuration");
    }
    mAudioRecorder = new AudioRecord(MediaRecorder.AudioSource.MIC, SAMPLE_RATE,
            AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT,
            minBufferSize * 2);
    if (mAudioRecorder.getState() != AudioRecord.STATE_INITIALIZED) {
        mAudioRecorder.release();
        mAudioRecorder = null;
        throw new IllegalStateException("AudioRecord initialization failed");
    }

    int ret = nui_instance.initialize(this, genInitParams(appkey, token),
            Constants.LogLevel.LOG_LEVEL_ERROR, false);
    if (ret != 0) {
        mAudioRecorder.release();
        mAudioRecorder = null;
    }
    return ret;
}

Configure parameters

Call setParams after initialization succeeds and before starting recognition. Set service_type to Constants.kServiceTypeSpeechTranscriber and match sample_rate to the recording sample rate.

private int configureRecognizer() throws JSONException {
    JSONObject nls_config = new JSONObject();
    nls_config.put("enable_intermediate_result", true);
    nls_config.put("sample_rate", SAMPLE_RATE);
    nls_config.put("sr_format", "pcm");
    // Configure recognition options as needed:
    // nls_config.put("enable_punctuation_prediction", true);
    // nls_config.put("enable_inverse_text_normalization", true);
    // nls_config.put("max_sentence_silence", 800);
    // nls_config.put("enable_words", false);

    JSONObject parameters = new JSONObject();
    parameters.put("nls_config", nls_config);
    parameters.put("service_type", Constants.kServiceTypeSpeechTranscriber);
    return nui_instance.setParams(parameters.toString());
}

For initialization, recognition, and task parameter definitions, see Mobile SDK API reference. If a parameter is not listed but is supported by the service, you can set it in the extend_config object under nls_config. When using HttpDNS, you can set direct_ip as needed.

Start recognition

After configuration succeeds, start recognition in TYPE_P2T mode. After refreshing the token, you can pass the new valid token in the next startDialog call.

private int startRecognizer(String token) throws JSONException {
    JSONObject dialog_params = new JSONObject();
    dialog_params.put("token", token);
    return nui_instance.startDialog(Constants.VadMode.TYPE_P2T,
            dialog_params.toString());
}

Handle callbacks

Start, stop, or release AudioRecord in the recording state callback. Supply recorded audio to the SDK in the audio data callback.

@Override
public void onNuiAudioStateChanged(Constants.AudioState state) {
    if (mAudioRecorder == null) {
        return;
    }
    if (state == Constants.AudioState.STATE_OPEN) {
        mAudioRecorder.startRecording();
    } else if (state == Constants.AudioState.STATE_PAUSE) {
        if (mAudioRecorder.getRecordingState() == AudioRecord.RECORDSTATE_RECORDING) {
            mAudioRecorder.stop();
        }
    } else if (state == Constants.AudioState.STATE_CLOSE) {
        mAudioRecorder.release();
        mAudioRecorder = null;
    }
}

@Override
public int onNuiNeedAudioData(byte[] buffer, int len) {
    if (mAudioRecorder == null
            || mAudioRecorder.getState() != AudioRecord.STATE_INITIALIZED) {
        return -1;
    }
    return mAudioRecorder.read(buffer, 0, len);
}

In the event callback, asrResult.asrResult contains the recognition result. When non-empty, allResponse contains the full JSON response, including a task ID for troubleshooting. Replace the result and error handling in the following example with your application's UI and error handling logic.

@Override
public void onNuiEventCallback(Constants.NuiEvent event, int resultCode, int arg2,
        KwsResult kwsResult, AsrResult asrResult) {
    if (event == Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT
            || event == Constants.NuiEvent.EVENT_SENTENCE_END) {
        if (asrResult != null) {
            Log.i(TAG, "Recognition result: " + asrResult.asrResult);
        }
    } else if (event == Constants.NuiEvent.EVENT_TRANSCRIBER_COMPLETE) {
        Log.i(TAG, "Transcription completed");
    } else if (event == Constants.NuiEvent.EVENT_ASR_ERROR
            || event == Constants.NuiEvent.EVENT_MIC_ERROR) {
        Log.e(TAG, "Recognition error: " + resultCode);
    }
}

@Override
public void onNuiAudioRMSChanged(float val) {
    // Use the value for a volume animation if needed.
}

@Override
public void onNuiVprEventCallback(Constants.NuiVprEvent event) {
    // Voiceprint events are not used in this example.
}

Stop or cancel recognition

After calling stopDialog, wait for EVENT_TRANSCRIBER_COMPLETE to confirm completion. Continue receiving any remaining valid speech results through the sentence result callback. Stopping does not produce a new sentence result if there is no valid speech.

int ret = nui_instance.stopDialog();

Call nui_instance.cancelDialog() to cancel a task without waiting for the final server result.

Release resources

Release resources when the SDK is no longer needed. To use it again, recreate the recorder and initialize the SDK.

int ret = nui_instance.release();
if (mAudioRecorder != null) {
    mAudioRecorder.release();
    mAudioRecorder = null;
}

Initialization and configuration

initialize sets the callback, initialization parameters, log level, and whether to save log files. Pass parameters as a JSON string. Lower numeric log levels produce more detailed logs. To save logs, specify a writable debug_path. To also save audio for debugging, set save_wav. Clean up debug files to prevent ongoing storage use from filling the device.

public synchronized int initialize(INativeNuiCallback callback, String parameters,
        Constants.LogLevel level, boolean save_log);

setParams configures recognition parameters as a JSON string. Call it after initialization succeeds and before starting recognition.

public synchronized int setParams(String params);

Task control and resource management

startDialog starts recognition. For real-time recognition, use Constants.VadMode.TYPE_P2T and pass task parameters as a JSON string.

public synchronized int startDialog(Constants.VadMode vad_mode, String dialog_params);

stopDialog sends a request to stop recognition. Receive remaining results and the completion notification through the event callback.

public synchronized int stopDialog();

cancelDialog cancels a task without waiting for the final server recognition result.

public synchronized int cancelDialog();

release releases SDK resources.

public synchronized int release();

GetVersion returns the SDK version as a string.

public synchronized String GetVersion();

Callback interface

The main INativeNuiCallback callbacks are listed below. You must also implement onNuiVprEventCallback, even if voiceprint events are not used. An empty implementation is sufficient.

onNuiAudioStateChanged notifies the app to start, pause, or close recording.

void onNuiAudioStateChanged(Constants.AudioState state);

onNuiNeedAudioData continuously requests audio data. Write up to len bytes to buffer and return the number of bytes written. A negative value indicates a recording error; 0 indicates that no data is available. The SDK triggers EVENT_MIC_ERROR if no audio data is supplied for 2 consecutive seconds.

int onNuiNeedAudioData(byte[] buffer, int len);

onNuiEventCallback delivers recognition events. resultCode contains an error code, and arg2 is reserved. kwsResult is not used in this real-time recognition example. asrResult contains the recognition result or error response.

void onNuiEventCallback(Constants.NuiEvent event, int resultCode, int arg2,
        KwsResult kwsResult, AsrResult asrResult);

onNuiAudioRMSChanged returns an audio energy value from -160 to 0, which can be used for volume animations.

void onNuiAudioRMSChanged(float val);

Events

How do I configure audio for an 8 kHz model?

Select an 8 kHz model for the project, set both the AudioRecord sample rate and nls_config.sample_rate to 8000, and set sr_format to lowercase pcm. In the example, change SAMPLE_RATE to 8000 to keep recording and request parameters consistent.

Does recognition resume automatically after the network recovers?

In SDK 2.7.4, automatic reconnection is disabled by default. Set enable_reconnection in the initialization parameters to enable it. The SDK then attempts to reconnect when the connection is interrupted. If reconnection is disabled, the app must handle the error and restart the task.

Add the following before genInitParams returns the parameters:

parameters.put("enable_reconnection", "true");

Which Android versions are supported?

The current SDK requires Android 5.0 (API 21) or later.

For the platform requirements of a specific SDK version, see Select and download a mobile SDK.

Can I supply OPUS audio directly to the SDK?

Real-time speech recognition accepts 16-bit mono PCM audio at 8 kHz or 16 kHz. sr_format controls the audio transmission format and takes the lowercase value pcm or opus. When set to opus, the SDK compresses the input PCM data before transmission to reduce bandwidth usage.

How do I resolve an audio recorder initialization error?

Check that the app has recording permission and that AudioRecord initialized successfully. Use 16-bit mono PCM and match the sample rate to the recognition configuration. Call startRecording() before reading, stop() to pause recording, and release() when the recorder is no longer needed. Recreate a released recorder before recording again.

Use the following check in the audio callback before reading data:

public int onNuiNeedAudioData(byte[] buffer, int len) {
    if (mAudioRecorder == null
            || mAudioRecorder.getState() != AudioRecord.STATE_INITIALIZED) {
        Log.e(TAG, "Audio recorder not initialized");
        return -1;
    }
    return mAudioRecorder.read(buffer, 0, len);
}

How do I resolve initialization error 240021?

240021 indicates a FILE_ACCESS_FAIL file access error. Check read and write permissions on directories used by the SDK configuration. If local resources are used, check that resources and configuration files have been copied to the SDK workspace.

Use CommonUtils.copyAssetsData if local resources need to be copied from the AAR, and set the workspace path in the workspace initialization parameter. This step is not required for the cloud-only example.

if (!com.alibaba.idst.nui.CommonUtils.copyAssetsData(this)) {
    Log.e(TAG, "Failed to copy SDK assets");
    return;
}
String workspace = com.alibaba.idst.nui.CommonUtils.getModelPath(this);