Mobile Android Lite SDK

The MultiModalDialog SDK, provided by the Alibaba Cloud Tongyi team, supports end-to-end multi-modal real-time audio and video interaction. You can use the SDK to integrate with the Qwen Large Language Model (LLM) and various backend agents to enable capabilities such as voice conversation, weather, music, and news. It also supports LLM conversation capabilities for video and images.

Prerequisites

• Enable an Alibaba Cloud Model Studio real-time multimodal interaction application to get your workspace ID, app ID, and API key.

SDK Integration

Interactive Data Link Description

The multi-modal conversation Android Lite SDK only supports the WebSocket data link for server-side interaction and AudioOnly audio mode for conversations.

• For audio interaction, use a WebSocket connection because it offers faster connection speeds and lower performance requirements.

  • WebSocket Data Link Audio Format Description.

    • Uplink supports PCM and Opus audio formats for speech recognition.

    • Downlink supports PCM and MP3 audio streams.

Interactive Mode Description

The SDK supports three interactive modes: Push2Talk, Tap2Talk, and Duplex (full-duplex).

• Push2Talk: Press and hold to speak. Release to end recording.

• Tap2Talk: Tap to start speaking. The system automatically detects when the user finishes speaking.

• Duplex: This mode provides full-duplex interaction. After the connection starts, you can speak at any time and interrupt speech.

  Note: Full-duplex interaction requires the client to integrate an Acoustic Echo Cancellation (AEC) algorithm. The Lite SDK does not provide this algorithm module. You must integrate the Acoustic Echo Cancellation algorithm module yourself or use the full-featured Mobile Android SDK for Alibaba Cloud Model Studio multi-modal conversations.

Invocation Description

The SDK and its invocation demo are described below.

• You can download the SDK and demo dashscope-lite-android-1.0.2.zip and configure the necessary environment and dependencies.

• Then, import the example code from the compressed package and integrate the SDK by following the invocation process.

SDK Reference

You can import dependency libraries.

• app/src/main/libs

  • dashscope-multimodal-dialog-lite-1.*.aar // Alibaba Cloud multi-modal interactive Lite SDK

• Other dependencies imported by the demo application are listed below.

  For more information, see app/build.gradle.

  implementation 'com.alibaba:fastjson:1.1.76.android'
  implementation 'org.java-websocket:Java-WebSocket:1.5.7'

Key Parameters

Demo Overview

• EntranceActivity: This is the entry page. You can modify information such as `url`, `api_key`, and `app_id`.

  • On this page, you can select conversation modes such as Tap2Talk or Push2Talk.

• MultimodalDialogActivity: This is the conversation interaction implementation class.

  • The demo page references `TYAudioRecorder` as the audio input. You must replace it with your own implementation.

  • The demo page uses `AudioPlayer` as the audio playback output. You can choose to use your own implementation class.

  • In audio interaction mode, the demo supports the Visual Question Answering (VQA) (image-to-text) feature. Say "take a photo to identify xxx" to trigger the service to issue a photo-taking intent.

    • You can take a photo locally and upload it to OSS (or generate a public link using other content services) to trigger recognition and conversation for a single image.

    • You can also directly upload the image's Base64 encoding to request the service or the conversation result of the image.

Interface Design

MultimodalDialog Conversation Entry Class

1. MultiModalDialog

   You can initialize the conversation class by passing in necessary global parameters.

   /**
    * Initializes     
    * @param context: Context
    * @param url: Server address
    * @param workspaceId: Workspace ID
    * @param appId: Application ID
    * @param dialogMode: Conversation mode
    * */
       fun MultiModalDialog(
           context: Context,
           url: String?,
           workspaceId: String?,
           appId: String?,
           dialogMode: DialogMode?
       )

2. createConversation

   You can create a session.

   /**
    * After successful initialization, start the conversation flow.
    * @param params Initialization parameters
    * @param chatCallback Main callback. All messages, except those during initialization, are passed to the upper layer through this callback.
    */
    fun createConversation(@NonNull MultiModalRequestParam params,chatCallback: IConversationCallback)

3. start

   You can start a conversation.

   /**
    * Connects and starts a conversation.
    * @param apiKey: Model Studio service API key. A short-term API key is recommended.
    * @param dialogId: Conversation ID (optional). Pass a value to inherit previous conversation history.
    */
    fun start(String apiKey, String dialogId)

4. stop

   You can end a conversation.

   /**
    * Disconnects and ends a conversation.
    */
   fun stop()

5. destroy

   /**
    * Destroys the instance.
    */
   fun destroy()

6. interrupt

   You can interrupt interaction.

   /**
    * Interrupts AI speech.
    */
   fun interrupt()

7. startSpeech

   You can notify the server to start uploading audio. Note that you can only call this method when in the Listening state and only in Push2Talk mode.

   /**
    * Notifies the server to start uploading audio. Note that you can only call this method when in the Listening state.
    * Call this method only in Push2Talk mode.
    */
   fun startSpeech()

8. sendAudioData

   You can notify the server to upload audio.

   /**
    * Notifies the server to upload audio.
    * @param audioData Audio frame data
    */
   fun sendAudioData(byte[] data)

9. stopSpeech

   You can notify the server to stop uploading audio. Call this method only in Push2Talk mode.

   /**
    * Notifies the server to stop uploading audio. Call this method only in Push2Talk mode.
    * Push2Talk user finishes speaking.
    */
   fun stopSpeech()

10. requestToRespond

    You can request the server to answer a specific question or play Text-to-Speech (TTS).

    /**
     * Requests the server to answer a specific question or play Text-to-Speech (TTS).
     * @param type: "transcript" means converting text directly to speech; "prompt" means sending text to the Large Language Model (LLM) for a response.
     * @param text: Corresponding text
     * @param params: Additional parameters
     */
    fun requestToRespond(type: String, text: String, params: MultiModalRequestParam)

11. Other Interfaces.

    /**
     * Sends speech response ended.
     */
    fun sendResponseEnded()
        
    /**
     * Sends speech response started.
     */
    fun sendResponseStarted()
        

Key Parameter Enumerations

MultiModalRequestParam Client-Cloud Interaction Parameter Details

IConversationCallback (Callback Interface)

/**
  * Authentication passed, WebSocket connection established.
  */
fun onConnected()

/**
  * Conversation established, start conversation flow.
  */
fun onStarted(dialogId: String)

/**
   * State switch.
   * Includes DIALOG_IDLE, DIALOG_LISTENING, DIALOG_RESPONDING, DIALOG_THINKING.
   */
fun onConvStateChangedCallback(state: State.DialogState)

/**
   * Timeout exceeding the user's valid input timeout. Upon receiving this event, restart or end the conversation.
   * @param timeout Timeout duration
   */
fun onSpeechTimeout(timeout: Long)

/**
   * Exception information during the conversation.
   * @param errorInfo Exception information
   */
fun onErrorReceived(errorCode: Int, errorMessage: String)

/**
   * Callback for synthesized TTS audio.
   * @param bytes Audio data
   */
fun onSynthesizedSpeech(bytes: ByteArray)

/**
   * Conversation ended.
   */
fun onStopped()

/**
   * User started speaking detected.
   */
fun onSpeechStarted()

/**
   * User finished speaking detected.
   */
fun onSpeechEnded()

/**
   * Interrupt request accepted.
   */
fun onRequestAccepted()

/**
   * Cloud starts sending TTS response.
   */
fun onRespondingStarted()

/**
   * Cloud finishes sending TTS response.
   */
fun onRespondingEnded(output: Map<String, Any>)

/**
   * Speech recognition content.
   */
fun onSpeechContent(output: Map<String, Any>)

/**
   * Conversation response content.
   */
fun onRespondingContent(output: Map<String, Any>)

/**
   * WebSocket connection closed.
   */
fun onClosed(closeCode: Int, reason: String)

Important Callback Method Descriptions

• onSpeechContent(output: Map<String, Any>)

  This provides speech recognition content.

  SpeechContent - Response

  Primary Parameter	Secondary Parameter	Tertiary Parameter	Type	Required	Description
  output	event		string	Yes	Event name: SpeechContent
  	dialog_id		string	Yes	Conversation ID
  	text		string	Yes	The text recognized from the user's speech, streamed as full output.
  	finished		bool	Yes	Indicates whether the output has finished.

  {
      "header": {
          "request_id": "9B32878******************3D053",
          "service_id": "368208df",
          "status_code": 200,
          "status_name": "Success",
          "status_message": "Success.",
          "attributes":{
            "user_id":"1234557879x"
          }
      },
      "payload": {
          "output":{
            "event": "SpeechContent",
            "dialog_id": "b39398c9dd8147********35cdea81f7",
            "text": "one two three",
            "finished": false
          },
          "usage":{
            "invoke":10,
            "model_x":10
          }
      }
  }

• onRespondingContent(output: Map<String, Any>)

  This provides text returned by the Large Language Model (LLM).

  Primary Parameter	Secondary Parameter	Tertiary Parameter	Type	Required	Description
  output	finished		bool	Yes	Indicates whether the output has finished.
  	dialog_id		string	Yes	Conversation ID
  	event		string	Yes	Message type
  	text		string	No	The text result returned by the LLM.
  	spoken		string	No	The text of the content to be played, returned by the LLM. This may differ from the `text` field.
  	extra_info		object	No	Other extension information. Currently supports: commands: command string agent_info: agent information tool_calls: information returned by the plugin dialog_debug: conversation debug information timestamps: timestamps of each node in the data link

  Example:

  {
      "header": {
          "event": "result-generated",
          "task_id": "9B32878******************3D053"
      },
      "payload": {
          "output": {
              "event": "RequestAccepted",
              "dialog_id": "b39398c9dd8147********35cdea81f7",
              "text": "You entered the number sequence \"12345\". If you have questions about these numbers or need me to use them for a task, provide more details and I will do my best to help.",
              "spoken": "You entered the number sequence \"12345\". If you have questions about these numbers or need me to use them for a task, provide more details and I will do my best to help.",
              "finished": true,
              "extra_info": {
                  "commands": "[{\"name\":\"VOLUME_SET\",\"params\":[{\"name\":\"series\",\"normValue\":\"70\",\"value\":\"70\"}]}]",
                  "tool_result": [
                      {
                          "id": "",
                          "type": "function",
                          "function": {
                              "name": "function_name",
                              "arguments": "{\"id\": \"123\", \"name\": \"test\"}",
                              "outputs": "Function call result",
                              "status": {
                                  "code": 200,
                                  "message": "Success."
                              }
                          }
                      }
                  ]
              }
          },
          "usage": {
              "invoke": 10,
              "model_x": 10
          }
      }
  }
  
  

Error Handling

onErrorReceived - Response

This section lists error codes and their corresponding error messages.

Invocation Timing

Half-duplex Interaction

@Override
public void onRespondingContent(@NotNull final Map<String, ?> output) {
  Log.i(TAG,"onRespondingContent output:{"+output+"}");     
  if (output.containsKey("extra_info")) {
    JSONObject extraInfo = (JSONObject) output.get("extra_info");
    try {
        if (extraInfo.containsKey("commands") && extraInfo.getString("commands") != null && extraInfo.getString("commands").contains("visual_qa")) {
        // Contains VQA instruction, simple implementation here.
            uploadVQAImg();
            }
        } catch (JSONException e) {
            throw new RuntimeException(e);
        }
    }
}

// Upload photo.
private void uploadVQAImg(){
    MultiModalRequestParam updateParams = null;
    JSONObject imageObject = new JSONObject();
    try{
        imageObject.put("type", "url");
        imageObject.put("value", vqaImgLink);
        List<JSONObject> images = new ArrayList<>();
        images.add(imageObject);
            updateParams= MultiModalRequestParam
                .builder()
                .images(images)
                    .build();
    }catch (JSONException e){
        e.printStackTrace();
    }
    Log.i(TAG, "uploadVQAImg: " + JSON.toJSONString(updateParams));
    multimodalDialog.requestToRespond("prompt", "",updateParams);
}

// 1. Set the interaction type to AudioAndVideo.
MultiModalRequestParam.UpStream.builder().mode(this.dialogMode.getValue()).type("AudioAndVideo").build();

// 2. Send a voicechat_video_channel request after establishing the connection.
private void sendVideoChatCommand(){
    try {
        MultiModalRequestParam updateParams = MultiModalRequestParam.builder().build();
        org.json.JSONObject videoObj = new org.json.JSONObject();
        videoObj.put("action", "connect");
        videoObj.put("type", "voicechat_video_channel");

        List<org.json.JSONObject> videos = new ArrayList<>();
        videos.add(videoObj);

        updateParams.setBizParams(MultiModalRequestParam.BizParams.builder()
                .videos(videos).build());

        multimodalDialog.requestToRespond("prompt", "", updateParams);

    } catch (org.json.JSONException e) {
        Log.e(TAG, "Failed to start video mode", e);
    }
}

// 3. Submit one video frame image data every 500 ms. Note that the image size must be less than 180 KB.
/**
 * Demonstrates implementing LiveAI in a WebSocket data link.
 * Starts a video frame stream, sending one image frame every 500 ms.
 */
private void startVideoFrameStreaming() {
    Thread videoStreamingThread = new Thread(new Runnable() {
        @Override
        public void run() {
            try {
                while (isVideoChatting && !Thread.currentThread().isInterrupted()) {
                    Thread.sleep(500);

                    MultiModalRequestParam updateParams = null;
                    try {

                        updateParams = MultiModalRequestParam
                                .builder()
                                .images(MultimodalDialogActivity.this.getImageList())
                                .build();
                    } catch (JSONException e) {
                        e.printStackTrace();
                    }

                    multimodalDialog.updateInfo(updateParams);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            } catch (JSONException e) {
                throw new RuntimeException(e);
            }
        }
    });

    videoStreamingThread.setDaemon(true);
    videoStreamingThread.setName("LiveAI-VideoStreaming");
    videoStreamingThread.start();

}

/**
 * Builds images list request.
 * */
private List getImageList() {
    AssetManager assetManager = getAssets();
    org.json.JSONObject imageObject = new org.json.JSONObject();
    List<org.json.JSONObject> images = new ArrayList<>();
    try{

        imageObject.put("type", "base64");
        imageObject.put("value", getImageBase64(assetManager.open("jpeg-bridge.jpg")));

        images.add(imageObject);

    }catch (Exception e){
        e.printStackTrace();
    }
    return images;
}

private String getImageBase64(InputStream file) { // Images smaller than 200 KB.
    try {

        byte[] bytes = new byte[file.available()];
        file.read(bytes);
        return Base64.encodeToString(bytes, Base64.NO_WRAP); // Use the NO_WRAP parameter to avoid line breaks.
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}

multimodalDialog.requestToRespond("transcript","Happiness is a skill. It is the inner peace you feel after letting go of excessive external desires.",null);

// Set user_prompt_params in biz_params when building the request parameters.
HashMap<String, String> userPromptParams = new HashMap<>();
userPromptParams.put("user_name", "Alice");
        MultiModalRequestParam.BizParams bizParams = MultiModalRequestParam.BizParams
                .builder()
                .userPromptParams(userPromptParams)
                .build();