Mobile Android SDK

Alibaba Cloud's MultiModalDialog SDK enables end-to-end, real-time multimodal interaction using audio and video. By integrating with the Qwen large model and various backend agents, the SDK offers a wide range of capabilities, including voice conversations, weather, music, news, and visual conversations based on video and image inputs.

Prerequisites

• Activate the real-time multimodal interactive application in Alibaba Cloud Model Studio and get your Workspace ID, APP ID, and API key.

• Download the SDK and demo, and configure the required environment and dependencies.

• Import the sample code and follow the call flow to integrate the SDK.

• Run the APK test program directly from the compressed package and enter the three required parameters.

SDK integration

Interactive data links

The SDK communicates with the server via the WebSocket protocol and supports two modes: AudioOnly and AudioAndVideo.

• audio interaction: Enables audio and text conversations only.

• audio-video interaction: Allows you to enter video call mode using commands, where the client continuously sends an image sequence to the server to enable end-to-end, multimodal audio-video interaction.

Interactive mode description

The SDK features three interaction modes: Push2Talk, Tap2Talk, and Duplex (full-duplex).

• Push2Talk: Press and hold to speak, and release to end recording.

• Tap2Talk: Tap to start speaking; recording stops automatically when speech ends.

• Duplex: Full-duplex interaction that lets you speak at any time once connected and supports barge-in.

Usage

SDK and Demo

SDK reference

Import the required library.

• app/src/main/libs

  • convsdk-release_*.aar // Alibaba Cloud Voice Chat SDK

  • multimodal_dialog_sdk.aar // Alibaba Cloud Multi-modal Dialog SDK

  • multimodal_dialog_tongyimetathings.aar // Authentication SDK for the license mode.

• Other dependencies for the demo app:

  Refer to app/build.gradle for the complete list.

  implementation libs.androidbootstrap
  implementation libs.okhttp
  implementation libs.okio // Handles I/O for OkHttp3 on Android
  implementation libs.http.logging.interceptor
  implementation libs.core.ktx

Key parameters

Demo

• EntranceActivity: The entry page for modifying information such as url, api_key, and app_id.

  • On this page, you can select a dialog mode, such as Tap2Talk or Duplex.

  • To use license mode, see Integrate the Android SDK using license mode.

• MultimodalDialogActivity: The class that implements the dialog interaction.

  • The demo uses TYAudioRecorder for audio input. You can replace it with your own implementation.

  • The demo uses AudioPlayer for audio playback. You can replace it with your own implementation.

  • In audio interaction mode, the demo supports VQA. For example, a voice command such as "take a photo and identify xxx" triggers the service to send a take photo intent. Then:

    • You can take a photo locally and upload it to OSS (or another content service) to generate a public URL. You then use this URL to trigger single-image recognition and a dialog.

    • Alternatively, you can directly upload the image's base64 data to the service to receive a recognition result and start a dialog.

  • In the VideoAndAudio mode, the demo accepts an image sequence from an external capture source. This approach simplifies integrating capture and video dialog for IoT devices like smart glasses.

API design

MultimodalDialog

1. MultiModalDialog

   Initializes the dialog class.

   /**
        * Initializes the class.     
        * @param context The context.
        * @param url The server URL.
        * @param chainMode The chain mode.
        * @param workspaceId The workspace ID.
        * @param appId The app ID.
        * @param dialogMode The dialog mode.
        * */
       fun MultiModalDialog(
           context: Context,
           url: String?,
           chainMode: ChainMode?,
           workspaceId: String?,
           appId: String?,
           dialogMode: DialogMode?
       )

2. createConversation

   Creates a conversation.

   /**
        * Creates a conversation after successful initialization.
        * @param params The initialization parameters.
        * @param chatCallback The main callback. Receives all messages from the conversation, except for those sent during initialization.
        */
       fun createConversation(@NonNull MultiModalRequestParam params,chatCallback: IConversationCallback)

3. start

   Starts the conversation.

   /**
        * Establishes a connection and starts the conversation.
        */
       fun start()

4. stop

   Stops the conversation.

   /**
        * Disconnects and stops the conversation.
        */
       fun stop()

5. destroy

       /**
        * Destroys the instance.
        */
       fun destroy()

6. setConversationTimeout

   Sets the conversation timeout.

       /**
        * Sets the conversation timeout. If the service does not detect user speech within the specified time, it reports a timeout event.
        * @param timeout The timeout in milliseconds (ms).
        */
       fun setConversationTimeout(timeout: Long)

7. interrupt

   Interrupts the AI's speech.

       /**
        * Interrupts the AI's speech.
        */
       fun interrupt()

8. startSpeech

   Notifies the server to start uploading audio. This method is required only in Push-to-Talk mode and must be called while in the Listening state.

   /**
    * Notifies the server to start uploading audio. This method must be called when in the Listening state.
    * It is only required in Push-to-Talk mode.
    */
   fun startSpeech()

9. sendAudioData

   Sends audio data to the server.

   /**
    * Sends audio data to the server.
    * @param audioData The audio frame data.
    */
   fun sendAudioData(audioData: ByteArray?)

10. stopSpeech

    Notifies the server to stop uploading audio. This method is required only in Push-to-Talk mode.

    /**
     * Notifies the server to stop uploading audio. This method is only required in Push-to-Talk mode.
     * In Push-to-Talk mode, this indicates that the user has finished speaking.
     */
    fun stopSpeech()

11. requestToRespond

    Requests the server to answer a question or play a response using text-to-speech (TTS).

    /**
         * Requests the server to answer a specific question or play a response by using text-to-speech (TTS).
         * @param type Use 'transcript' to convert the text directly to speech, or 'prompt' to get an answer from the large model.
         * @param text The corresponding text.
         * @param params Additional parameters.
         */
        fun requestToRespond(type: String, text: String, params: JSONObject)

12. Other methods

        /**
         * Checks if the current configuration is in duplex mode.
         */
        fun isDuplexMode(): Boolean
    
        /**
         * Cancels the user's speech in Push-to-Talk mode.
         */
        fun cancelSpeech()
    
        /**
         * Notifies the system that the voice response has ended.
         * */
        fun sendResponseEnded()
        
        /**
         * Notifies the system that the voice response has started.
         * */
        fun sendResponseStarted()
    
        /**
         * Pushes a video frame.
         *
         * @param videoFrame The video frame.
         * @param callback The callback for the frame push result.
         */
        fun sendVideoFrame(videoFrame: TYVideoFrame,callback: IVideoChatPushFrameCallback)
    
        

Key parameters

MultiModalRequestParam

{
    "header": {
        "action":"run-task",
        "task_id": "f894c16f-f20e-4c1d-837e-89e0fbc63a43",
        "streaming":"duplex"
    },
    "payload": {
        "task_group":"aigc",
        "task":"multimodal-generation", // Task: multimodal generation
        "function":"generation",
        "model":"multimodal-dialog", // Model: multimodal dialog (note this is different from the task)
        "input":{
          "directive": "Start",
          "workspace_id": "llm-***********",
          "app_id": "****************"
        },
        "parameters":{
          "upstream":{
            "type": "AudioOnly",
            "mode": "duplex"
          },
          "downstream":{
            "voice": "longxiaochun_v2",
            "sample_rate": 24000
          },
          "client_info":{
            "user_id": "bin********207",
            "device":{
              "uuid": "432k*********k449"
            },
            "network":{
              "ip": "203.0.113.10"
            },
            "location":{
              "city_name": "Beijing"
            }
          },
          "biz_params":{
            "user_defined_params": {
                "extra_config": {
                    "enable_web_search": false
                },
                "agent_id_xxxxx": {
                    "name": "value"
                }
            },
            "user_prompt_params": {
                "name": "value"
            },
            "user_query_params": {
                "name": "value"
            }
          }
        }
    }
}

IConversationCallback

    /**
     * Invoked after the client successfully authenticates and joins the channel.
     */
    fun onStartResult(isSuccess: Boolean, errorInfo: TYError?)

    /**
     * Invoked when an interruption occurs, including manual and voice interruptions.
     */
    fun onInterruptResult(isSuccess: Boolean, errorInfo: TYError?)

    /**
     * Invoked when the conversation is ready, after all endpoints (client, VoiceChat, and video understanding) have joined the channel. Call business APIs only after this callback is invoked.
     */
    fun onReadyToSpeech()

    /**
     * Invoked on a conversation state transition.
     * Possible states include DIALOG_IDLE, DIALOG_LISTENING, DIALOG_RESPONDING, and DIALOG_THINKING.
     */
    fun onConvStateChangedCallback(state: DialogState)

    /**
     * Reports the current sound level.
     * @param audioType See {@link Constant.TYVolumeSourceType}.
     * @param audioLevel The sound level, ranging from 0 to 100.
     */
    fun onConvSoundLevelCallback(audioLevel: Float, audioType: Constant.TYVolumeSourceType)

    /**
     * Invoked when a speech timeout occurs. You must then either restart or end the conversation.
     * @param timeout The timeout duration.
     */
    fun onSpeechTimeout(timeout: Long)

    /**
     * Provides loopback audio data for client-side Acoustic Echo Cancellation (AEC).
     * @param bytes The audio data.
     */
    fun onPlaybackAudioData(bytes: ByteArray)

    /**
     * Invoked when an error occurs during the conversation.
     * @param errorInfo The error information.
     */
    fun onErrorReceived(errorInfo: TYError)

    /**
     * Invoked when an event occurs during the conversation.
     * @param var1 The event.
     */
    fun onConvEventCallback(var1: ConvEvent?)

    /**
     * Reports key log messages generated during runtime.
     * @param level The log level. See Android.Log for reference.
     * @param type The log type.
     * @param debugInfo The log message.
     */
    fun onDebugInfoTrack(level: Int, type: Constant.TYDebugInfoType, debugInfo: String)

onConvEventCallback: AI conversation response

• EVENT_HUMAN_SPEAKING_DETAIL

  Speech recognition content

  SpeechContent - response

  Level 1 parameter	Level 2 parameter	Level 3 parameter	Type	Required	Description
  output	event		string	Yes	The event name. Always SpeechContent.
  	dialog_id		string	Yes	The dialog ID.
  	text		string	Yes	The text transcribed from the user's speech. The service streams the cumulative result.
  	finished		bool	Yes	Indicates whether the output is complete.

  {
      "header": {
          "request_id": "9B32878******************3D053",
          "service_id": "368208df",
          "status_code": 200,
          "status_name": "Success",
          "status_message": "Success.",
          "attributes":{
            "user_id":"1234557879x"
          }
      },
      "payload": {
          "output":{
            "event": "SpeechContent",
            "dialog_id": "b39398c9dd8147********35cdea81f7",
            "text": "one two three",
            "finished": false
          },
          "usage":{
            "invoke":10,
            "model_x":10
          }
      }
  }

• EVENT_RESPONDING_DETAIL

  Response from the large model

  output	finished		bool	Yes	Indicates whether the output is complete.
  	dialog_id		string	Yes	The dialog ID.
  	event		string	Yes	The event type.
  	text		string	No	The text returned by the large model.
  	spoken		string	No	The text to be played back. This may differ from the text field.
  	extra_info		object	No	Additional information. Currently supports: commands: A command string. agent_info: Information about the agent. tool_calls: The results returned by tools. dialog_debug: Debug information for the dialog. timestamps: Timestamps for each node in the trace.

  Example:

  {
      "header": {
          "event":"result-generated",
          "task_id": "9B32878******************3D053"
      },
      "payload": {
          "output":{
            "event": "RequestAccepted",
            "dialog_id": "b39398c9dd8147********35cdea81f7",
            "text": "You entered the number sequence '12345'. If you have any questions about these numbers or need me to perform a task with them, please provide more details, and I will do my best to help you.",
            "spoken": "You entered the number sequence '12345'. If you have any questions about these numbers or need me to perform a task with them, please provide more details, and I will do my best to help you.",
            "finished": true,
          "extra_info": {
            "commands": "[{\"name\":\"VOLUME_SET\",\"params\":[{\"name\":\"series\",\"normValue\":\"70\",\"value\":\"70\"}]}]",
            "tool_result": [{
              "id": "",
              "type": "function",
              "function": {
                "name": "function_name",
                "arguments": "{\"id\": \"123\", \"name\": \"test\"}",
                "outputs": "function call result",
                "status": {
                  "code": 200,
                  "message": "Success."
                }
              }
            }]
          }
          },
          "usage":{
            "invoke":10,
            "model_x":10
          }
      }
  }

Exception handling

onErrorReceived - response

Call sequence

Full-duplex interaction

Half-duplex communication

multiModalDialog.sendAudioData(data);

multiModalDialog.sendRefData(data);

private void executeCmd(String cmd,String dialogId,String taskId){
  String cmdName = null;
  String cmdParamValue = null;
  try {
    cmdName = new JSONArray(cmd).getJSONObject(0).getString("name");
  }catch (Exception ex) {
    ex.printStackTrace();
  }
  switch (cmdName){
     case "visual_qa":
       JSONObject extraObject = new JSONObject();
       extraObject.put("images", imagesArray);
       multimodalConversation.requestToRespond(type, text, extraObject.toString());
       break;
  ......
}

// Upload the photo
public static JSONArray getMockOSSImage() {
    JSONObject imageObject = new JSONObject();
    JSONArray images = new JSONArray();
    try{
      if (vqaUseUrl){
        imageObject.put("type", "url");
        imageObject.put("value", "https://help-static-aliyun-doc.aliyuncs.com/assets/img/zh-CN/7043267371/p909896.png");
      }else {
        imageObject.put("type", "base64");
        imageObject.put("value", getLocalImageBase64());
      }
      images.put(imageObject);

    }catch (Exception e){
      e.printStackTrace();
    }
    return images;
  }

// 1. Set the interaction type to AudioAndVideo.
MultiModalRequestParam.UpStream.builder().mode("duplex").type("AudioAndVideo").build();

// 2. After establishing a connection, send a voicechat_video_channel request.
private void startVideoMode() {
    if (isVideoMode) return;

    try {
        MultiModalRequestParam updateParams = MultiModalRequestParam.builder().build();
        JSONObject videoObj = new JSONObject();
        videoObj.put("action", "connect");
        videoObj.put("type", "voicechat_video_channel");

        List<JSONObject> videos = new ArrayList<>();
        videos.add(videoObj);

        updateParams.setBizParams(MultiModalRequestParam.BizParams.builder()
                .videos(videos).build());

        multiModalDialog.requestToRespond("prompt", "", updateParams.getParametersAsJson());
        multiModalDialog.setVideoContainer(videoContainer, uiHandler);

        isVideoMode = true;
        runOnUiThread(() -> videoContainer.setVisibility(View.VISIBLE));

    } catch (JSONException e) {
        Log.e(TAG, "Failed to start video mode", e);
    }
}

// 3. Submit a video frame every 500 ms. Note that the image size must be smaller than 180 KB.
/**
 * This flow is not called in the demo. It demonstrates how to implement LiveAI over a WebSocket connection.
 * Starts the video frame stream and sends a frame every 500 ms.
 */
private void startVideoFrameStreaming() {
    Thread videoStreamingThread = new Thread(() -> {
        try {
            while ( !Thread.currentThread().isInterrupted()) {
                Thread.sleep(500);

                JSONObject extraObject = new JSONObject();
                extraObject.put("images", getMockOSSImage());

                multiModalDialog.updateInfo(extraObject);
                }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } catch (JSONException e) {
            throw new RuntimeException(e);
        }
    });

    videoStreamingThread.setDaemon(true);
    videoStreamingThread.setName("LiveAI-VideoStreaming");
    videoStreamingThread.start();

}

/**
 * Builds the image list request.
 * */
private static JSONArray getMockOSSImage() {
    JSONObject imageObject = new JSONObject();
    JSONArray images = new JSONArray();
    try{

        imageObject.put("type", "base64");
        imageObject.put("value", getLocalImageBase64());

        images.put(imageObject);

    }catch (Exception e){
        e.printStackTrace();
    }
    return images;
}

// Reads a local image to a base64 string.
private static String getLocalImageBase64(){
    return "";
}

conversation.requestToRespond("transcript","Happiness is a skill, a state of inner peace achieved by letting go of external desires.",null);

// Pass biz_params.user_prompt_params when you build the request parameters.
HashMap<String, String> userPromptParams = new HashMap<>();
userPromptParams.put("user_name", "Da Mi");
        MultiModalRequestParam.BizParams bizParams = MultiModalRequestParam.BizParams
                .builder()
                .userPromptParams(userPromptParams)
                .build();

// Build the AsrPostProcessing object.
MultiModalRequestParam.UpStream.ReplaceWord replaceWord = new MultiModalRequestParam.UpStream.ReplaceWord();
    replaceWord.setTarget("one plus one");
    replaceWord.setSource("1 plus 1");
    replaceWord.setMatchMode("partial");
// Add the word replacement rules to the UpStream parameters.
.upStream(MultiModalRequestParam.UpStream.builder()
.asrPostProcessing(Collections.singletonList(replaceWord))
.build())