Java SDK
This topic explains how to use the server-side Java SDK for real-time multimodal interaction with large model services in Alibaba Cloud Model Studio, including SDK installation, key APIs, and code samples.
Multimodal real-time interactive architecture
Prerequisites
Enable an Alibaba Cloud Model Studio real-time multimodal interaction application to get your workspace ID, app ID, and API key.
Environment and installation
You can integrate the official Alibaba Cloud Dashscope SDK, version 2.21.16 or later, using maven or gradle.
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>dashscope-sdk-java</artifactId>
<version>2.21.16</version>
</dependency>
Interaction modes
The SDK supports three interaction modes: Push2Talk, Tap2Talk, and Duplex (full-duplex).
- Push2Talk: Press and hold to speak, then release to stop (or click to start and click again to stop).
- Tap2Talk: Click to start speaking. The SDK automatically stops recording when you finish speaking.
- Duplex: Enables full-duplex interaction, allowing users to speak at any time once connected. This mode supports barge-in and is recommended for video interaction.
Audio format
Server-side integration supports only the websocket protocol.
-
Audio formats:
- Uplink supports pcm and opus audio for speech recognition.
- Downlink supports pcm and mp3 audio streams.
API
MultiModalDialog
Service entry class.
1. MultiModalDialog(MultiModalRequestParam param, MultiModalDialogCallback callback)
Initializes the service object and sets the dialog parameters and callback.
/**
* Constructor initializes service options and creates a duplex communication API instance.
*
* @param param Request parameter
* @param callback Callback interface
*/
public MultiModalDialog(MultiModalRequestParam param, MultiModalDialogCallback callback)
2. start
Starts the dialog service and triggers the onStarted callback. Note that the onStarted callback returns the dialogId.
/**
* Starts the conversation session.
*/
public void start()
3. startSpeech
Notifies the server to start uploading audio. This method can only be called when the dialog is in the Listening state and is only required in Push-to-Talk mode.
/**
* Starts upload speech.
*/
public void startSpeech()
4. sendAudioData
Sends audio data to the server.
/**
* Sends audio frame.
*
* @param audioFrame Audio frame data
*/
public void sendAudioData(ByteBuffer audioFrame)
5. stopSpeech
Notifies the server to stop uploading audio. This method is only required in Push-to-Talk mode.
/**
* Stops upload speech.
*/
public void stopSpeech()
6. interrupt
Notifies the server that the client is interrupting the current interaction to start speaking. The server then responds with RequestAccepted.
/**
* Interrupts current operation.
*/
public void interrupt()
7. localRespondingStarted
Notifies the server that the client has started playing the TTS audio.
/**
* Local player start broadcast tts
*/
public void localRespondingStarted()
8. localRespondingEnded
Notifies the server that the client has finished playing the TTS audio.
/**
* Local player broadcast tts end
*/
public void localRespondingEnded()
9. stop
Ends the current dialog turn.
/**
* Stops the conversation.
*/
public void stop()
10. getDialogState
Gets the current dialog state. The state is represented by the DialogState enum.
/**
* Gets current dialogue state.
*
* @return Current dialogue state
*/
public State.DialogState getDialogState()
11. requestToRespond
Lets the client directly initiate Text-to-Speech synthesis from text or send other requests, like images, to the server.
/**
* Requests response.
*
* @param type Response type
* @param text Response text
* @param updateParams Update parameters
*/
public void requestToRespond(String type, String text, MultiModalRequestParam.updateParams updateParams)
12. updateInfo
Updates information, such as dialog parameters.
/**
* Updates information.
*
* @param updateParams Update parameters
*/
public void updateInfo(MultiModalRequestParam.updateParams updateParams)
13. sendHeartBeat
Sends a heartbeat to prevent a 60-second timeout error. For scenarios requiring a persistent connection, call this method every 20 seconds.
/**
* Send heart beat.
*
*/
public void sendHeartBeat()
MultiModalDialogCallback
Defines callbacks for multi-modal conversation events.
import com.alibaba.dashscope.multimodal.State;
/**
* Abstract class representing callbacks for multi-modal conversation events.
*
* @author songsong.shao
* @date 2025/4/27
*/
public abstract class MultiModalDialogCallback {
/**
* Called when the conversation is connected.
*/
public abstract void onConnected();
/**
* Called when a conversation starts with a specific dialog ID.
*
* @param dialogId The unique identifier for the dialog.
*/
public abstract void onStarted(String dialogId);
/**
* Called when a conversation stops with a specific dialog ID.
*
* @param dialogId The unique identifier for the dialog.
*/
public abstract void onStopped(String dialogId);
/**
* Called when speech starts in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
*/
public abstract void onSpeechStarted(String dialogId);
/**
* Called when speech ends in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
*/
public abstract void onSpeechEnded(String dialogId);
/**
* Called when an error occurs during a conversation.
*
* @param dialogId The unique identifier for the dialog.
* @param errorCode The error code associated with the error.
* @param errorMsg The error message associated with the error.
*/
public abstract void onError(String dialogId, String errorCode, String errorMsg);
/**
* Called when the conversation state changes.
*
* @param state The new state of the conversation.
*/
public abstract void onStateChanged(State.DialogState state);
/**
* Called when speech audio data is available.
*
* @param audioData The audio data as a ByteBuffer.
*/
public abstract void onSpeechAudioData(ByteBuffer audioData);
/**
* Called when responding starts in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
*/
public abstract void onRespondingStarted(String dialogId);
/**
* Called when responding ends in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
* @param content The content of the response as a JsonObject.
*/
public abstract void onRespondingEnded(String dialogId, JsonObject jsonObject);
/**
* Called when responding content is available in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
* @param content The content of the response as a JsonObject.
*/
public abstract void onRespondingContent(String dialogId, JsonObject content);
/**
* Called when speech content is available in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
* @param content The content of the speech as a JsonObject.
*/
public abstract void onSpeechContent(String dialogId, JsonObject content);
/**
* Called when a request is accepted in a specific dialog.
*
* @param dialogId The unique identifier for the dialog.
*/
public abstract void onRequestAccepted(String dialogId);
/**
* Called when the conversation closes.
*/
public abstract void onClosed();
}
MultimodalRequestParam
Request parameter class.
All request parameters can be configured using the builder pattern. The following tables describe each parameter.
Start request parameters
Level 1 | Level 2 | Level 3 | Level 4 | Type | Required | Description |
task_group | The task group name. The value must be "aigc". | |||||
task | The task name. The value must be "multimodal-generation". | |||||
function | The function to call. The value must be "generation". | |||||
model | The service name. The value must be "multimodal-dialog". | |||||
input | workspace_id | string | Yes | The workspace ID. | ||
app_id | string | Yes | The ID of the application you created in the console. This ID determines which dialog system is used. | |||
directive | string | Yes | The directive name. The value must be Start. | |||
dialog_id | string | No | The dialog ID. If specified, the service continues the existing conversation. | |||
parameters | upstream | type | string | Yes | The upstream type:
| |
mode | string | No | The mode used by the client. Optional values:
The default value is tap2talk. | |||
audio_format | string | No | The audio format. Supported formats are pcm and opus. The default value is pcm. | |||
downstream | voice | string | No | The voice for the synthesized audio. | ||
sample_rate | int | No | The sample rate for the synthesized voice. The default is 24 kHz. | |||
intermediate_text | string | No | Controls which intermediate text is returned to the user:
You can specify multiple types, separated by a comma. The default value is transcript. | |||
audio_format | string | No | The audio format. Supported formats are pcm and mp3. The default value is pcm. | |||
client_info | user_id | string | Yes | The end user ID. This is used for user-specific handling. | ||
device | uuid | string | No | A globally unique ID for the client device. This ID must be generated by the caller and passed to the SDK. | ||
network | ip | string | No | The public IP address of the caller. | ||
location | latitude | string | No | The latitude of the caller. | ||
longitude | string | No | The longitude of the caller. | |||
city_name | string | No | The city of the caller. | |||
biz_params | user_defined_params | object | No | Custom parameters to pass through to the agent. | ||
user_defined_tokens | object | No | Custom authentication tokens to pass through to the agent. | |||
tool_prompts | object | No | Custom prompts to pass through to the agent. | |||
user_query_params | object | No | Custom parameters to pass through with the user request. | |||
user_prompt_params | object | No | Custom parameters to pass through with the user prompt. |
RequestToRespond request parameters
Level 1 | Level 2 | Level 3 | Type | Required | Description |
input | directive | string | Yes | The directive name. The value must be RequestToRespond. | |
dialog_id | string | Yes | The dialog ID. | ||
type | string | Yes | The type of interaction the service should perform:
| ||
text | string | Yes | The text to process. It can be an empty string ("") but must not be null. | ||
parameters | images | list[] | No | Information about the images to be analyzed. | |
biz_params | object | No | Same as the biz_params in the Start message, used to pass custom parameters to the dialog system. The biz_params in a RequestToRespond request apply only to that request. |
UpdateInfo request parameters
Level 1 | Level 2 | Level 3 | Type | Required | Description |
input | directive | string | Yes | The directive name. The value must be UpdateInfo. | |
dialog_id | string | No | The dialog ID. | ||
parameters | images | list[] | No | The image data. | |
client_info | status | object | No | The current status of the client. | |
biz_params | object | No | Same as the biz_params in the Start message, used to pass custom parameters to the dialog system. The biz_params in an UpdateInfo request apply only to that request. |
Dialog state (DialogState)
The voicechat service has three states: LISTENING, THINKING, and RESPONDING, which represent:
IDLE("Idle"), 系统尚未建立连接
LISTENING("Listening"), 表示机器人正在监听用户输入。用户可以发送音频。
THINKING("Thinking"),表示机器人正在思考。
RESPONDING("Responding")表示机器人正在生成语音或语音回复中。
LLM dialogue output
onRespondingContent
Top-level parameter | Second-level parameter | Third-level parameter | Type | Required | Description |
output | event | string | Yes | The event name, such as RequestAccepted. | |
dialog_id | string | Yes | Unique dialogue ID. | ||
round_id | string | Yes | Unique ID for the current turn. | ||
llm_request_id | string | Yes | The request ID for the LLM call. | ||
text | string | Yes | Text content for display. Each streaming event provides the full, cumulative text. | ||
spoken | string | Yes | Text content for speech synthesis. Each streaming event provides the full, cumulative text. | ||
finished | bool | Yes | Indicates whether the output is complete. | ||
extra_info | object | No | Contains additional information. Supported fields include:
|
Call sequence diagram
Example
1. Initialize parameters
Call the builder methods of the inner classes in MultiModalRequestParam to construct the parameters.
MultiModalRequestParam params =
MultiModalRequestParam.builder()
.customInput(
MultiModalRequestParam.CustomInput.builder()
.build())
.upStream(
MultiModalRequestParam.UpStream.builder()
.mode("push2talk")
.audioFormat("pcm")
.build())
.downStream(
MultiModalRequestParam.DownStream.builder()
.voice("longxiaochun_v2")
.sampleRate(48000)
.build())
.dialogAttributes(
MultiModalRequestParam.DialogAttributes.builder().prompt("你好").build())
.clientInfo(
MultiModalRequestParam.ClientInfo.builder()
.userId("1234")
.device(MultiModalRequestParam.ClientInfo.Device.builder().uuid("1234").build())
.build())
.apiKey("sk-9d11****************************")
.model("multimodal-dialog")
.build();
2. Instantiate callback
public static MultiModalDialogCallback getCallback() {
return new MultiModalDialogCallbackImpl();
}
public static class MultiModalDialogCallbackImpl extends MultiModalDialogCallback {
...
}
3. Create a MultiModalDialog object
conversation = new MultiModalDialog(params, getCallback());
Full example
Use a build tool such as Maven or Gradle to run this code example. Add the required dependencies, including Gson, OkHttp, RxJava, Apache Log4j, Lombok, and JUnit, to the corresponding configuration file (pom.xml for Maven or build.gradle for Gradle).
Configuration
The configuration may vary depending on the version of Maven or Gradle you use. The following content is for reference only. Adjust it according to your actual environment.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.nls</groupId>
<artifactId>bailian-demo</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>dashscope-sdk-java</artifactId>
<version>2.20.5</version>
</dependency>
<dependency>
<groupId>io.reactivex.rxjava2</groupId>
<artifactId>rxjava</artifactId>
<version>2.2.21</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.26</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.10.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.10.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit-pioneer</groupId>
<artifactId>junit-pioneer</artifactId>
<version>1.9.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.javalin</groupId>
<artifactId>javalin</artifactId>
<version>4.4.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.9</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>2.0.7</version>
</dependency>
<!-- dependent by okhttp -->
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.8.21</version>
</dependency>
<!-- dependent by okhttp -->
<dependency>
<groupId>com.squareup.okio</groupId>
<artifactId>okio</artifactId>
<version>3.6.0</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>mockwebserver</artifactId>
<version>4.12.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>logging-interceptor</artifactId>
<version>4.12.0</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp-sse</artifactId>
<version>4.12.0</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.12.0</version>
</dependency>
<!-- function call support -->
<dependency>
<groupId>com.github.victools</groupId>
<artifactId>jsonschema-generator</artifactId>
<version>4.31.1</version>
</dependency>
</dependencies>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
</project>
plugins {
id 'java'
}
group = 'com.test'
version = '1.0-SNAPSHOT'
repositories {
mavenCentral()
maven {
url 'https://maven.aliyun.com/repository/public'
}
maven {
url 'https://plugins.gradle.org/m2/'
}
}
dependencies {
// Dashscope SDK
implementation 'com.alibaba:dashscope-sdk-java:2.20.5'
// 第三方库
// https://mvnrepository.com/artifact/com.google.code.gson/gson
implementation 'com.google.code.gson:gson:2.13.1'
// https://mvnrepository.com/artifact/io.reactivex.rxjava2/rxjava
implementation 'io.reactivex.rxjava2:rxjava:2.2.21'
// https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp
implementation 'com.squareup.okhttp3:okhttp:4.12.0' // 核心 OkHttp
implementation 'com.squareup.okhttp3:logging-interceptor:4.12.0' // 日志拦截器
// 日志相关依赖
implementation 'org.apache.logging.log4j:log4j-core:2.17.1'
implementation 'org.apache.logging.log4j:log4j-api:2.17.1'
// 如果你使用 SLF4J 门面
implementation 'org.apache.logging.log4j:log4j-slf4j-impl:2.17.1'
// 测试相关依赖
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.10.0'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.10.0'
testImplementation 'org.apache.logging.log4j:log4j-core:2.17.1'
testImplementation 'org.apache.logging.log4j:log4j-slf4j-impl:2.17.1'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.30'
// Lombok 支持
implementation 'org.projectlombok:lombok:1.18.30'
annotationProcessor 'org.projectlombok:lombok:1.18.30'
}
test {
useJUnitPlatform()
}
Java code
package org.alibaba.speech.examples.speech_plus;
import com.alibaba.dashscope.multimodal.MultiModalDialog;
import com.alibaba.dashscope.multimodal.MultiModalDialogCallback;
import com.alibaba.dashscope.multimodal.MultiModalRequestParam;
import com.alibaba.dashscope.multimodal.State;
import com.alibaba.dashscope.utils.Constants;
import com.alibaba.dashscope.utils.JsonUtils;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import org.alibaba.speech.examples.speech_plus.utils.FileWriterUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import static java.lang.Thread.sleep;
/**
* Multi-modal dialog test cases for demonstrating various interaction modes
* including push-to-talk, tap-to-talk, duplex communication, text synthesis,
* visual Q&A, and agent DJ functionality.
*
* @author songsong.shao
* @date 2025/4/28
*/
public class MultiModalDialogTestCases {
private static final Logger log = LoggerFactory.getLogger(MultiModalDialogTestCases.class);
// Constants for audio processing
private static final int AUDIO_CHUNK_SIZE = 3200; // Audio chunk size in bytes
private static final int SLEEP_INTERVAL_MS = 100; // Sleep interval in milliseconds
private static final int WAIT_TIMEOUT_MS = 2000; // Wait timeout in milliseconds
private static final int VIDEO_FRAME_INTERVAL_MS = 500;
// State management variables
private static State.DialogState currentState;
private static MultiModalDialog conversation;
private static int enterListeningTimes = 0;
private static FileWriterUtil fileWriterUtil;
private static boolean vqaUseUrl = false;
private volatile boolean isVideoStreamingActive = false;
private Thread videoStreamingThread;
// Configuration parameters - should be set before running tests
public static String workspaceId = "";
public static String appId = "";
public static String dialogId = "";
public static String apiKey = "";
public static String model = "";
/**
* Test push-to-talk mode interaction
* Flow: Set push2talk -> Wait for listening -> Start speech -> Send audio -> Stop speech -> Wait for response
*/
public void testMultimodalPush2Talk() {
log.info("############ Starting Push2Talk Test ############");
try {
// Build request parameters for push-to-talk mode
MultiModalRequestParam params = buildBaseRequestParams("push2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation with callback
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for the system to enter listening state
waitForListeningState();
// Start speech recognition
conversation.startSpeech();
// Send audio data from file
sendAudioFromFile("../../../../sample-data/1_plus_1.wav");
// Stop speech recognition
conversation.stopSpeech();
// Wait for conversation completion
waitForConversationCompletion(2);
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in push2talk test: ", e);
} finally {
log.info("############ Push2Talk Test Completed ############");
}
}
/**
* Test tap-to-talk mode interaction
* Flow: Set tap2talk -> Wait for listening -> Start speech manually -> Send audio -> Wait for response
*/
public void testMultimodalTap2Talk() {
log.info("############ Starting Tap2Talk Test ############");
try {
// Build request parameters for tap-to-talk mode
MultiModalRequestParam params = buildBaseRequestParams("tap2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Manually start speech (required for tap2talk mode)
conversation.startSpeech();
// Send audio data
sendAudioFromFile("../../../../sample-data/1_plus_1.wav");
// Wait for conversation completion
waitForConversationCompletion(2);
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in tap2talk test: ", e);
} finally {
log.info("############ Tap2Talk Test Completed ############");
}
}
/**
* Test duplex mode interaction
* Flow: Set duplex -> Wait for listening -> Send audio -> Wait for response
*/
public void testMultimodalDuplex() {
log.info("############ Starting Duplex Test ############");
try {
// Build request parameters for duplex mode
MultiModalRequestParam params = buildBaseRequestParams("duplex");
log.info("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Send audio data directly (no manual start needed for duplex)
sendAudioFromFile("../../../../sample-data/1_plus_1.wav");
// Wait for conversation completion
waitForConversationCompletion(2);
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in duplex test: ", e);
} finally {
log.info("############ Duplex Test Completed ############");
}
}
/**
* Test text synthesis functionality
* Flow: Send text for TTS -> Save synthesized audio
*/
public void testMultimodalTextSynthesizer() {
log.info("############ Starting Text Synthesizer Test ############");
try {
// Initialize file writer for audio output
fileWriterUtil = new FileWriterUtil();
// Build request parameters
MultiModalRequestParam params = buildBaseRequestParams("push2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Create output file for synthesized audio
fileWriterUtil.createFile("./test.pcm");
// Request text synthesis
String textToSynthesize = "幸福是一种技能,是你摒弃了外在多余欲望后的内心平和。";
conversation.requestToRespond("transcript", textToSynthesize, null);
// Wait for synthesis completion
waitForConversationCompletion(2);
// Finalize audio file
fileWriterUtil.finishWriting();
// Clean up
conversation.stop();
} catch (IOException e) {
log.error("File operation error in text synthesizer test: ", e);
} catch (Exception e) {
log.error("Error in text synthesizer test: ", e);
} finally {
log.info("############ Text Synthesizer Test Completed ############");
}
}
/**
* Test Visual Q&A functionality using URL-based images
* Flow: Send VQA request -> Receive visual_qa command -> Send image list -> Get response
*/
public void testMultimodalVQA() {
log.info("############ Starting VQA Test (URL-based) ############");
try {
vqaUseUrl = true;
// Build request parameters
MultiModalRequestParam params = buildBaseRequestParams("push2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Send VQA request
conversation.requestToRespond("prompt", "拍照看看前面有什么东西", null);
// Wait for VQA processing completion
waitForConversationCompletion(3);
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in VQA test: ", e);
} finally {
log.info("############ VQA Test Completed ############");
}
}
/**
* Test Visual Q&A functionality using Base64-encoded images
* Flow: Send VQA request -> Receive visual_qa command -> Send base64 image -> Get response
*/
public void testMultimodalVQABase64() {
log.info("############ Starting VQA Test (Base64-based) ############");
try {
vqaUseUrl = false;
// Build request parameters
MultiModalRequestParam params = buildBaseRequestParams("push2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Send VQA request
conversation.requestToRespond("prompt", "拍照看看前面有什么东西", null);
// Wait for VQA processing completion
waitForConversationCompletion(3);
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in VQA Base64 test: ", e);
} finally {
log.info("############ VQA Base64 Test Completed ############");
}
}
/**
* Test brightness adjustment functionality
* Flow: Send brightness adjustment request -> Process response
*/
public void testMultimodalAdjustBrightness() {
log.info("############ Starting Brightness Adjustment Test ############");
try {
vqaUseUrl = true;
// Build request parameters
MultiModalRequestParam params = buildBaseRequestParams("push2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Send brightness adjustment request
conversation.requestToRespond("prompt", "调高亮度到10", null);
// Create status update for bluetooth announcement
JsonObject status = new JsonObject();
JsonObject bluetoothAnnouncement = new JsonObject();
bluetoothAnnouncement.addProperty("status", "stopped");
status.add("bluetooth_announcement", bluetoothAnnouncement);
MultiModalRequestParam.UpdateParams updateParams = MultiModalRequestParam.UpdateParams.builder()
.clientInfo(MultiModalRequestParam.ClientInfo.builder()
.status(status)
.build())
.build();
// Wait for processing completion
waitForConversationCompletion(2);
log.info("############ Before stopping conversation ############");
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in brightness adjustment test: ", e);
} finally {
log.info("############ Brightness Adjustment Test Completed ############");
}
}
/**
* Test Agent DJ functionality
* Flow: Send radio station request -> Process response
*/
public void testMultimodalAgentDJ() {
log.info("############ Starting Agent DJ Test ############");
try {
vqaUseUrl = false;
// Build request parameters
MultiModalRequestParam params = buildBaseRequestParams("push2talk");
log.debug("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Wait for listening state
waitForListeningState();
// Send DJ request
conversation.requestToRespond("prompt", "打开新闻电台", null);
// Wait for DJ processing completion
waitForConversationCompletion(3);
// Clean up
stopConversation();
} catch (Exception e) {
log.error("Error in Agent DJ test: ", e);
} finally {
log.info("############ Agent DJ Test Completed ############");
}
}
/**
* Test LiveAI functionality with real-time video streaming and audio interaction
* Flow: Set duplex mode -> Start video streaming -> Wait for listening -> Connect video channel
* -> Send audio queries -> Process responses with visual context
*/
public void testMultimodalLiveAI() {
log.info("############ Starting LiveAI Test ############");
try {
// Build request parameters for duplex mode
MultiModalRequestParam params = buildBaseRequestParams("duplex");
log.info("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Start send video frame loop
startVideoFrameStreaming();
// Wait for listening state
waitForListeningState();
conversation.sendHeartBeat();
// Send video channel connect request, will response command : switch_video_call_success
conversation.requestToRespond("prompt", "", connectVideoChannelRequest());
// Send audio data directly (no manual start needed for duplex)
sendAudioFromFile("./src/main/resources/what_in_picture.wav");
// Wait for listening state twice
waitForListeningState();
sendAudioFromFile("./src/main/resources/what_color.wav");
// Wait for conversation completion
waitForConversationCompletion(3);
// Clean up
stopConversation();
isVideoStreamingActive = false;
if (videoStreamingThread != null && videoStreamingThread.isAlive()) {
videoStreamingThread.interrupt();
}
} catch (Exception e) {
log.error("Error in LiveAI test: ", e);
} finally {
log.info("############ LiveAI Test Completed ############");
}
}
// ==================== Helper Methods ====================
/**
* Build base request parameters with common configuration
* @param mode The interaction mode (push2talk, tap2talk, duplex)
* @return Configured MultiModalRequestParam
*/
private MultiModalRequestParam buildBaseRequestParams(String mode) {
return MultiModalRequestParam.builder()
.customInput(
MultiModalRequestParam.CustomInput.builder()
.workspaceId(workspaceId)
.appId(appId)
.build())
.upStream(
MultiModalRequestParam.UpStream.builder()
.mode(mode)
.audioFormat("pcm")
.build())
.downStream(
MultiModalRequestParam.DownStream.builder()
.voice("longxiaochun_v2")
.sampleRate(48000)
.build())
.clientInfo(
MultiModalRequestParam.ClientInfo.builder()
.userId("1234")
.device(MultiModalRequestParam.ClientInfo.Device.builder()
.uuid("1234")
.build())
.build())
.apiKey(apiKey)
.model(model)
.build();
}
/**
* Build video channel connection request for LiveAI
* This establishes the video streaming channel that enables visual context for AI responses
*
* @return UpdateParams containing video channel connection configuration
*/
private MultiModalRequestParam.UpdateParams connectVideoChannelRequest(){
Map<String, String> video = new HashMap<>();
video.put("action", "connect");
video.put("type", "voicechat_video_channel");
ArrayList<Map<String, String>> videos = new ArrayList<>();
videos.add(video);
MultiModalRequestParam.BizParams bizParams = MultiModalRequestParam.BizParams.builder().videos(videos).build();
return MultiModalRequestParam.UpdateParams.builder().bizParams(bizParams).build();
}
/**
* Wait for the system to enter listening state
*/
private void waitForListeningState() {
while (currentState != State.DialogState.LISTENING) {
try {
Thread.sleep(SLEEP_INTERVAL_MS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Interrupted while waiting for listening state", e);
}
}
log.info("System entered listening state");
}
/**
* Send audio data from a file to the conversation
* @param filePath Path to the audio file
*/
private void sendAudioFromFile(String filePath) {
File audioFile = new File(filePath);
if (!audioFile.exists()) {
log.error("Audio file not found: {}", filePath);
return;
}
try (AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(audioFile)) {
byte[] audioBuffer = new byte[AUDIO_CHUNK_SIZE];
int bytesRead;
int totalBytesRead = 0;
log.info("Starting to send audio data from: {}", filePath);
// Read and send audio data in chunks
while ((bytesRead = audioInputStream.read(audioBuffer)) != -1) {
totalBytesRead += bytesRead;
// Send audio chunk to conversation
conversation.sendAudioData(ByteBuffer.wrap(audioBuffer, 0, bytesRead));
// Add small delay to simulate real-time audio streaming
Thread.sleep(SLEEP_INTERVAL_MS);
}
log.info("Finished sending audio data. Total bytes sent: {}", totalBytesRead);
} catch (Exception e) {
log.error("Error sending audio from file: {}", filePath, e);
}
}
/**
* Wait for conversation completion
* @param expectedListeningTimes Expected number of listening state entries
*/
private void waitForConversationCompletion(int expectedListeningTimes) {
while (enterListeningTimes < expectedListeningTimes) {
try {
Thread.sleep(WAIT_TIMEOUT_MS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Interrupted while waiting for conversation completion", e);
}
}
log.info("Conversation completed after {} listening cycles", expectedListeningTimes);
}
/**
* Stop the conversation and clean up resources
*/
private void stopConversation() {
try {
if (conversation != null) {
conversation.stop();
Thread.sleep(1000); // Allow time for cleanup
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
log.warn("Interrupted while stopping conversation");
}
}
/**
* Get the callback implementation for handling conversation events
* @return MultiModalDialogCallback instance
*/
public static MultiModalDialogCallback getCallback() {
return new MultiModalDialogCallbackImpl();
}
/**
* Implementation of MultiModalDialogCallback for handling various conversation events
*/
public static class MultiModalDialogCallbackImpl extends MultiModalDialogCallback {
@Override
public void onConnected() {
log.info("WebSocket connection established");
}
@Override
public void onStarted(String dialogId) {
log.info("Dialog started with ID: {}", dialogId);
}
@Override
public void onStopped(String dialogId) {
log.info("Dialog stopped with ID: {}", dialogId);
}
@Override
public void onSpeechStarted(String dialogId) {
log.info("Speech recognition started for dialog: {}", dialogId);
}
@Override
public void onSpeechEnded(String dialogId) {
log.info("Speech recognition ended for dialog: {}", dialogId);
}
@Override
public void onError(String dialogId, String errorCode, String errorMsg) {
log.error("Error occurred - Dialog: {}, Code: {}, Message: {}", dialogId, errorCode, errorMsg);
enterListeningTimes++; // Force quit dialog test on error
}
@Override
public void onStateChanged(State.DialogState state) {
log.info("Dialog state changed to: {}", state);
currentState = state;
if (currentState == State.DialogState.LISTENING) {
enterListeningTimes++;
log.info("Entered listening state {} times", enterListeningTimes);
}
}
@Override
public void onSpeechAudioData(ByteBuffer audioData) {
try {
// Write audio data to file if file writer is available
if (fileWriterUtil != null) {
fileWriterUtil.Writing(audioData);
}
} catch (IOException e) {
log.error("Error writing audio data to file: ", e);
throw new RuntimeException("Failed to write audio data", e);
}
}
@Override
public void onRespondingStarted(String dialogId) {
log.info("Response generation started for dialog: {}", dialogId);
if (conversation != null) {
conversation.localRespondingStarted();
}
}
@Override
public void onRespondingEnded(String dialogId, JsonObject jsonObject) {
log.info("Response generation ended for dialog: {}", dialogId);
if (conversation != null) {
conversation.localRespondingEnded();
}
}
@Override
public void onRespondingContent(String dialogId, JsonObject content) {
log.info("Response content received - Dialog: {}, Content: {}", dialogId, content);
// Handle visual Q&A commands
handleVisualQACommands(content);
}
@Override
public void onSpeechContent(String dialogId, JsonObject content) {
log.info("Speech content received - Dialog: {}, Content: {}", dialogId, content);
}
@Override
public void onRequestAccepted(String dialogId) {
log.info("Request accepted for dialog: {}", dialogId);
}
@Override
public void onClosed() {
log.info("Connection closed");
enterListeningTimes++; // Increment to trigger test completion
}
/**
* Handle visual Q&A commands from response content
* @param content Response content containing potential commands
*/
private void handleVisualQACommands(JsonObject content) {
if (!content.has("extra_info")) {
return;
}
JsonObject extraInfo = content.getAsJsonObject("extra_info");
if (!extraInfo.has("commands")) {
return;
}
try {
String commandsStr = extraInfo.get("commands").getAsString();
log.info("Processing commands: {}", commandsStr);
JsonArray commands = new Gson().fromJson(commandsStr, JsonArray.class);
for (JsonElement command : commands) {
JsonObject commandObj = command.getAsJsonObject();
if (commandObj.has("name")) {
String commandName = commandObj.get("name").getAsString();
if ("visual_qa".equals(commandName)) {
log.info("Visual Q&A command detected - triggering image capture");
// Send mock image data for visual Q&A
MultiModalRequestParam.UpdateParams updateParams =
MultiModalRequestParam.UpdateParams.builder()
.images(getMockImageRequest())
.build();
if (conversation != null) {
conversation.requestToRespond("prompt", "", updateParams);
}
}
}
}
} catch (Exception e) {
log.error("Error processing visual Q&A commands: ", e);
}
}
}
/**
* Create mock image data for testing purposes
* @return List of image objects (URL or Base64 based on vqaUseUrl flag)
*/
public static List<Object> getMockImageRequest() {
List<Object> images = new ArrayList<>();
try {
JsonObject imageObject = new JsonObject();
if (vqaUseUrl) {
// Use URL-based image
imageObject.addProperty("type", "url");
imageObject.addProperty("value", "https://help-static-aliyun-doc.aliyuncs.com/assets/img/zh-CN/7043267371/p909896.png");
imageObject.addProperty("bucket", "bucketName");
imageObject.add("extra", new JsonObject());
} else {
// Use Base64-encoded image
imageObject.addProperty("type", "base64");
imageObject.addProperty("value", getLocalImageBase64());
}
images.add(imageObject);
// log.info("Created mock image data with type: {}", vqaUseUrl ? "URL" : "Base64");
} catch (Exception e) {
log.error("Error creating mock image data: ", e);
}
return images;
}
/**
* Convert local image file to Base64 string
* @return Base64-encoded image string
*/
public static String getLocalImageBase64() {
String imagePath = "./src/main/resources/jpeg-bridge.jpg";
try (FileInputStream fileInputStream = new FileInputStream(new File(imagePath))) {
byte[] imageBytes = new byte[fileInputStream.available()];
fileInputStream.read(imageBytes);
String base64Image = Base64.getEncoder().encodeToString(imageBytes);
log.info("Successfully converted image to Base64, size: {} bytes", imageBytes.length);
return base64Image;
} catch (IOException e) {
log.error("Error converting image to Base64: {}", imagePath, e);
return null;
}
}
/**
* Start continuous video frame streaming for LiveAI
*/
private void startVideoFrameStreaming() {
log.info("Starting continuous video frame streaming for LiveAI...");
vqaUseUrl = false;
isVideoStreamingActive = true;
videoStreamingThread = new Thread(() -> {
try {
while (isVideoStreamingActive && !Thread.currentThread().isInterrupted()) {
Thread.sleep(VIDEO_FRAME_INTERVAL_MS);
MultiModalRequestParam.UpdateParams videoUpdate =
MultiModalRequestParam.UpdateParams.builder()
.images(getMockImageRequest())
.build();
if (conversation != null && isVideoStreamingActive) {
conversation.updateInfo(videoUpdate);
log.debug("Video frame sent to LiveAI");
}
}
} catch (InterruptedException e) {
log.info("Video streaming thread interrupted - stopping video stream");
Thread.currentThread().interrupt();
} catch (Exception e) {
log.error("Error in video streaming thread: ", e);
} finally {
log.info("Video streaming thread terminated");
}
});
videoStreamingThread.setDaemon(true);
videoStreamingThread.setName("LiveAI-VideoStreaming");
videoStreamingThread.start();
log.info("Video streaming thread started successfully");
}
/**
* Main method to run the test cases
* Configure the required parameters before running
*/
public static void main(String[] args) {
log.info("############ Initializing Multi-modal Dialog Tests ############");
// Configure WebSocket API URL
Constants.baseWebsocketApiUrl = "wss://dashscope.aliyuncs.com/api-ws/v1/inference";
log.info("WebSocket API URL: {}", Constants.baseWebsocketApiUrl);
// Reset state variables
enterListeningTimes = 0;
// Configure test parameters (replace with actual values)
MultiModalDialogTestCases.workspaceId = "";
MultiModalDialogTestCases.appId = "";
MultiModalDialogTestCases.apiKey = "";
MultiModalDialogTestCases.model = "multimodal-dialog";
// Validate configuration
if (apiKey.isEmpty() || workspaceId.isEmpty() || appId.isEmpty()) {
log.error("Please configure workspaceId, appId, and apiKey before running tests");
return;
}
// Create test instance and run specific test
MultiModalDialogTestCases testCases = new MultiModalDialogTestCases();
try {
// Run the desired test case
testCases.testMultimodalPush2Talk();
// Uncomment other test cases as needed:
// testCases.testMultimodalTap2Talk();
// testCases.testMultimodalDuplex();
// testCases.testMultimodalTextSynthesizer();
// testCases.testMultimodalVQA();
// testCases.testMultimodalVQABase64();
// testCases.testMultimodalAdjustBrightness();
// testCases.testMultimodalAgentDJ();
// testCases.testMultimodalLiveAI();
} catch (Exception e) {
log.error("Error running test cases: ", e);
}
log.info("############ Multi-modal Dialog Tests Completed ############");
}
}
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
/**
* 文件写入工具类。
* 支持创建文件、开始写入数据、追加写入数据和结束写入操作。
* 使用 ByteBuffer 存储数据并将数据写入指定的文件中。
*/
public class FileWriterUtil {
private File file;
private FileOutputStream fos;
private FileChannel fc;
/**
* 创建一个新的文件。
*
* @param filePath 文件路径
*/
public void createFile(String filePath) throws IOException {
file = new File(filePath);
if (!file.exists()) {
file.createNewFile();
}
fos = new FileOutputStream(file, true);
fc = fos.getChannel();
}
/**
* 写入数据。
*
* @param data 要追加的数据
*/
public void Writing(ByteBuffer data) throws IOException {
fc.position(fc.size());
fc.write(data);
}
/**
* 结束写入操作。
*/
public void finishWriting() throws IOException {
if (fc != null) {
fc.close();
}
if (fos != null) {
fos.close();
}
}
}
Advanced SDK usage
VQA interaction
VQA (Visual Question Answering) combines images and voice for multimodal interaction, allowing you to send images during a conversation.
A voice or text request with a photo-taking intent triggers the "visual_qa" command.
When you receive this command, send an image as either a URL or a base64-encoded string. The image size must be less than 180 KB.
- Initiate a photo request after establishing a connection.
@Test
void testMultimodalVQA() {
/*
step1. 发送”看看前面有什么东西“,onRespondingContent 返回visual_qa 指令
step2. 发送图片列表
step3. 返回图片的对话结果
*/
System.out.println("############ Start Test VQA ############");
vqaUseUrl = true;
log.debug("params: {}", JsonUtils.toJson(params));
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
while (currentState != State.DialogState.LISTENING) {
try {
sleep(100);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
conversation.requestToRespond("prompt","拍照看看前面有什么东西",null);
// 增加交互流程等待
while (enterListeningTimes < 3) {
try {
sleep(2000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
conversation.stop();
try {
sleep(1000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
System.out.println("############ End Test Push2Talk ############");
}
- Handle the "visual_qa" command and upload the photo.
@Override
public void onRespondingContent(String dialogId, JsonObject content) {
log.info("onRespondingContent: {}, {}", dialogId, content);
if (content.has("extra_info")) {
JsonObject extraInfo = content.getAsJsonObject("extra_info");
if (extraInfo.has("commands")) {
String commandsStr = extraInfo.get("commands").getAsString();
log.info("commandsStr: {}", commandsStr);
//"[{\"name\":\"visual_qa\",\"params\":[{\"name\":\"shot\",\"value\":\"拍照看看\",\"normValue\":\"True\"}]}]"
JsonArray commands = new Gson().fromJson(commandsStr, JsonArray.class);
for (JsonElement command : commands) {
JsonObject commandObj = command.getAsJsonObject();
if (commandObj.has("name")) {
String commandStr = commandObj.get("name").getAsString();
if (commandStr.equals("visual_qa")) {
log.info("拍照了!!!!");
MultiModalRequestParam.UpdateParams updateParams = MultiModalRequestParam.UpdateParams.builder()
.images(getMockOSSImage())
.build();
conversation.requestToRespond("prompt","",updateParams);
}
}
}
}
}
}
//上传拍照
public static List<Object> getMockOSSImage() {
JsonObject imageObject = new JsonObject();
JsonObject extraObject = new JsonObject();
List<Object> images = new ArrayList<>();
try{
if (vqaUseUrl){
imageObject.addProperty("type", "url");
imageObject.addProperty("value", "https://help-static-aliyun-doc.aliyuncs.com/assets/img/zh-CN/7043267371/p909896.png");
imageObject.addProperty("bucket", "bucketName");
imageObject.add("extra", extraObject);
}else {
imageObject.addProperty("type", "base64");
imageObject.addProperty("value", getLocalImageBase64());
}
images.add(imageObject);
}catch (Exception e){
e.printStackTrace();
}
return images;
}
TTS synthesis
The SDK supports server-side audio synthesis directly from text.
You need to send a requestToRespond request when the client is in the Listening state.
If the current state is not Listening, you must first call the Interrupt API to interrupt the current playback.
@Test
void testMultimodalTextSynthesizer() {
/*
step1. 发送文本合成tts
step2. 保存tts音频
*/
System.out.println("############ Start Test TextSynthesizer ############");
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
while (currentState != State.DialogState.LISTENING) {
try {
sleep(100);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
try {
fileWriterUtil.createFile("./test.pcm");
} catch (IOException e) {
log.error(e.getMessage());
throw new RuntimeException(e);
}
conversation.requestToRespond("transcript","幸福是一种技能,是你摒弃了外在多余欲望后的内心平和。",null);
// 增加交互流程等待
while (enterListeningTimes < 2) {
try {
sleep(2000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
try {
fileWriterUtil.finishWriting();
} catch (IOException e) {
log.error(e.getMessage());
throw new RuntimeException(e);
}
conversation.stop();
System.out.println("############ End Test TextSynthesizer ############");
}
Request LiveAI via WebSocket
LiveAI, an official agent from Model Studio for multimodal interaction, enables video calls. Implement video calls by sending a sequence of images through the Java SDK. We recommend using Real-Time Communication (RTC) to transmit video and audio between your server and client (web page or app). Then, send the video frames captured by the server to the SDK at a rate of one frame every 500 ms, while maintaining a real-time audio input stream.
Note: When sending images to LiveAI, only base64 encoding is supported, and each image must be smaller than 180 KB.
- LiveAI call sequence
- Key code examples
For the complete code, see the GitHub sample code.
public void testMultimodalLiveAI() {
log.info("############ Starting LiveAI Test ############");
try {
// Build request parameters for duplex mode
MultiModalRequestParam params = buildBaseRequestParams("duplex");
log.info("Request parameters: {}", JsonUtils.toJson(params));
// Initialize conversation
conversation = new MultiModalDialog(params, getCallback());
conversation.start();
// Start send video frame loop
startVideoFrameStreaming();
// Wait for listening state
waitForListeningState();
conversation.sendHeartBeat();
// Send video channel connect request, will response command : switch_video_call_success
conversation.requestToRespond("prompt", "", connectVideoChannelRequest());
// Send audio data directly (no manual start needed for duplex)
sendAudioFromFile("./src/main/resources/what_in_picture.wav");
// Wait for listening state twice
waitForListeningState();
sendAudioFromFile("./src/main/resources/what_color.wav");
// Wait for conversation completion
waitForConversationCompletion(3);
// Clean up
stopConversation();
isVideoStreamingActive = false;
if (videoStreamingThread != null && videoStreamingThread.isAlive()) {
videoStreamingThread.interrupt();
}
} catch (Exception e) {
log.error("Error in LiveAI test: ", e);
} finally {
log.info("############ LiveAI Test Completed ############");
}
}
/**
* Build video channel connection request for LiveAI
*
* @return UpdateParams containing video channel connection configuration
*/
private MultiModalRequestParam.UpdateParams connectVideoChannelRequest(){
Map<String, String> video = new HashMap<>();
video.put("action", "connect");
video.put("type", "voicechat_video_channel");
ArrayList<Map<String, String>> videos = new ArrayList<>();
videos.add(video);
MultiModalRequestParam.BizParams bizParams = MultiModalRequestParam.BizParams.builder().videos(videos).build();
return MultiModalRequestParam.UpdateParams.builder().bizParams(bizParams).build();
}
/**
* Start continuous video frame streaming for LiveAI
*/
private void startVideoFrameStreaming() {
log.info("Starting continuous video frame streaming for LiveAI...");
vqaUseUrl = false;
isVideoStreamingActive = true;
videoStreamingThread = new Thread(() -> {
try {
while (isVideoStreamingActive && !Thread.currentThread().isInterrupted()) {
Thread.sleep(VIDEO_FRAME_INTERVAL_MS);
MultiModalRequestParam.UpdateParams videoUpdate =
MultiModalRequestParam.UpdateParams.builder()
.images(getMockImageRequest())
.build();
if (conversation != null && isVideoStreamingActive) {
conversation.updateInfo(videoUpdate);
log.debug("Video frame sent to LiveAI");
}
}
} catch (InterruptedException e) {
log.info("Video streaming thread interrupted - stopping video stream");
Thread.currentThread().interrupt();
} catch (Exception e) {
log.error("Error in video streaming thread: ", e);
} finally {
log.info("Video streaming thread terminated");
}
});
videoStreamingThread.setDaemon(true);
videoStreamingThread.setName("LiveAI-VideoStreaming");
videoStreamingThread.start();
log.info("Video streaming thread started successfully");
}
Use custom prompt variables
- In the console, navigate to your project's Prompt settings to configure custom variables.
The following example defines a user_name field that represents a user nickname. The user_name variable is then inserted into the Prompt as the placeholder ${user_name}.
The Custom Variable button is located in the Available Variables area at the top of the prompt configuration page.
- Set variables in your code.
For example, set "user_name" = "大米".
//在请求参数构建中传入biz_params.user_prompt_params
MultiModalRequestParam.builder()
.bizParams(
MultiModalRequestParam.BizParams.builder()
.userPromptParams(setUserPromptParams())
.build())
.apiKey(apiKey)
.model(model)
.build();
private HashMap<String, String> setUserPromptParams() {
HashMap<String, String> promptSettings = new HashMap<>();
promptSettings.put("user_name", "大米");
return promptSettings;
}
- Request a response.
The agent uses Dear Dami to address the user in the response, for example: "Dear Dami, what is the weather like where you are today?", which indicates that the user_name variable was successfully passed and took effect in the Prompt.
Real-time ASR correction
During a conversation, ASR (Automatic Speech Recognition) results may contain errors or unexpected outputs. In addition to configuring hotwords, you can use the real-time correction feature to upload a word list and fix these errors immediately.
- Parameter description
Configure the correction word list by setting the AsrPostProcessing parameter in UpStream.
Parameter | Level 1 parameter | Level 2 parameter | Type | Description |
|---|---|---|---|---|
AsrPostProcessing | Object | The ASR correction word list. | ||
ReplaceWord[] | List | A list of replacement rules. Each ReplaceWord object defines a rule for a word or phrase. | ||
source | String | The text to be replaced. | ||
target | String | The replacement text. | ||
match_mode | String | The match mode. Default: exact. exact: Exact match. The entire recognized text must be identical to the source value. partial: Partial match. A match occurs if the source value is found as a substring within the recognized text. |
NoteTo use this feature, your Java SDK version must be 2.21.14 or later.
//构建AsrPostProcessing对象
MultiModalRequestParam.UpStream.AsrPostProcessing asrPostProcessing =
MultiModalRequestParam.UpStream.AsrPostProcessing
.builder()
.replaceWords(Collections.singletonList(
ReplaceWord.builder()
.source("1加1")
.target("一加一")
.matchMode("partial")
.build()
))
.build();
// 在创建对话请求时将asrPostProcessing 传入UpStream.AsrPostProcessing
More examples
For more examples, see the GitHub sample code.