HTTP protocol
This topic describes the multimodal interaction API based on the HTTP protocol.
The HTTP protocol in the multimodal interaction development suite supports sending http-sse requests with plain text or images to retrieve results from the large model.
ImportantWhen you use the HTTP protocol to access the multimodal interaction development suite, the following features are not supported:
-
Pure vision applications are not supported.
-
The following agents are not supported:
Video Call, Express Video/Voice Call, Voice Translation, News Radio, Children's Stories, Audio Recording Summary, Proactive Tour Guide, and Proactive Companion.
Prerequisites
You must activate the service and obtain and configure an API key. To prevent security risks, configure the API key as an environment variable instead of hard coding it in your code.
NoteFor client-side calls, handling the API key on the client poses a security risk. You can obtain a temporary authentication token from the server-side using your API key, and then send the token to the client for use. For more information, see Generate a temporary API key.
Endpoint
https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation
NoteWhen you send an HTTP request, the header must include X-DashScope-SSE: enable.
Authentication
When you send an HTTP request, include the API key in the HTTP header. Replace your_api_key with your actual API key.
"Authorization": "Bearer your_api_key"
Message types
The HTTP protocol supports only text messages, which are JSON-formatted strings.
Send a request
Request - Input Message
You can send an HTTP request.
Top-level parameter | Second-level parameter | Type | Required | Description |
model | string | Yes | The name of the Alibaba Cloud Model Studio model. This is fixed to "multimodal-dialog". Copy and use this value directly. | |
input | directive | string | Yes | Instruction name: Request |
app_id | string | Yes | The ID of the application created by the customer (Get APP ID). You can view this ID on the "My Applications" page of the multimodal interaction development suite console. | |
dialog_id | string | No | The dialog ID. If you leave this empty, a new dialog is started. The server automatically generates a dialog ID and returns it in the response. Example format: "12345678-1234-1234-1234-1234567890ab" (36 characters). To continue a previous conversation, pass the dialog_id that the server returned. | |
text | string | Yes | The text to process.
| |
parameters | client_info | object | Yes | For parameter descriptions, see the parameters.client_info parameters table below. |
history | list[] | No | The Q&A history organized by the customer. It overrides the default Q&A history maintained by the service and affects all subsequent Q&A. | |
images | list[] | No | The image data to analyze. Only multimodal applications support image Q&A. For parameter descriptions, see the parameters.images parameters table below. | |
biz_params | object | No | Configure as needed. For parameter descriptions, see the parameters.biz_params parameters table below. |
The following table describes the parameters for parameters.client_info.
Top-level parameter | Second-level parameter | Type | Required | Description |
|---|---|---|---|---|
user_id | string | Yes | The end user ID. Generate this ID based on your business rules to implement customized features for different end users. The maximum length is 36 characters. | |
device | uuid | string | No | A globally unique ID for the client. You must generate this ID and pass it to the software development kit (SDK). The maximum length is 40 characters. An end user can have multiple devices. Each device has a different UUID, but the user_id is the same. |
network | ip | string | No | The public IP address of the caller. |
location | latitude | string | No | The latitude of the caller. Submit this in scenarios that require the precise location of the client. |
longitude | string | No | The longitude of the caller. Submit this in scenarios that require the precise location of the client. | |
city_name | string | No | The city where the caller is located. This indicates the approximate location of the client. |
The following table describes the parameters for each record in parameters.history.
Top-level parameter | Type | Required | Description |
|---|---|---|---|
role | string | Yes | The role of the speaker of this record. Only user and assistant are supported. Note: system is not supported. |
content | string | Yes | The specific text content. |
The following table describes the parameters for parameters.images.
Top-level parameter | Type | Required | Description |
|---|---|---|---|
type | string | Yes | The image type. Supported types are base64 and url. |
value | string | Yes | The image content.
|
The following table describes the parameters for parameters.biz_params.
Top-level parameter | Type | Required | Description |
|---|---|---|---|
user_defined_params | json object | No | Parameters to pass through to the agent. For information about the parameters passed by different agents, see Call official agents. |
user_prompt_params | json object | No | Used to set custom prompt variables. You can define the key and value in the JSON object. For information about how to configure custom prompt variables in the console, see Application configuration - Prompts. |
user_query_params | json object | No | Used to set custom dialog variables. You can define the key and value in the JSON object. For information about how to configure custom dialog variables in the console, see Application configuration - Dialog variables. |
Example
{
"model": "multimodal-dialog",
"input": {
"directive": "Request",
"app_id": "****************",
"text":"Hello, what is the weather like in Beijing today?"
},
"parameters": {
"client_info": {
"user_id": "bi**********07",
"device":{
"uuid": "432k*********k449"
},
"network":{
"ip": "x.x.x.x"
},
"location":{
"city_name": "Beijing"
}
},
"images":[{
"type": "base64",
"value": "base64String"
}]
}
}
Text event
RespondingContent - Output Message
Top-level parameter | Second-level parameter | Type | Required | Description |
|---|---|---|---|---|
output | event | string | Yes | Event name: RespondingContent |
dialog_id | string | Yes | Dialog ID | |
round_id | string | Yes | The ID of the current interaction round. | |
llm_request_id | string | Yes | The request_id for the large language model (LLM) call. | |
text | string | Yes | The text output from the system. This is a streamed, incremental output. | |
spoken | string | Yes | The text used for speech synthesis. This is a streamed, incremental output. | |
finished | bool | Yes | Indicates whether the output is complete. | |
finish_reason | string | No | The reason the output finished. Currently, only one reason is available:
| |
extra_info | object | No | Other extension information. The following are supported:
|
Example
{
"output": {
"event": "RespondingContent",
"dialog_id": "e788xxxx-xxxx-xxxx-xxxx-xxxx9637",
"round_id": "d8****************************29",
"llm_request_id": "fb****************************67",
"text": "Hello",
"spoken": "Hello",
"finished": true,
"finish_reason":"stop",
"extra_info": {
"agent_info": {
"round": 1,
"device": {
"device_id": ""
},
"intent_infos": [
{
"intent": "***",
"domain": "***"
}
]
},
"query": "Hello, what is the weather like in Beijing today?"
}
},
"request_id": "6481xxxx-xxxx-xxxx-xxxx-xxxx1aed0"
}
Error event
This event provides error information. For a description of error codes, see Multimodal interaction suite - Error codes.
Error - Output Message
Top-level parameter | Type | Required | Description |
|---|---|---|---|
code | string | Yes | Error code |
message | string | Yes | Error message |
request_id | string | Yes | Request ID |
{
"code": "InternalLLMError",
"message": "Internal LLM error",
"request_id": "0aa0xxxx-xxxx-xxxx-xxxx-xxxxaa21"
}
Call examples
The following is an example of a request.
The header must include X-DashScope-SSE: enable.
NoteReplace your_api_key, app_id, text, and other parameters in the parameters object with your actual values.
curl --location "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer your_api_key" \
--header "X-DashScope-SSE: enable" \
--data '{
"model": "multimodal-dialog",
"input": {
"directive": "Request",
"app_id": "****************",
"text":"Hello, what is the weather like in Beijing today?"
},
"parameters": {
"images":[{
"type": "base64",
"value": "base64String"
}],
"client_info": {
"user_id": "bin********207",
"device":{
"uuid": "432k*********k449"
},
"network":{
"ip": "x.x.x.x"
},
"location":{
"city_name": "Beijing"
}
}
}
}'
import okhttp3.*;
import okhttp3.sse.EventSource;
import okhttp3.sse.EventSourceListener;
import okhttp3.sse.EventSources;
import org.jetbrains.annotations.Nullable;
import java.io.IOException;
import java.util.concurrent.CountDownLatch;
public class SseTest {
public static void main(String[] args){
String sseUrl = "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation";
String apiKey = "your_api_key";
String appId = "your_app_id";
String model = "multimodal-dialog";
OkHttpClient okHttpClient = new OkHttpClient();
// Build headers
okhttp3.Headers.Builder headersBuilder = new okhttp3.Headers.Builder();
headersBuilder.add("Accept", "text/event-stream");
headersBuilder.add("Content-Type", "application/json");
headersBuilder.add("Authorization", "Bearer " + apiKey);
headersBuilder.add("X-DashScope-SSE", "enable");
MediaType mediaType = MediaType.parse("application/json; charset=utf-8");
String body = "{\n" +
" \"model\": \"" + model + "\",\n" +
" \"input\": {\n" +
" \"app_id\": \""+appId+"\",\n" +
" \"text\":\"Hello, what is the weather like in Beijing today?\",\n" +
" \"directive\": \"Request\"\n" +
" },\n" +
" \"parameters\": {\n" +
" \"client_info\": {\n" +
" \"user_id\": \"xxxxxxxxxxxxxxx\",\n" +
" \"device\": {\n" +
" \"uuid\": \"xxxxxx\"\n" +
" }\n" +
" }\n" +
" }\n" +
"}";
Request request = new Request.Builder()
.url(sseUrl)
.headers(headersBuilder.build())
.post(RequestBody.create(mediaType, body))
.build();
CountDownLatch latch = new CountDownLatch(1);
EventSourceListener listener = new EventSourceListener() {
@Override
public void onOpen(EventSource eventSource, Response response) {
System.out.println("onOpen response: " + response);
}
@Override
public void onEvent(EventSource eventSource, @Nullable String id, @Nullable String type, String data) {
System.out.println("onEvent: id: " + id + " | data: " + data);
}
@Override
public void onClosed(EventSource eventSource) {
System.out.println("onClosed");
latch.countDown();
}
@Override
public void onFailure(EventSource eventSource, @Nullable Throwable t, @Nullable Response response) {
System.out.println("onFailure "+t + " | response: " + response);
if(response != null && response.body() != null){
try {
System.err.println("onFailure response body: " + response.body().string());
} catch (IOException e) {
System.err.println("onFailure response body RuntimeException: " + e);
throw new RuntimeException(e);
}
}
latch.countDown();
}
};
EventSource.Factory factory = EventSources.createFactory(okHttpClient);
System.out.println("Connecting to SSE endpoint...");
EventSource eventSource = factory.newEventSource(request, listener);
// Block the main thread until the connection is closed
try {
latch.await();
} catch (InterruptedException e) {
System.err.println("latch.await RuntimeException: " + e.getMessage());
e.printStackTrace();
} finally {
//Clean up resources
eventSource.cancel();
okHttpClient.dispatcher().executorService().shutdown();
okHttpClient.connectionPool().evictAll();
}
System.out.println("SSE Connection closed!");
}
}