Multimodal embedding
The multimodal embedding (multi-modal-embedding) service converts inputs such as text and images into semantic vectors. It supports single-modality and combined multimodal inputs for scenarios such as cross-modal retrieval. For the input types and parameter ranges supported by each model, see the following table and the request parameter descriptions.
|
Model ID (service_id) |
Dimension |
Service description |
QPS limit |
|
ops-m2-encoder |
768 dimensions |
Bilingual (Chinese-English) multimodal service trained on BM-6B with 6 billion image-text pairs (3 billion Chinese, 3 billion English). Supports cross-modal retrieval (text-to-image, image-to-text) and image classification. Note
This model does not accept both text and an image in the same input object. |
10 Note
To request a higher API QPS limit, submit a ticket to technical support. |
|
ops-m2-encoder-large |
1024 dimensions |
A Chinese-English multimodal service with 1B parameters, more than the m2-encoder model. The larger parameter count improves its representation capabilities and performance on multimodal tasks. Note
This model does not accept both text and an image in the same input object. |
|
|
ops-gme-qwen2-vl-2b-instruct |
1536 dimensions |
Multimodal embedding service based on Qwen2-VL MLLMs. Supports single-modal and multimodal inputs for text, image, and combined data. |
|
|
ops-mm-embedding-v1-2b |
1536 dimensions |
A multimodal embedding model developed by the Alibaba Cloud OpenSearch-AI team and fine-tuned from Qwen2-VL 2B Instruct. It supports text, images, and combined text-image inputs for tasks such as cross-modal retrieval. |
|
|
ops-mm-embedding-v1-7b |
3584 dimensions |
A multimodal embedding model developed by the Alibaba Cloud OpenSearch-AI team and fine-tuned from Qwen2-VL 7B Instruct. It supports text, images, and combined text-image inputs for tasks such as cross-modal retrieval. |
|
|
ops-mm-embedding-face-001 |
512 dimensions |
Face retrieval model that encodes single or multiple image inputs into high-dimensional semantic vectors for face search and comparison. |
|
|
qwen3-vl-embedding |
2,560 dimensions by default. You can also select 2,048, 1,536, 1,024, 768, 512, or 256 dimensions. |
Supports text, images, videos, and combined multimodal inputs. Each request can contain up to 20 content elements, including up to 10 images and 1 video. You can use options.enable_fusion to fuse multiple inputs into a single vector. |
|
|
ops-mm-embedding-ecom-001 |
128 dimensions |
Designed for e-commerce scenarios. Supports text, images, and combined text-image inputs. |
Prerequisites
-
Get authentication credentials
The AI Search open platform requires an API key for authentication. For instructions, see Get an API key.
-
Get the service endpoint
You can call the service via the public network or a VPC. For details, see Get the service endpoint.
Request
Request body limit
The request body cannot exceed 8 MB.
Request method
POST
URL
{host}/v3/openapi/workspaces/{workspace_name}/multi-modal-embedding/{service_id}
-
host: The service endpoint. You can call the API service over the Internet or a VPC. For more information, see Obtain a service endpoint.
In the left navigation pane, select your workspace (for example, default) and click API Keys to find the public API endpoint and private API endpoint.
-
workspace_name: The workspace name, such as default.
-
service_id: The ID of a built-in service, such as ops-m2-encoder.
Request parameters
Header parameters
API key authentication
|
Parameter |
Type |
Required |
Description |
Example |
|
Content-Type |
String |
Yes |
Content-Type: application/json |
application/json |
|
Authorization |
String |
Yes |
The API key for authentication. |
Bearer OS-d1**2a |
Body parameters
|
Parameter |
Type |
Required |
Description |
Example |
|
input |
List[ContentObject] |
Yes |
The content to embed. For qwen3-vl-embedding, each request supports up to 20 content elements, including up to 10 images and 1 video. The other listed models support up to 32 input items per request. When options.enable_fusion is false, a separate vector is generated for each ContentObject. When it is true, all content in input is fused into a single vector. enable_fusion applies only to qwen3-vl-embedding. |
|
|
options |
OptionObject |
No |
Model settings, including image resizing and vector output settings. For the models to which each field applies, see OptionObject. |
|
ContentObject
|
Parameter |
Type |
Required |
Description |
Example |
|
text |
String |
No |
Text to convert into an embedding. |
|
|
image |
String |
No |
The image information, specified as a URL or Base64-encoded data.
|
or
|
|
video |
String |
No |
The URL of a video file. Only qwen3-vl-embedding supports this parameter. Setting video for other models causes an error. |
|
OptionObject
The following fields belong to the options object in the request body.
|
Parameter |
Type |
Required |
Description |
Example |
|
resize_method |
String |
No |
The image resizing method:
|
fixed_size |
|
resize_options |
Object |
No |
The resizing parameters used when resize_method is fixed_size or proportional_scaling. For fixed_size, you must specify width and height in pixels. For proportional_scaling, use min_pixels and max_pixels to specify the minimum and maximum pixel counts after resizing. |
fixed_size:
proportional_scaling:
|
|
enable_fusion |
Boolean |
No |
Specifies whether to generate a fused vector. This parameter applies only to qwen3-vl-embedding and defaults to false. If false, a separate vector is generated for each input item. If true, all content in input is fused into a single vector. Supported combinations include text and images, text and video, multiple images and text, and images, video, and text. |
true |
|
instruct |
String |
No |
A custom task instruction that guides the model in understanding the query intent. This parameter applies only to qwen3-vl-embedding. English instructions are recommended. |
Represent the input for cross-modal retrieval. |
|
dimension |
Int |
No |
The output vector dimension. qwen3-vl-embedding supports 2560, 2048, 1536, 1024, 768, 512, and 256. The default is 2560. |
1024 |
Response parameters
|
Parameter |
Type |
Description |
Example |
|
result.embeddings |
List |
The embedding results of this request. When fusion is disabled, each array element corresponds to an input item. When options.enable_fusion is enabled for qwen3-vl-embedding, all inputs are fused into a single vector. |
|
|
result.embeddings[].index |
Int |
When fusion is disabled, the zero-based index of the corresponding item in the input array. |
0 |
|
result.embeddings[].embedding |
List[Double] |
The resulting embedding vector. |
[0.003143,0.009750,-0.017395] (only some elements are shown) |
|
usage.image_token |
Int |
The number of image tokens used. |
0 |
|
usage.text_token |
Int |
The number of text tokens used. |
6 |
|
usage.image |
Int |
The number of images. |
1 |
|
usage.token_count |
Int |
The number of text tokens used. |
6 |
cURL request example
Replace {host} with the service endpoint and the placeholder in Authorization with your API key. Replace the image and video URLs with URLs of resources that are accessible.
Text embedding
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/qwen3-vl-embedding" \
-d '{
"input": [
{
"text": "Science and technology are the primary productive forces."
}
],
"options": {
"enable_fusion": false,
"dimension": 1024
}
}'
Fused text-image embedding
Use qwen3-vl-embedding to fuse text and an image into a single vector.
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/qwen3-vl-embedding" \
-d '{
"input": [
{
"text": "Retrieve content similar to this description and image."
},
{
"image": "https://example.com/image.jpg"
}
],
"options": {
"enable_fusion": true,
"instruct": "Represent the input for cross-modal retrieval.",
"dimension": 1024
}
}'
Fused text-video embedding
The video parameter applies only to qwen3-vl-embedding. Each request supports up to 1 video.
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/qwen3-vl-embedding" \
-d '{
"input": [
{
"text": "Retrieve content similar to this description and video."
},
{
"video": "https://example.com/video.mp4"
}
],
"options": {
"enable_fusion": true,
"dimension": 1024
}
}'
Image resizing
The following example uses ops-m2-encoder to resize the input image to 224 × 224 pixels.
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/ops-m2-encoder" \
-d '{
"input": [
{
"image": "https://example.com/image.jpg"
}
],
"options": {
"resize_method": "fixed_size",
"resize_options": {
"width": 224,
"height": 224
}
}
}'
Response examples
Sample success response
The following example shows the response to a qwen3-vl-embedding text embedding request. Only some elements of the embedding array are shown.
{
"request_id": "f4483a215f4d68dbc6d6c6c37f006854",
"latency": 259,
"usage": {
"image_token": 0,
"text_token": 24
},
"result": {
"embeddings": [
{
"index": 0,
"embedding": [
-0.00211523100733757,
-0.03339154273271561,
0.04578055068850517,
0.017939193174242973,
0.0033657182939350605,
-0.06284935027360916
]
}
]
}
}
Sample error response
If a request fails, code and message in the response indicate the cause of the error.
{
"request_id": "651B3087-8A07-****-B931-9C4E7B60F52D",
"latency": 0,
"code": "InvalidParameter",
"message": "JSON parse error: Cannot deserialize value of type `InputType` from String \"xxx\""
}
Status codes
All status codes are documented in Status codes.