Multimodal embedding API-OpenSearch(Open Search)-阿里云帮助中心

The multimodal embedding service is trained on the Qwen2-VL multimodal large language models (MLLMs). It supports both single-modal and multimodal inputs, and efficiently processes text, image, and combined data types.

Model ID (service_id)	Dimension	Service description	QPS limit
ops-m2-encoder	768 dimensions	A bilingual (Chinese-English) multimodal service trained on the BM-6B model with a dataset of 6 billion image-text pairs (3 billion Chinese and 3 billion English). This model supports cross-modal retrieval (including text-to-image and image-to-text searches) and image classification tasks. Note This model does not accept both text and an image in the same input object.	10 Note To request a higher QPS limit, submit a ticket to technical support.
ops-m2-encoder-large	1024 dimensions	A bilingual (Chinese-English) multimodal service. This model has a larger parameter count (1 billion) than the `ops-m2-encoder` model, offering enhanced expressive power and performance for multimodal tasks. Note This model does not accept both text and an image in the same input object.
ops-gme-qwen2-vl-2b-instruct	1536 dimensions	A multimodal embedding service trained on the Qwen2-VL multimodal large language models (MLLMs). It supports both single-modal and multimodal inputs, and efficiently processes text, image, and combined data types.
ops-mm-embedding-v1-2b	1536 dimensions	A multimodal embedding model developed by the Alibaba Cloud AI Search Open Platform team. It is fine-tuned on Qwen2-VL 2B Instruct and supports single-modal and multimodal inputs, including text, images, and video. It encodes these inputs into a unified semantic vector, making it suitable for cross-modal retrieval and understanding tasks.
ops-mm-embedding-v1-7b	3584 dimensions	A multimodal embedding model developed by the Alibaba Cloud AI Search Open Platform team. It is fine-tuned on Qwen2-VL 7B Instruct and supports single-modal and multimodal inputs, including text, images, and video. It encodes these inputs into a unified semantic vector, making it suitable for cross-modal retrieval and understanding tasks.
ops-mm-embedding-face-001	512 dimensions	Designed for face retrieval tasks. This model uses an advanced face embedding model to encode single or multiple image inputs into high-dimensional semantic vectors, enabling efficient and accurate face search and comparison.

Prerequisites

Obtain authentication credentials
You must authenticate calls to the AI Search Open Platform API. To obtain an API key, see Obtain an API key.
Get the service endpoint
You can call the service API over the public network or through a VPC. For details, see Get service endpoints.

Request

Request body limit

The request body cannot exceed 8 MB.

Request method

POST

URL

{host}/v3/openapi/workspaces/{workspace_name}/multi-modal-embedding/{service_id}

host: The endpoint for the service. You can call the API over the public network or through a VPC. For details, see Get service endpoints.

From the left-side navigation pane, select the target workspace (e.g., default) and click API Keys. The public API endpoint and private API endpoint are listed at the top of the page.
workspace_name: The name of the workspace, such as default.
service_id: The ID of the model, such as ops-m2-encoder.

Request parameters

Header parameters

API key authentication

Parameter	Type	Required	Description	Example
Content-Type	String	Yes	The request content type. Set this to `application/json`.	application/json
Authorization	String	Yes	The API key for authentication.	Bearer OS-d1**2a

Body parameters

Parameter	Type	Required	Description	Example
input	List[ContentObject]	Yes	The content to be converted into embeddings. You can provide multiple content objects. A single request supports up to 32 objects.	`[ { "text":"Science and technology are the primary productive forces" }, { "image":"http://***/a.jpg" } ]`

ContentObject

Parameter

Type

Required

Description

Example

text

String

The text to be converted into an embedding.

{
  "text":"Text input"
}

image

String

The image to be converted into an embedding. You can provide the image as a URL or as a Base64-encoded string.

If you provide a URL, it must be publicly accessible.
Image with BASE64 data: Pass the encoded BASE64 data to the image parameter in the format data:image/{format};base64,{base64_image}, where:

image/{format}: The format of the image. For example, use image/jpeg for a JPG image.

base64_image: The Base64-encoded image data.

{
  "image":"http://xxxxx/a.jpg"
}

{
  "image":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAoHCB..."
}

Response parameters

Parameter	Type	Description	Example
result.embeddings	List	The embedding results for the request. This is an array of objects, where each object corresponds to an object in the `input` array.	`[ { "index": 0, "embedding": [0.003143,0.009750,...,-0.017395] }, {} ]`
result.embeddings[].index	Int	The index of the corresponding input object.	0
result.embeddings[].embedding	List[Double]	The resulting embedding vector.	[0.003143,0.009750,...,-0.017395]

cURL request example

curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"http://****-hangzhou.opensearch.aliyuncs.com/v3/openapi/workspaces/default/multi-modal-embedding/ops-m2-encoder" \
-d '{
"input":[
  {
    "image":"http://***/a.jpg"
  }
]
}'

Response examples

Sample success response

{
    "request_id": "B4AB89C8-B135-****-A6F8-2BAB801A2CE4",
    "latency": 38,
    "usage": {
        "image":1,
        "token_count":28
    },
    "result": {
        "embeddings": [
            {
                "index": 0,
                "embedding": [
                   -0.033447265625,
                   0.10577392578125,
                   -0.0015211105346679688,
                   -0.044189453125,
                    ...
                   0.004688262939453125,
                   -4.5239925384521484E-5
                ]
            }
        ]
    }
}

Sample error response

If an error occurs, the response includes a code and a message that indicate the cause.

{
    "request_id": "651B3087-8A07-****-B931-9C4E7B60F52D",
    "latency": 0,
    "code": "InvalidParameter",
    "message": "JSON parse error: Cannot deserialize value of type `InputType` from String \"xxx\""
}

Status codes

For more information, see Status codes for AI Search Open Platform.