Multimodal embedding

Updated at:

The multimodal embedding (multi-modal-embedding) service converts inputs such as text and images into semantic vectors. It supports single-modality and combined multimodal inputs for scenarios such as cross-modal retrieval. For the input types and parameter ranges supported by each model, see the following table and the request parameter descriptions.

Model ID (service_id)

Dimension

Service description

QPS limit

ops-m2-encoder

768 dimensions

Bilingual (Chinese-English) multimodal service trained on BM-6B with 6 billion image-text pairs (3 billion Chinese, 3 billion English). Supports cross-modal retrieval (text-to-image, image-to-text) and image classification.

Note

This model does not accept both text and an image in the same input object.

10

Note

To request a higher API QPS limit, submit a ticket to technical support.

ops-m2-encoder-large

1024 dimensions

A Chinese-English multimodal service with 1B parameters, more than the m2-encoder model. The larger parameter count improves its representation capabilities and performance on multimodal tasks.

Note

This model does not accept both text and an image in the same input object.

ops-gme-qwen2-vl-2b-instruct

1536 dimensions

Multimodal embedding service based on Qwen2-VL MLLMs. Supports single-modal and multimodal inputs for text, image, and combined data.

ops-mm-embedding-v1-2b

1536 dimensions

A multimodal embedding model developed by the Alibaba Cloud OpenSearch-AI team and fine-tuned from Qwen2-VL 2B Instruct. It supports text, images, and combined text-image inputs for tasks such as cross-modal retrieval.

ops-mm-embedding-v1-7b

3584 dimensions

A multimodal embedding model developed by the Alibaba Cloud OpenSearch-AI team and fine-tuned from Qwen2-VL 7B Instruct. It supports text, images, and combined text-image inputs for tasks such as cross-modal retrieval.

ops-mm-embedding-face-001

512 dimensions

Face retrieval model that encodes single or multiple image inputs into high-dimensional semantic vectors for face search and comparison.

qwen3-vl-embedding

2,560 dimensions by default. You can also select 2,048, 1,536, 1,024, 768, 512, or 256 dimensions.

Supports text, images, videos, and combined multimodal inputs. Each request can contain up to 20 content elements, including up to 10 images and 1 video. You can use options.enable_fusion to fuse multiple inputs into a single vector.

ops-mm-embedding-ecom-001

128 dimensions

Designed for e-commerce scenarios. Supports text, images, and combined text-image inputs.

Prerequisites

  • Get authentication credentials

    The AI Search open platform requires an API key for authentication. For instructions, see Get an API key.

  • Get the service endpoint

    You can call the service via the public network or a VPC. For details, see Get the service endpoint.

Request

Request body limit

The request body cannot exceed 8 MB.

Request method

POST

URL

{host}/v3/openapi/workspaces/{workspace_name}/multi-modal-embedding/{service_id} 
  • host: The service endpoint. You can call the API service over the Internet or a VPC. For more information, see Obtain a service endpoint.

    In the left navigation pane, select your workspace (for example, default) and click API Keys to find the public API endpoint and private API endpoint.

  • workspace_name: The workspace name, such as default.

  • service_id: The ID of a built-in service, such as ops-m2-encoder.

Request parameters

Header parameters

API key authentication

Parameter

Type

Required

Description

Example

Content-Type

String

Yes

Content-Type: application/json

application/json

Authorization

String

Yes

The API key for authentication.

Bearer OS-d1**2a

Body parameters

Parameter

Type

Required

Description

Example

input

List[ContentObject]

Yes

The content to embed. For qwen3-vl-embedding, each request supports up to 20 content elements, including up to 10 images and 1 video. The other listed models support up to 32 input items per request. When options.enable_fusion is false, a separate vector is generated for each ContentObject. When it is true, all content in input is fused into a single vector. enable_fusion applies only to qwen3-vl-embedding.

[
  {
    "text":"Science and technology are the primary productive forces"
  },
  {
    "image":"http://***/a.jpg"
  }
]

options

OptionObject

No

Model settings, including image resizing and vector output settings. For the models to which each field applies, see OptionObject.

{
  "enable_fusion": false,
  "dimension": 1024
}

ContentObject

Parameter

Type

Required

Description

Example

text

String

No

Text to convert into an embedding.

{
  "text":"Text input"
}

image

String

No

The image information, specified as a URL or Base64-encoded data.

  • URLs must be publicly accessible.

  • For Base64 input, pass data to the image parameter in the format data:image/{format};base64,{base64_image}, where:

    image/{format}: The local image format. Set this value to the actual image format. For example, for a JPG image, use image/jpeg.

    base64_image: The Base64-encoded image data.

{
  "image":"http://xxxxx/a.jpg"
}

or

{
  "image":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAoHCB..."
}

video

String

No

The URL of a video file. Only qwen3-vl-embedding supports this parameter. Setting video for other models causes an error.

{
  "video": "https://example.com/video.mp4"
}

OptionObject

The following fields belong to the options object in the request body.

Parameter

Type

Required

Description

Example

resize_method

String

No

The image resizing method:

  • default: Uses the default resizing behavior built into the model.

  • none: Does not resize the image.

  • fixed_size: Resizes the image to the width and height specified in resize_options.

  • proportional_scaling: Preserves the aspect ratio and resizes the image to the pixel range specified in resize_options.

fixed_size

resize_options

Object

No

The resizing parameters used when resize_method is fixed_size or proportional_scaling. For fixed_size, you must specify width and height in pixels. For proportional_scaling, use min_pixels and max_pixels to specify the minimum and maximum pixel counts after resizing.

fixed_size:

{
  "width": 224,
  "height": 224
}

proportional_scaling:

{
  "min_pixels": 409600,
  "max_pixels": 1638400
}

enable_fusion

Boolean

No

Specifies whether to generate a fused vector. This parameter applies only to qwen3-vl-embedding and defaults to false. If false, a separate vector is generated for each input item. If true, all content in input is fused into a single vector. Supported combinations include text and images, text and video, multiple images and text, and images, video, and text.

true

instruct

String

No

A custom task instruction that guides the model in understanding the query intent. This parameter applies only to qwen3-vl-embedding. English instructions are recommended.

Represent the input for cross-modal retrieval.

dimension

Int

No

The output vector dimension. qwen3-vl-embedding supports 2560, 2048, 1536, 1024, 768, 512, and 256. The default is 2560.

1024

Response parameters

Parameter

Type

Description

Example

result.embeddings

List

The embedding results of this request. When fusion is disabled, each array element corresponds to an input item. When options.enable_fusion is enabled for qwen3-vl-embedding, all inputs are fused into a single vector.

[
  {
    "index": 0,
    "embedding": [
      0.003143,
      0.00975,
      -0.017395
    ]
  }
]

result.embeddings[].index

Int

When fusion is disabled, the zero-based index of the corresponding item in the input array.

0

result.embeddings[].embedding

List[Double]

The resulting embedding vector.

[0.003143,0.009750,-0.017395] (only some elements are shown)

usage.image_token

Int

The number of image tokens used.

0

usage.text_token

Int

The number of text tokens used.

6

usage.image

Int

The number of images.

1

usage.token_count

Int

The number of text tokens used.

6

cURL request example

Replace {host} with the service endpoint and the placeholder in Authorization with your API key. Replace the image and video URLs with URLs of resources that are accessible.

Text embedding

curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/qwen3-vl-embedding" \
-d '{
  "input": [
    {
      "text": "Science and technology are the primary productive forces."
    }
  ],
  "options": {
    "enable_fusion": false,
    "dimension": 1024
  }
}'

Fused text-image embedding

Use qwen3-vl-embedding to fuse text and an image into a single vector.

curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/qwen3-vl-embedding" \
-d '{
  "input": [
    {
      "text": "Retrieve content similar to this description and image."
    },
    {
      "image": "https://example.com/image.jpg"
    }
  ],
  "options": {
    "enable_fusion": true,
    "instruct": "Represent the input for cross-modal retrieval.",
    "dimension": 1024
  }
}'

Fused text-video embedding

The video parameter applies only to qwen3-vl-embedding. Each request supports up to 1 video.

curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/qwen3-vl-embedding" \
-d '{
  "input": [
    {
      "text": "Retrieve content similar to this description and video."
    },
    {
      "video": "https://example.com/video.mp4"
    }
  ],
  "options": {
    "enable_fusion": true,
    "dimension": 1024
  }
}'

Image resizing

The following example uses ops-m2-encoder to resize the input image to 224 × 224 pixels.

curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"https://{host}/v3/openapi/workspaces/default/multi-modal-embedding/ops-m2-encoder" \
-d '{
  "input": [
    {
      "image": "https://example.com/image.jpg"
    }
  ],
  "options": {
    "resize_method": "fixed_size",
    "resize_options": {
      "width": 224,
      "height": 224
    }
  }
}'

Response examples

Sample success response

The following example shows the response to a qwen3-vl-embedding text embedding request. Only some elements of the embedding array are shown.

{
  "request_id": "f4483a215f4d68dbc6d6c6c37f006854",
  "latency": 259,
  "usage": {
    "image_token": 0,
    "text_token": 24
  },
  "result": {
    "embeddings": [
      {
        "index": 0,
        "embedding": [
          -0.00211523100733757,
          -0.03339154273271561,
          0.04578055068850517,
          0.017939193174242973,
          0.0033657182939350605,
          -0.06284935027360916
        ]
      }
    ]
  }
}

Sample error response

If a request fails, code and message in the response indicate the cause of the error.

{
    "request_id": "651B3087-8A07-****-B931-9C4E7B60F52D",
    "latency": 0,
    "code": "InvalidParameter",
    "message": "JSON parse error: Cannot deserialize value of type `InputType` from String \"xxx\""
}

Status codes

All status codes are documented in Status codes.