End-to-end image search solution

You can import raw image data directly into OpenSearch for automatic vectorization and vector search. This enables you to implement features such as search by image and search by text.

Solution architecture

You can build an image search engine by using one of the following three methods to upload images:

• OpenSearch Vector Search Edition with Object Storage Service (OSS) and MaxCompute: Upload your images to an OSS bucket. Then, store your business data and the corresponding image paths in MaxCompute. An example of an image path is /image/1.jpg.

• OpenSearch Vector Search Edition with MaxCompute: Store your Base64-encoded images and the corresponding table data in MaxCompute.

• Push your Base64-encoded images and table data to your instance by using the OpenSearch Vector Search Edition API.

This tutorial demonstrates building an image search engine by using OpenSearch Vector Search Edition with Object Storage Service (OSS) and MaxCompute.

Prerequisites

1. Create an access key pair

When you create an Alibaba Cloud account and log on to the console for the first time, the system prompts you to create an access key before you can proceed.

• An access key for your primary account is required to create and use applications.

• After creating an access key for your primary account, you can create a RAM user and an access key for the user to manage access with specific permissions. For more information, see Create and authorize a RAM user.

2. Create and configure an OSS bucket

1. Activate OSS.

2. Create a bucket.

3. Upload an object.

This tutorial uses 1,000 images uploaded to an OSS bucket.

The following figure shows some of the uploaded images.

Purchase an OpenSearch Vector Search instance

1. Navigate to the OpenSearch console and switch to OpenSearch Vector Search Edition in the upper-left corner.

2. On the Instance Management page, click Create Instance.

3. On the buy page, select Vector Search Edition for Service Edition. Configure the Region, Number of Query Nodes, Query Node Specifications, Number of Data Nodes, Data Node Specifications, and Total Storage for a Single Data Node. Set the VPC and vSwitch. Finally, set a username and password for query authentication, which must be different from your Alibaba Cloud account credentials. Click Buy Now.

4. On the Confirm Order page, review the terms of service and click Activate Now.

5. After the purchase is successful, click Console to view your new instance on the Instance Management page.

6. A newly purchased instance has a default name. To rename it, click Manage in the Actions column to open the instance details page.

Click the edit icon, enter a new instance name in the dialog box, and click Confirm.

Configure the cluster

The status of a new instance is "Pending Configuration." An empty cluster is automatically deployed with the number and specifications of query nodes and data nodes that you purchased. Before the cluster can be searched, you must configure a data source, define an index schema, and perform reindexing.

1. Configure a data source

1. Configure a data source. Supported data sources include MaxCompute, API, and Object Storage Service (OSS). This example uses a MaxCompute data source. Click Add Data Source, select MaxCompute as the data source type, and configure the project, access key ID, access key secret, table, and partition key. You can also enable Automatic Reindexing based on your needs.

After the settings are validated, click OK to add the data source.

2. Configure an index schema

1. After the data source is configured, click Next to configure the index schema.

2. Click Create Index Table.

3. Configure the index table. From the template list, select Vector: Image Search. For Data, select Need to Convert Raw Data to Vector Data.

4. Configure fields. This section describes the configuration for two scenarios: images stored in OSS and Base64-encoded images.

Images stored in OSS

When you select the "Vector: Image Search" template, the system automatically creates four preset fields: id (primary key), cate_id (category field), vector (vector field), and vector_source_image (field to store the image path). After you select a MaxCompute data source, the fields synchronized from it are displayed below the preset fields.

4.1. Configure the vector_source_image field. (Field type must be STRING.)

You can rename the preset fields to match your business table, but you must ensure their advanced configurations are correct.

The configuration is as follows:

• Data Type: Select image(path).

• Source Content Type: Select oss.

• OSS Bucket: Enter the name of the OSS bucket where the images for vectorization are stored. Ensure the current account has access to this bucket.

4.2. Configure the vector field. (Field type must be FLOAT.)

After configuring the vector_source_image field to allow the engine to access images in OSS, you must also configure the vector field to convert the raw image data into vectors. You can rename the field, but ensure its advanced configuration is correct.

The configuration is as follows:

{
  "vector_model": "clip",
  "vector_modal": "image",
  "vector_source_field": "vector_source_image"
}

• vector_model: The vectorization model. Models for general-purpose images and e-commerce enhanced images are supported.

  • For general images, use "vector_model": "clip". This model converts a general-purpose image to a vector.

  • For e-commerce enhanced images, use "vector_model": "clip_ecom". This model converts an e-commerce image to a vector.

• vector_modal: Set to image.

• vector_source_field: The field that stores the OSS image path. The default is vector_source_image.

4.3. Schema example:

"fields": [
    {
      "field_name": "id",
      "field_type": "INT64",
      "compress_type": "equal"
    },
    {
      "user_defined_param": {
        "content_type": "oss",
        "oss_endpoint": "",
        "oss_bucket": "The name of your OSS bucket",
        "oss_secret": "The access key secret of an account that can access OSS",
        "oss_access_key": "The access key ID of an account that can access OSS"
      },
      "field_name": "vector_source_image",
      "field_type": "STRING",
      "compress_type": "uniq"
    },
    {
      "field_name": "cate_id",
      "field_type": "INT64",
      "compress_type": "equal"
    },
    {
      "user_defined_param": {
        "vector_model": "clip",
        "vector_modal": "image",
        "vector_source_field": "vector_source_image"
      },
      "field_name": "vector",
      "field_type": "FLOAT",
      "multi_value": true
    }
  ]

Base64-encoded images

When you select the "Vector: Image Search" template, the system creates four preset fields: id (primary key), cate_id (category field), vector (vector field), and vector_source_image (field to store the Base64-encoded image data). Fields synchronized from your MaxCompute data source are displayed below these preset fields.

4.1. Configure the vector_source_image field. (Field type must be STRING.)

You can rename the preset field to match your business table.

You must manually clear the advanced configuration for the vector_source_image field, leaving only {}.

4.2. Configure the vector field:

After configuring the vector_source_image field with Base64-encoded data, you must also configure the vector field to enable vectorization. You can rename the field, but ensure its advanced configuration is correct.

The configuration is as follows:

{
  "vector_model": "clip",
  "vector_modal": "image",
  "vector_source_field": "vector_source_image"
}

• vector_model: Set to clip.

• vector_modal: Set to image.

• vector_source_field: The field that stores the Base64-encoded image data. The default is vector_source_image.

4.3. Schema example:

 "fields": [
    {
      "field_name": "id",
      "field_type": "INT64",
      "compress_type": "equal"
    },
    {
      "field_name": "vector_source_image",
      "field_type": "STRING",
      "compress_type": "uniq"
    },
    {
      "field_name": "cate_id",
      "field_type": "INT64",
      "compress_type": "equal"
    },
    {
      "user_defined_param": {
        "vector_model": "clip",
        "vector_modal": "image",
        "vector_source_field": "vector_source_image"
      },
      "field_name": "vector",
      "field_type": "FLOAT",
      "multi_value": true
    }
  ]

5. When you select the "Vector: Image Search" template, two indexes are created by default: a primary key index (id) and a vector index (vector). After you complete the advanced configuration for the vector_source_image and vector fields, the advanced configuration for the vector index is automatically generated. The index type must be CUSTOMIZED.

Update the fields included in the vector index based on your field settings.

Schema example:

"indexs": [
    {
      "index_name": "id",
      "index_type": "PRIMARYKEY64",
      "index_fields": "id",
      "has_primary_key_attribute": true,
      "is_primary_key_sorted": false
    },
    {
      "index_name": "vector",
      "index_type": "CUSTOMIZED",
      "index_fields": [
        {
          "field_name": "id",
          "boost": 1
        },
        {
          "field_name": "vector",
          "boost": 1
        }
      ],
      "parameters": {
        "dimension": "512",
        "distance_type": "SquaredEuclidean",
        "vector_index_type": "Qc",
        "build_index_params": "{\"proxima.qc.builder.quantizer_class\":\"Int8QuantizerConverter\",\"proxima.qc.builder.quantize_by_centroid\":true,\"proxima.qc.builder.optimizer_class\":\"BruteForceBuilder\",\"proxima.qc.builder.thread_count\":10,\"proxima.qc.builder.optimizer_params\":{\"proxima.linear.builder.column_major_order\":true},\"proxima.qc.builder.store_original_features\":false,\"proxima.qc.builder.train_sample_count\":3000000,\"proxima.qc.builder.train_sample_ratio\":0.5}",
        "search_index_params": "{\"proxima.qc.searcher.scan_ratio\":0.01}",
        "embedding_delimiter": ",",
        "major_order": "col",
        "linear_build_threshold": "5000",
        "min_scan_doc_cnt": "20000",
        "enable_recall_report": "false",
        "is_embedding_saved": "false",
        "enable_rt_build": "false",
        "builder_name": "QcBuilder",
        "searcher_name": "QcSearcher"
      },
      "indexer": "aitheta2_indexer"
    }
  ]

3. Reindexing

1. After you finish the configuration, click Save Version. In the dialog box, you can add an optional description and then click Publish.

After the index is published, click Next to start reindexing.

2. In the reindexing step, configure the parameters and click Next.

• For an API data source:

Select the Data Source Name, confirm that the Data Source Type is API and the Associated Index Table is main. Set the Partition and Timestamp, then click Next.

• For a MaxCompute data source:

Select the Data Source Name (for example, ha-cn-jaj34s2fq07_store). Confirm that the Data Source Type is MaxCompute and select the Associated Index Table (for example, store_index). Enter the Partition (for example, ds=20221102) and Timestamp, then click Next.

3. Navigate to O&M Center > Change History > Data Source Changes to monitor the reindexing progress. You can perform query tests after the process is complete.

The page displays a timeline for the reindexing process. It shows the progress and status of each stage: init, trigger, scan, build, and switch. You can test queries after all stages are complete.

Test the solution

Syntax

query=image_index:'Base64-encoded text for the search&modal=text&n=10&search_params={}'

• modal: The modality type. For search by text, set this to text. For search by image, set this to image.

• n: The number of top-k results to return from the vector search.

• The text content must be Base64-encoded.

Search by text

On the Query Test page, run a Haval3 (HA3) query.

vector:'5pGp5omY6L2mJuWktOeblA==&modal=text&n=10&search_params={}'&&kvpairs=formula:proxima_score(vector)&&sort=+RANK

The 2042.png image in OSS is a photo of a black motorcycle helmet that matches the search keywords. This result verifies the accuracy of the search by text feature.

Search by image

A Base64-encoded image string is typically too long to be used directly in the Query Test page of the console. Use an SDK to perform a search by image.

Example:

vector:'Base64-encoded image data&modal=image&n=10&search_params={}'&&kvpairs=formula:proxima_score(vector)&&sort=+RANK

Searching with an SDK

Add the dependency:

pip install alibabacloud-ha3engine

Search demo:

# -*- coding: utf-8 -*-

from alibabacloud_ha3engine import models, client
from alibabacloud_tea_util import models as util_models
from Tea.exceptions import TeaException, RetryError
def search():
    Config = models.Config(
        endpoint="Your instance endpoint. You can find it on the instance details page in the API Endpoints section.",
        instance_id="",
        protocol="http",
        access_user_name="The username you set when purchasing the instance.",
        access_pass_word="The password you set when purchasing the instance."

    )

    # If requests take a long time, you can increase the timeout values. Unit: ms.
    # This parameter can be used in the search_with_options method.
    runtime = util_models.RuntimeOptions(
        connect_timeout=5000,
        read_timeout=10000,
        autoretry=False,
        ignore_ssl=False,
        max_idle_conns=50
    )

    # Initialize the HA3 Engine client.
    ha3EngineClient = client.Client(Config)

    optionsHeaders = {}

    try:
        # Example 1: Search directly by using an HA3 query string.
        # =====================================================
        query_str = "config=hit:4,format:json,fetch_summary_type:pk,qrs_chain:search&&query=image_index:'Base64-encoded text for the search&modal=text&n=10&search_params={}'&&cluster=general"
        haSearchQuery = models.SearchQuery(query=query_str)
        haSearchRequestModel = models.SearchRequestModel(optionsHeaders, haSearchQuery)
        hastrSearchResponseModel = ha3EngineClient.search(haSearchRequestModel)
        print(hastrSearchResponseModel)
    except TeaException as e:
        print(f"send request with TeaException : {e}")
    except RetryError as e:
        print(f"send request with Connection Exception  : {e}")

Usage notes

• For strict vector search latency requirements, we recommend using the Lock policy for vector indexes.

• The field that stores image paths or Base64-encoded image data must be of the STRING type.

• The vector index type must be set to CUSTOMIZED.

• This solution supports the HA3 syntax and RESTful APIs, but not SQL.