Multi-modal retrieval solution

Background

Features

• You can deploy services with pretrained models from Alibaba PAI EasyCV or EasyNLP, or with models that you train on the DSW platform, such as image feature extraction, text vectorization, and image-text CLIP models.

• By default, the service uses a persistent and fast-retrieval feature database based on DocArray-PAI and Object Storage Service (OSS). This allows you to store and retrieve features in OSS. For example, if the service stores 100,000 features, you can save the database and stop the service. When you restart the service, you can search for features without re-registering the data. You can also configure other retrieval products, such as Alibaba Cloud Elasticsearch.

Overview

• Currently, the multi-modal retrieval solution supports the following service types: image retrieval (image_retrieval), face retrieval (faceid_retrieval), text retrieval (text_retrieval), Chinese multi-modal retrieval (imagetext_retrieval_cn), and English multi-modal retrieval (imagetext_retrieval_en).

• Each service provides the following API groups: service initialization APIs (such as init and set_root_path), database management layer APIs (such as add and delete), and database layer APIs (such as db_set and db_get).

Workflow

The workflow is as follows.

1. Build a model

   Use a pretrained model provided by PAI or build your own model by following the examples in the DSW Gallery.

2. Deploy the model service

   Use Elastic Algorithm Service (EAS) to deploy your trained model or a pretrained model from PAI as an online service.

3. Invoke the model service

   Call the service initialization, database management layer, and database layer APIs to retrieve multi-modal data.

Prerequisites

Before you begin, ensure you have completed the following preparations:

• You have activated the pay-as-you-go billing method for PAI (including DSW and EAS) and created a default workspace. For more information, see Create and manage workspaces.

• You have created an Object Storage Service (OSS) bucket to store datasets, trained model files, and configuration files. For more information about how to create a bucket, see Create a bucket in the OSS console.

• You have created a DSW instance that is running. For more information, see Create a DSW instance. We recommend the following configurations:

  • Select a Pytorch 1.8 image, such as pytorch-develop:1.8pai-gpu-py36-cu101-ubuntu18.04.

  • For the instance type, use a GPU-accelerated P100 or V100 instance with at least 32 GB of memory, such as ecs.gn6v-c8g1.2xlarge.

Build a model

The following table lists the pretrained models provided by PAI.

Model type	Processor type	Predictor	Model path
image_retrieval (image retrieval)	EasyCV	easycv.predictors.feature_extractor.TorchFeatureExtractor	http://pai-vision-data-hz.oss-accelerate.aliyuncs.com/pretrained_models/deepfashion2/df2_1536_easycv061.pth
faceid_retrieval (face retrieval)	EasyCV	easycv.predictors.feature_extractor.TorchFaceFeatureExtractor	http://pai-vision-data-hz.oss-accelerate.aliyuncs.com/pretrained_models/faceid/resnet50/faceid_resnet50.tar.gz
imagetext_retrieval_cn (Chinese CLIP)	EasyNLP	easynlp.appzoo.CLIPPredictor	http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/easynlp_modelzoo/alibaba-pai/clip_chinese_roberta_base_vit_base.tgz
imagetext_retrieval_en (English CLIP)	EasyNLP	easynlp.appzoo.CLIPPredictor	http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/easynlp_modelzoo/openai/clip_vit_base_patch32.tgz
text_retrieval (Text vectorization model)	EasyNLP	easynlp.appzoo.FeatureVectorizationPredictor	http://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/easynlp_modelzoo/public/bert-base-uncased.tgz

You can also build your own model in DSW by following the examples in the Gallery. To do this, perform the following steps:

1. Go to the DSW Notebook Gallery page.

2. On the DSW Gallery page, build the following two types of models.

   • Multi-modal retrieval model

     In the search box, enter CLIP-based Multi-modal Text-Image Retrieval by Using EasyNLP and press Enter. In the CLIP-based Multi-modal Text-Image Retrieval by Using EasyNLP card, click Open in DSW. The required resources and tutorial file are automatically downloaded to your DSW instance, and the tutorial file opens after the download completes.

     In the easynlp_clip.ipynb file, run the commands sequentially.

     ! echo y | pip uninstall pai-easynlp easynlp
     ! git clone https://github.com/alibaba/EasyNLP.git
     ! pip install -r EasyNLP/requirements.txt -i http://mirrors.aliyun.com/pypi/simple
     ! cd EasyNLP && python setup.py install

   • Image retrieval model

     In the search box, enter Image Classification Based on Swin Transformer and press Enter. On the Image Classification Based on Swin Transformer card, click Open in DSW. The required resources and tutorial file are automatically downloaded to your DSW instance, and the tutorial file opens after the download is complete.

     In the easycv_classification_Swin.ipynb file, run the commands sequentially. Before installing dependencies, run the following code to get the CUDA and PyTorch versions of the current environment:

     import torch
     import os
     os.environ['CUDA']='cu'+torch.version.cuda.replace('.','')
     os.environ['Torch']='torch'+torch.version.__version__.replace('+PAI','')
     !echo "cuda version: $CUDA"
     !echo "pytorch version: $Torch"

     Expected output:

     [2023-04-13 18:56:06,771.771 dsw-234398-67dcbf944b-w5hf9:294 INFO utils.py:30] NOTICE: PAIDEBUGGER is turned off.
     cuda version: cu101
     pytorch version: torch1.8.2

3. Download the trained model to your local machine, and then upload it to OSS. For more information, see Upload and download objects and Upload objects.

Deploy the model service

With Elastic Algorithm Service (EAS), you can deploy a trained model as an online service. Perform the following steps:

1. Go to the Elastic Algorithm Service (EAS) page.

   1. Log on to the PAI console.

   2. In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the workspace that you want to manage.

   3. In the left-side navigation pane, choose Model Deployment > Elastic Algorithm Service (EAS).

2. Deploy the model service.

   1. On the Elastic Algorithm Service (EAS) page, click Deploy Service. In the Deploy a Model by Using Custom JSON section, click JSON Deployment.

   2. On the JSON Deployment page, enter the following configuration in the editor.

      {
          "cloud": {
              "computing": {
                  "instance_type": "ecs.gn6i-c16g1.4xlarge"
              }
          },
          "containers": [
              {
                  "image": "eas-registry-vpc.<region>.cr.aliyuncs.com/pai-eas/diffuser-inference:easyretrieval1.1.0-pytorch1.8.1-gpu-py309-cu111-ubuntu22.04",
                  "port": 8000,
                  "script": "python /ml/code/app.py --model_type imagetext_retrieval_cn"
              }
          ],
          "features": {
              "eas.aliyun.com/extra-ephemeral-storage": "100Gi"
          },
          "metadata": {
              "cpu": 16,
              "gpu": 1,
              "instance": 1,
              "memory": 62000,
              "name": "<your_service_name>"
          },
          "storage": [
              {
                  "mount_path": "/mnt/models",
                  "oss": {
                      "path": "oss://pai-quickstart-{region}/easycv/models/easyretrieval/",
                      "readOnly": true
                  },
                  "properties": {
                      "resource_type": "model"
                  }
              }
          ]
      }

      The following table describes the key parameters. Use the default values for other parameters. For more information about parameters, see JSON Deployment.

      Parameter		Description
      metadata	name	Specify a custom name for your service. The name must be unique within the same region.
      containers	image	Replace <region> with the ID of the region where your service is deployed. For example, the ID for China (Shanghai) is cn-shanghai. For a list of regions and availability zones, see Regions and availability zones.
      storage	oss.path	This solution uses the public pretrained model path provided by PAI. You only need to replace <region> with the ID of the region where your service is deployed. For example, the ID for China (Shanghai) is cn-shanghai. For a list of regions and availability zones, see Regions and availability zones. You can also use the pretrained models or the models you trained in DSW as described in the previous steps.
      cloud	computing.instance_type	The instance type. You must use a GPU-accelerated instance type.

   3. Click Deploy. Deployment is complete when the Service Status changes to Running.

3. View the service endpoint and token.

   1. On the Elastic Algorithm Service (EAS) page, click the name of the target service. In the basic information section, click View Calling Information.

   2. In the Calling Information panel, obtain the service endpoint and token. In the Calling Information dialog box, select the Shared Gateway or VPC Direct Connect tab to get the public endpoint, VPC endpoint, and token, which are required for service invocation.

Invoke the model service

Calling the service

The following is the complete code for a POST request.

import requests,json
head = {
    "Authorization":"xxxxxx"  # The authorization token for the service, which is provided by EAS after deployment.
}
our_oss_io_config = dict(ak_id='xxxxxx',
                         ak_secret='xxxxxx',
                         hosts='oss-*****-internal.aliyuncs.com',
                         buckets=['examplebucket'])
datas = json.dumps({
    "function_name": "XXX",  # Required. The name of the API to call.
    "function_params": {
        param1: XXX,  # For more information, see the parameter description for each API.
        param2:
        ....
    },
})
hosts = "xxxxxx/test"
r = requests.post(hosts, data=datas, headers=head)

The following table describes the key parameters.

Parameter	Description
Authorization	Set this to the service token that you obtained in the previous step.
our_oss_io_config.ak_id	The AccessKey ID of your Alibaba Cloud account.
our_oss_io_config.ak_secret	The AccessKey Secret of your Alibaba Cloud account.
our_oss_io_config.hosts	The endpoint for your OSS region. For more information about the mappings between OSS regions and endpoints, see Regions and Endpoints. Important You must use the internal endpoint for the corresponding region. Otherwise, the access will fail.
our_oss_io_config.buckets	The name of the OSS bucket that you created. The bucket must be in the same region as the service.
function_name	The name of the API. Valid values include: Service initialization APIs: init, set_root_path, set_oss_config, and set_predictor. Database management layer APIs: add, delete, get, load, and save. Database layer APIs: db_set, db_get, db_delete, and db_search.
function_params	The parameter configuration. For more information, see the API reference corresponding to the function_name. Chinese service initialization API reference Chinese service feature reference
hosts	Set this to the public endpoint of the service that you obtained in the previous step.

The following are examples of how to use the APIs:

• Service initialization API - set_root_path

  • Example POST request

    datas = json.dumps({
        "function_name" :"set_root_path",
        "function_params" : {
            "root_path": "oss://examplebucket/ljh-xiaoling/da_test/",
        },
    })

  • Example POST response (successful request)

    {
      "request_id": "b576c16e-7fe2-46d4-9502-35771*******",
      "success": True, 
      "info": "None", 
    }

• Database management layer API - save

  • Example POST request

    datas = json.dumps({
        "function_name" : "save",
        "function_params" : {},
    })

  • Example POST response (successful request)

    {
      "request_id": "f355c3d9-d235-41dd-b262-695966******",
      "success": True, 
      "save": "Done", 
    }

• Database layer API - db_search

  • Example POST request

    datas = json.dumps({
        "function_name": "db_search",
        "function_params": {
            "database_name": "test_1",
            "uri": "oss://examplebucket/ljh-xxxx/da_test/000001.jpeg",
            "search_topk": 2,
            "metric" : "cosine"
        },
    })

  • Example POST response (successful request)

    {
      "request_id": "70c2d344-adf9-4b07-b888-cabce0******",
      "success": True, 
      "db_search": [[{'scores': 0.4860914349555969, 'uri': 'oss://examplebucket/ljh-xiaoling/data/image/000015.jpg', 'text': '', 'group_id': '0', 'intra_id': 0}]]
    }

Example service calls

Using the multi-modal image retrieval service involves creating a database, registering data for feature extraction, and performing similarity searches. This process uses APIs for database initialization, adding a database, adding data, and searching data. The following example shows the basic flow.

1. Create the following Python scripts.

   • Database initialization script retrieval_init.py:

     import requests
     import json
     head = {
         "Authorization": "xxxxxx"  # The authorization token for the service, provided by EAS after deployment.
     }
     our_oss_io_config = dict(ak_id='xxxxxx',  # The AccessKey ID of your Alibaba Cloud account.
                              ak_secret='xxxxxx',  # The AccessKey Secret of your Alibaba Cloud account.
                              hosts='oss-cn-****-internal.aliyuncs.com',  # The internal endpoint.
                              buckets=['examplebucket'])  # The OSS bucket.
     datas = json.dumps({
         "function_name": "init",
         "function_params": {
             "backend": "oss",
             "root_path": "oss://examplebucket/xxx",
             "oss_io_config": our_oss_io_config,
         },
     })
     hosts = 'http://16640818xxxxxx.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/img_retrieval_001'
     r = requests.post(hosts, data=datas, headers=head)
     print(r.content)
     print("test ending")
     

     Replace examplebucket with the name of your OSS bucket. The bucket must be in the same region as the service. For more information about other parameters, see Calling the service.

   • Add database script retrieval_add.py:

     import requests
     import json
     ENCODING = 'utf-8'
     datas = json.dumps({
         'function_name': 'add',
         'function_params': {
             'database_name': 'test_1',
         },
     })
     hosts = 'http://16640818xxxxxx.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/img_retrieval_001'
     head = {
         "Authorization": "NTE3Y2M2ZmEzZGQ3ZGRkOGM4ZDE1MDxxxxxxxxxxxxxx"
     }
     r = requests.post(hosts, data=datas, headers=head)
     print(r.content)
     print("test ending")
                   

     Set hosts and Authorization to the service endpoint and token that you obtained in the previous step.

   • Set predictor script retrieval_set_predictor.py:

     import requests
     import json
     ENCODING = 'utf-8'
     datas = json.dumps({
         "function_name": "set_predictor",
         "function_params": {
             'preprocess': 'load_uri_to_base64str',
         },
     })
     hosts = 'http://16640818xxxxxx.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/img_retrieval_001'
     head = {
         "Authorization": "NTE3Y2M2ZmEzZGQ3ZGRkOGM4ZDE1MDxxxxxxxxxxxxxx"
     }
     r = requests.post(hosts, data=datas, headers=head)
     # r = requests.post("http://0.0.0.0:8000/test", data=datas, timeout=1500)
     print(r.content)
     print("test ending")
                   

     Set hosts and Authorization to the service endpoint and token that you obtained in the previous step.

   • Add data script retrieval_db_set.py:

     import requests
     import json
     ENCODING = 'utf-8'
     datas = json.dumps({
         'function_name': 'db_set',
         'function_params': {
             'database_name': 'test_1',
             'uri': [
                 'oss://examplebucket/ljh-xxxx/data/image/000015.jpg',
             ],
             'group_id': ['0'],  # The group name to which the data is saved.
             'embedding_attr': 'uri'
         },
     })
     hosts = 'http://16640818xxxxxx.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/img_retrieval_001'
     head = {
         "Authorization": "NTE3Y2M2ZmEzZGQ3ZGRkOGM4ZDE1MDxxxxxxxxxxxxxx"
     }
     r = requests.post(hosts, data=datas, headers=head)
     print(r.content)
     print("test ending")
     

     Set hosts and Authorization to the service endpoint and token that you obtained in the previous step. Set uri to the OSS path of the image.

   • Search data script retrieval_db_search.py:

     import requests
     import json
     ENCODING = 'utf-8'
     datas = json.dumps({
         'function_name': 'db_search',
         'function_params': {
             'database_name': 'test_1',
             'uri': 'oss://examplebucket/ljh-xxxx/da_test/000001.jpeg',
             'search_topk': 1,  # The number of most similar data entries to return.
             'metric': 'cosine'
         },
     })
     hosts = 'http://16640818xxxxxx.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/img_retrieval_001'
     head = {
         "Authorization": "NTE3Y2M2ZmEzZGQ3ZGRkOGM4ZDE1MDxxxxxxxxxxxxxx"
     }
     r = requests.post(hosts, data=datas, headers=head)
     print(r.content)
     print("test ending")
     

     Set hosts and Authorization to the service endpoint and token that you obtained in the previous step. Set uri to the OSS path of the image.

2. In your terminal, navigate to the script directory and run the initialization, add database, set predictor, and add data scripts sequentially.

   python <retrieval_xxx.py>

   Replace <retrieval_xxx.py> with the actual name of your Python script.

3. For the deployed database, run the search script to perform image retrieval.

   python retrieval_db_search.py

   The inference result is as follows:

   {"request_id": "d4b4c7c5-2330-49d5-8ffb-a15e490b****", "success": true, "info": [[{"scores": 0.0, "uri": "oss://examplebucket/xxx/000001.jpeg", "text": "", "group_id": "0", "intra_id": 0}]]}

   The service returns the URL of the most similar image from the database.