Write and query automatic embeddings

Prerequisites

• The vector engine must be enabled. For instructions, see Enable vector engine.

• The search engine must be enabled, and its version must be 3.9.10 or later. To enable the engine, see the Activation guide. To view or upgrade your current version, see Search engine version information and Upgrade a minor version.

  Important

  If your search engine version is earlier than 3.9.10 but the console indicates that it is the latest version, contact Lindorm technical support (DingTalk ID: s0s3eg3).

• The AI engine must be enabled. For instructions, see the Activation guide. If you need to use Alibaba Cloud Model Studio models, select the Dashscope model option during activation. If this option is not available, contact Lindorm technical support (DingTalk ID: s0s3eg3).

  Note

  • The AI engine depends on the wide table engine, which must also be enabled.

• Your client IP address must be added to the Lindorm whitelist. For instructions, see Configure a whitelist.

Procedure overview

Deploy embedding model (Optional)

For instructions on deploying models in the AI engine, see Model Management and Examples of Using the AI engine RESTful API with curl commands.

The following example shows how to deploy the BGE_VISUALIZED model. For parameter details, see Model Management.

curl -i -k --location --header 'x-ld-ak:<username>' --header 'x-ld-sk:<password>' -X POST http://<URL>/v1/ai/models/create  -H "Content-Type: application/json" -d '{
          "model_name": "bge_visualized_model",
          "model_path": "huggingface://BAAI/bge-visualized",
          "task": "FEATURE_EXTRACTION",
          "algorithm": "BGE_VISUALIZED_M3",
          "settings": {"instance_count": "2"}
     }'

Create pipelines in the search engine

Create two types of pipelines in the search engine to automate vectorization for data ingestion and queries.

curl -u <username>:<password> -H "Content-Type: application/json" -XPUT "http://<URL>/_ingest/pipeline/<ingest_pipeline_name>" -d '{
  "description": "demo embedding pipeline",
  "processors": [
    {
      "text-embedding": {
        "inputFields": ["input_field"], 
        "outputFields": ["embedding_field"],
        "userName": "root",
        "password": "test****",
        "url": "http://ld-xxxx-proxy-ai-vpc.lindorm.aliyuncs.com:9002/dashscope/compatible-mode/v1/embeddings",
        "modeName": "text-embedding-v4"
      }
    }
  ]
}'

curl -u <username>:<password> -H "Content-Type: application/json" -XPUT "http://<URL>/_search/pipeline/<search_pipeline_name>" -d '{
  "request_processors": [
    {
      "text-embedding" : {
        "tag" : "auto-query-embedding",
        "description" : "Auto query embedding",
        "model_config" : {
          "inputFields": ["text_field"],
          "outputFields": ["text_field_embedding"],
          "userName": "root",
          "password": "test****",
          "url": "http://ld-xxxx-proxy-ai-vpc.lindorm.aliyuncs.com:9002/dashscope/compatible-mode/v1/embeddings",
          "modeName": "text-embedding-v4"
        }
      }
    }
  ]
}'

curl -u <username>:<password> -H "Content-Type: application/json" -XPUT "http://<URL>/_ingest/pipeline/<ingest_pipeline_name>" -d '{
  "description": "demo embedding pipeline",
  "processors": [
    {
      "multimodal-embedding": {
        "input_fields": ["input_field"], 
        "output_fields": ["embedding_field"], 
        "user_name": "root",
        "password": "test****",
        "url": "http://ld-xxxx-proxy-ai-vpc.lindorm.aliyuncs.com:9002/dashscope/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding",
        "model_name": "tongyi-embedding-vision-plus",
        "input_type": "image"
      }
    }
  ]
}'

curl -u <username>:<password> -H "Content-Type: application/json" -XPUT "http://<URL>/_search/pipeline/<search_pipeline_name>" -d '{
  "description": "demo embedding pipeline",
  "request_processors": [
    {
      "multimodal-embedding": {
        "model_config" : {
          "input_fields": ["input_field"], 
          "output_fields": ["embedding_field"], 
          "user_name": "root",
          "password": "test****",
          "url": "http://ld-xxxx-proxy-ai-vpc.lindorm.aliyuncs.com:9002/dashscope/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding",
          "model_name": "tongyi-embedding-vision-plus"
        }
      }
    }
  ]
}'

Create an index and specify a pipeline

When creating or modifying a vector index, you must specify the required pipeline.

curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://<URL>/<index_name>"  -d '
{
 "settings" : {
    "index": {
      "number_of_shards": 2,
      "knn": true,
      "default_pipeline": <ingest_pipeline_name>,
      "search.default_pipeline": <search_pipeline_name>
    }
  },
  "mappings": {
    "_source": {
      "excludes": ["embedding_field"]
    },
    "properties": {
      "input_field": {
        "type": "text",
        "analyzer": "ik_max_word" 
      },
      "embedding_field": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "engine": "lvector",
          "name": "hnsw", 
          "space_type": "cosinesimil",
          "parameters": {
            "m": 24,
            "ef_construction": 500
         }
       }
      },
      "tag": {
          "type": "keyword"
      },
      "brand": {
          "type": "keyword"
      },
      "merit" : {
        "type": "text",
        "analyzer": "ik_max_word"  
      }
    }
  }
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XPUT "http://<URL>/<index_name>/_settings"  -d '
{
  "index": {
       "default_pipeline": <ingest_pipeline_name>,
       "search.default_pipeline": <search_pipeline_name>
    }
}
'

Write data

Because a write pipeline is specified, the write process not only writes the scalar field input_field, but also uses the pipeline to encode input_field into a vector, which is then written as the embedding_field.

curl -u <username>:<password> -H 'Content-Type: application/json' -XPOST "http://<URL>/_bulk?pretty"  -d '
{ "index" : { "_index" : "<index_name>", "_id" : "3982" } }
{ "input_field" : "Brand A Stylish Power-saving Wireless Mouse (Green) (12-month battery life, high-precision optical engine, 10m range)", "tag": ["Mouse", "Electronics"], "brand":"Brand A", "merit":"easy to use, stylish design"}
{ "index" : { "_index" : "<index_name>", "_id" : "323519" } }
{ "input_field" : "Brand B Optical Mouse (Black) (auto-pairing, 1000 DPI optical engine)", "tag": ["Mouse", "Electronics"], "brand":"Brand B", "merit":"good quality, fast delivery, stylish design, easy to use"}
{ "index" : { "_index" : "<index_name>", "_id" : "300265" } }
{ "input_field" : "Brand C In-ear Headphones (White) (classic and stylish)", "tag": ["Headphones", "Electronics"], "brand":"Brand C", "merit":"stylish design, good quality"}
{ "index" : { "_index" : "<index_name>", "_id" : "6797" } }
{ "input_field" : "Brand D Dual-head rechargeable electric shaver", "tag": ["Home Appliances", "Electric Shaver"], "brand":"Brand D", "merit":"easy to use, stylish design"}
{ "index" : { "_index" : "<index_name>", "_id" : "8195" } }
{ "input_field" : "Brand E Class 4 32GB microSD Card for mobile phones", "tag": ["Storage", "Memory Card", "SD Card"], "brand":"Brand E", "merit":"large capacity, fast, easy to use, good quality"}
{ "index" : { "_index" : "<index_name>", "_id" : "13316" } }
{ "input_field" : "Brand E 101 G2 32GB USB Drive", "tag": ["Storage","USB Drive"], "brand":"Brand E", "merit":"easy to use, large capacity, fast"}
{ "index" : { "_index" : "<index_name>", "_id" : "14103" } }
{ "input_field" : "Brand B 64GB Extreme High-Speed Mobile Memory Card (UHS-1, up to 30 MB/s read/write speed)", "tag": ["Storage", "Memory Card", "SD Card"], "brand":"Brand B", "merit":"large capacity, fast, easy to use"}
'

Data query

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-xx-proxy-search-vpc.lindorm.aliyuncs.com:30070/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "_source": true,
  "query": {
    "knn": {
      "embedding_field": {
        "query_text": "memory card",
        "k": 10
      }
    }
  },
  "ext": {
    "lvector": {
      "ef_search": "200"
    }
  }
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-xx-proxy-search-vpc.lindorm.aliyuncs.com:30070/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "_source": true,
  "query": {
    "knn": {
      "embedding_field": {
        "query_image": "https://img.alicdn.com/imgextra/i2/O1CN019eO00F1HDdlU4Syj5_!!6000000000724-2-tps-2476-1158.png",
        "k": 10
      }
    }
  },
  "ext": {
    "lvector": {
      "ef_search": "200"
    }
  }
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-xx-proxy-search-vpc.lindorm.aliyuncs.com:30070/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "_source": true,
  "query": {
    "knn": {
      "embedding_field": {
        "query_text": "memory card",
        "k": 10,
        "filter": {
          "bool": {
            "filter": [{
              "match": {
                "merit": "good quality"
              }
            },
            {
              "term": {
                "brand": "Brand E"
              }
            },
            {
              "terms": {
                "tag": ["SD card", "memory card"]
              }
            }]
          }
        }
      }
    }
  },
  "ext": {
    "lvector": {
      "filter_type": "efficient_filter",
      "ef_search": "200"
    }
  }
}'

curl -u <username>:<password> -H 'Content-Type: application/json' -XGET "http://ld-xx-proxy-search-vpc.lindorm.aliyuncs.com:30070/<index_name>/_search?pretty"  -d '
{
  "size": 10,
  "_source": true,
  "query": {
    "knn": {
      "embedding_field": {
        "query_text": "memory card",
        "filter": {
          "bool": {
            "must": [{
              "bool": {
                "must": [{
                  "match": {
                    "input_field": {
                      "query": "memory card"
                    }
                  }
                }]
              }
            },
            {
              "bool": {
                "filter": [{
                  "match": {
                    "merit": "good quality"
                  }
                },
                {
                  "term": {
                    "brand": "Brand E"
                  }
                },
                {
                  "terms": {
                    "tag": ["SD card", "memory card"]
                  }
                }]
              }
            }]
          }
        },
        "k": 10
      }
    }
  },
  "ext": {
    "lvector": {
      "filter_type": "efficient_filter",
      "hybrid_search_type": "filter_rrf",
      "rrf_rank_constant": "1",
      "ef_search": "200"
    }
  }
}'