E-commerce template: Multimodal search

Introduction

The multimodal search feature in OpenSearch combines text search and vector search to deliver higher accuracy with lower latency and compute costs than searches using OR logic. This approach has proven effective in scenarios such as question answering. The multimodal search architecture can also be applied to image vector retrieval, formula retrieval, and personalized retrieval.

Why use multimodal search?

• Retrieves relevant documents that traditional text searches might miss due to limitations in tokenization, synonym matching, or spelling correction.

• Improves recall for long-tail queries that would otherwise return few or no results.

Example:

A user searches for "fast charge". A standard tokenizer might break this into two terms: fast and charge.

If a document contains the phrase "supports quick charging", its tokens might be supports, quick, and charging. A text search based on token matching would fail to retrieve this document. However, a vector search operates on semantic meaning, not just exact words. It would recognize the close relationship between "fast charge" and "quick charging" and successfully retrieve the document.

Multimodal search configuration

1. Configure an Industry-specific Enhanced Edition for E-commerce application. Once the application is created, verify that the required vector index is configured. The example in the following steps uses the Vector - General E-commerce Vector analyzer.

In the index field list, verify that a vector index exists. It must be tagged as a vector index, include the title field, and use the Vector - General E-commerce Vector analyzer.

2. Create a query analysis rule. Add the text vectorization feature and link it to the vector index from Step 1.

In the left-side navigation pane, click Query Analysis. In the query analysis settings, click the rule that you want to modify. The Edit Rule dialog box opens. In the Feature Selection section, select the text vectorization checkbox. Then, on the right, set Vector index to vector and click OK.

3. Set up sort policies. By default, the Industry-specific Enhanced Edition for E-commerce template creates two sort policies: sys_first_default for rough sort and sys_second_default for fine sort.

On the Sort Policies > Policy Management page, you can view these two sort policies and their ranking expressions.

4. Create a multimodal search policy. Name the policy, configure the search paths, and set up the combined ranking.

In the Configure Search Policy dialog box, set the Search Policy Name to sys_strategy. In the Multimodal Search Configuration area, select the vector search and text search checkboxes, and set the Recall Ratio for both to 50%. In the Merge Sort Configuration area, set merge sort to sys_second_default, set the Number of Sort Documents to 500, and then click OK.

5. Test in the console:

Assume the query is "search test", the multimodal search policy is sys_strategy, and the query analysis rule is sys_default. Go to the Search Test page in the console. In the Parameters section, set the search_strategy, raw_query, and qp fields accordingly. Note: All three parameters are required, but you can omit the qp parameter if you use the default query analysis rule. The request format is:

query=default:'search test'&search_strategy=sys_strategy&raw_query=search test&qp=sys_default

SDK parameter configuration (Java example):

...
   
// Define a Config object to set parameters for the config clause, such as pagination and result format.
Config config = new Config(Lists.<String>newArrayList(appName));
config.setStart(0);
config.setHits(10);
​
// Set the result format to JSON.
config.setSearchFormat(SearchFormat.FULLJSON);
​
​
// Create a SearchParams object.
SearchParams searchParams = new SearchParams(config);
​
// Set the parameters required for the query.
HashMap<String,String> paraMap=new HashMap<String,String>();
// Set the raw_query.
paraMap.put("raw_query","search test");
// Set the multimodal search policy.
paraMap.put("search_strategy","sys_strategy");
searchParams.setCustomParam(paraMap);
​
// Set the query analysis rule.
List<String> qpName = new ArrayList<String>();
qpName.add("sys_default");  // Specify the name of the query analysis rule.
searchParams.setQueryProcessorNames(qpName);
​
...

Notes

• The multimodal search feature is available only for exclusive applications.

• The multimodal search feature does not support aggregate or distinct clauses.

• You can create a maximum of 10 multimodal search policies per application.

• In the multimodal search configuration, both text search and vector search are required. The sum of their recall ratios must be 100%.

• In the multimodal search configuration for text search and vector search, you must select a rough sort policy and a fine sort policy from your sort configuration.

• In the combined ranking configuration, you must select a policy for merge sort. You can select a fine sort policy or "None". "None" is the default. The number of documents to be sorted must be an integer from 1 to 5,000.

• When configuring the text vectorization feature in query analysis, you cannot select a vector index that was not automatically generated by the system.

• To use a custom vector index with the multimodal search feature, contact us.