Education industry template: Multimodal search

Introduction

The OpenSearch multimodal search feature combines text search and vector search. It delivers higher accuracy than using OR logic alone, with lower latency and less computational overhead. The effectiveness of this approach has been proven in education Q&A search scenarios. The multimodal search architecture can also be used for other scenarios, such as vector-based image retrieval, formula retrieval, and personalized retrieval.

Text search vs. multimodal search

Why use multimodal search for photo-based Q&A search in education? Photo-based Q&A search in education differs significantly from standard web or e-commerce text search for the following reasons:

• Queries are long: Standard search queries often have a limit of 30 terms, but Q&A search queries may contain up to 100 terms.

• Queries can contain OCR errors: Queries are often generated from text extracted from images using optical character recognition (OCR). Recognition errors in key terms can severely affect result retrieval and ranking.

Text-only search solutions

1. OR logic queries:

• To reduce no-result queries, Q&A systems for education commonly use the default OR logic of Elasticsearch. This approach leads to high search latency and heavy computational load.

• OpenSearch also supports OR logic and can use parallel processing to reduce high latency, but the overall computational load remains high.

2. AND logic queries:

• Compared to OR logic, a generic query analysis module yields a higher rate of no-result queries and lower overall accuracy.

• A query analysis module that is optimized and customized for the education sector significantly improves performance, delivering accuracy that approaches that of OR logic.

Text and vector retrieval

Goal: The goal is to expand recall by using text vectorization combined with AND logic queries. This approach achieves higher accuracy than OR logic while maintaining lower search latency and computational overhead. Vector retrieval uses a BERT model with the following special optimizations for education Q&A search:

• The BERT model uses StructBERT, which is developed by Alibaba DAMO Academy and customized for the education industry.

• The vector search engine is the Proxima vector search engine, developed by the Alibaba DAMO Academy. It offers significantly higher accuracy and speed than open source alternatives.

• Training data can be continuously accumulated from customer search logs, leading to ongoing performance improvements.

Results:

1. The recall rate matches that of OR logic.

2. The accuracy is 3% to 5% higher than that of OR logic.

3. The total number of retrieved documents is reduced by a factor of 40, and search latency is reduced by a factor of more than 10.

Configure multimodal search

1. Configure an Industry-specific Enhanced Edition for education. After completing the configuration, verify that the vector index is configured. This example uses "Vector - K12 Education Vector". On the Index Structure page, the Index Fields list shows three indexes: default (the default text index, which includes the title and content fields and uses the "Industry - Education Q&A" analyzer, for example, query=default:"cloud search"), vector (the vector index, which includes the title field and uses the Vector - K12 Education Vector analyzer, for example, query=vector:"cloud search"), and id (includes the id field and uses the "Keyword" analyzer, for example, query=id:"cloud search"). The vector index and its corresponding Vector - K12 Education Vector analyzer are the key settings for implementing multimodal search. 2. Create a query analysis rule. Add the text vectorization feature and configure the vector index that you specified in Step 1. In the left-side navigation pane of the console, choose Search Algorithm Center > Query Analysis. Find the desired query analysis rule, such as sys_default, and click Edit. In the Edit Rule dialog box, set Industry Type to Education, Selected Category to Q&A, and Query Rewriting Policy to AND. In the Features section, select the Text Vectorization check box. From the Vector Index drop-down list, select vector and click OK. 3. Set a sort policy. By default, the Industry-specific Enhanced Edition for education creates two Cava script-based fine sort policies: sys_text_rank for the text index and sys_vector_rank for the vector index. In the left-side navigation pane of the console, choose Search Algorithm Center > Sort Configuration to go to the Policy Management tab. The policy list contains the system-built sys_text_rank and sys_vector_rank fine sort policies. Their status is Published.

4. Create a multimodal search policy. The procedure involves specifying a policy name, configuring multimodal search settings (vector and text indexes), and configuring merge sort settings (sort logic and the number of documents to sort). Select a Target Application and enter a Query Policy Name, such as test_strategy_1. In the Multimodal Search Configuration section, select both the Vector Search and Text Search check boxes. For vector search, set Fine Sort to sys_vector_rank and Result Proportion to 50%. For text search, set Fine Sort to sys_text_rank and the result proportion to 50%. In the Merge Sort Configuration section, set Merge Sort to sys_text_rank and Number of Documents to Sort to 5000. Then, click OK.

5. Test in the console:

Assume that the query is "search test", the multimodal search policy is "sys_strategy", and the query analysis rule is "sys_default". Note: All three parameters are required. You can omit the qp parameter if the corresponding query analysis rule is the default. The query request format is as follows:

query=default:'search test'&search_strategy=sys_strategy&raw_query=search test&qp=sys_default

SDK parameter configuration: (The following is an example in Java.)

...
// Define the Config object to set parameters for the config clause, such as pagination or response format.
Config config = new Config(Lists.<String>newArrayList(appName));
config.setStart(0);
config.setHits(10);
​
// Set the response format to JSON.
config.setSearchFormat(SearchFormat.FULLJSON);
​
​
// Create a SearchParams object.
SearchParams searchParams = new SearchParams(config);
​
// Set the parameters required for the query.
HashMap<String,String> paraMap=new HashMap<String,String>();
// Set raw_query.
paraMap.put("raw_query","search test");
// Set the multimodal search policy.
paraMap.put("search_strategy","sys_strategy");
searchParams.setCustomParam(paraMap);
​
// Set the query analysis rule.
List<String> qpName = new ArrayList<String>();
qpName.add("sys_default");  // Set the name of the query analysis rule.
searchParams.setQueryProcessorNames(qpName);
​
...

Limitations

• The multimodal search feature is available only for exclusive applications.

• The multimodal search feature does not support the aggregate or distinct clause.

• You can create a maximum of 10 multimodal search policies for each application.

• In the multimodal search configuration, both Text Search and Vector Search must be enabled, and the sum of their result proportions must be 100%.

• In the multimodal search configuration, select a rough sort policy and a fine sort policy for both Text Search and Vector Search from those available in Sort Configuration.

• In the merge sort configuration, Merge Sort is a required parameter. You must select a fine sort policy. You can also select "None", which is the default selection. Number of Documents to Sort is a required parameter. You must enter a value that is greater than 0 and no more than 5,000.

• For the text vectorization feature in query analysis, you can only use vector indexes that are automatically generated by the system.

• To use the multimodal search feature with a custom vector index, contact us.