Term weight intervention dictionaries

Introduction

You can use the term weight feature to create a term weight intervention dictionary and apply it in query analysis to adjust term weights. This lets you manually intervene in the system's built-in term weight dictionary. The intervention process consists of four steps:

1. Create a term weight intervention dictionary. Go to Search Algorithm Center > Retrieval Configuration > Dictionary Management. On the Query Analysis Intervention Dictionaries page, click Create in the upper-right corner. Select a dictionary type and enter a name for the dictionary. The new dictionary is created and appears in the dictionary list.

2. Add intervention entries. After the dictionary is created, click its name or Manage Entries in the list to open the details page. On this page, you can add and manage intervention entries. Select an analyzer to tokenize the query. Available analyzers include the General Analyzer, E-commerce Analyzer, IT Content Analyzer, and custom analyzers based on these three. After tokenization, you can adjust the weight of each term. You can set the weight to High, Medium, or Low.

3. Use the intervention dictionary. After you create the term weight intervention dictionary and add entries, you can select it in the query rules for an application.

4. Test and publish the dictionary. After you apply the intervention dictionary to a query analysis rule, test the search results to see if they meet your expectations. Then, publish the changes.

Activation rules

Intervention data takes effect based on the following rules:

1. An exact match of the full query has the highest priority.

2. Priority is given based on the match position.

3. If multiple entries match at the same starting position, the longest match has priority. Length is measured by the number of tokenized terms. Each term has a length of 1. The maximum match length is 5 terms. For example, the query "MySQL database" is tokenized into two terms: "MySQL" and "database". The length is 2.

4. When you configure a term weight intervention dictionary for query analysis, you can choose whether to ignore spaces in the query.

Example

• The intervention data is as follows:

  1. database permission management -> database: 7 | permission: 4 | management: 1

  2. MySQL database -> MySQL: 7 | database: 1

  3. database permission -> database: 4 | permission: 1

  4. MySQL database permission management in linux environment -> linux: 7 | environment: 1 | in: 1 | of: 1 | MySQL: 7 | database: 4 | permission: 1 | management: 1 (The match length of this entry exceeds the 5-term limit)

• Search queries:

  • MySQL database permission management => Rule b takes effect (position priority).

  • sqlserver database permission management => Rule a takes effect (for the same position, length has priority).

  • how to set database permission => Rule c takes effect (position priority).

  • database set permission => No intervention rule takes effect (no match).

  • MySQL database management permission settings in linux environment => Rule d takes effect (exact match of the full query).

  • guide to MySQL database management permission settings in linux environment => Rule b takes effect. Although rule d is part of this query, it is not matched because it contains more than 5 tokenized terms.

  • MySQL database's database permission settings => Rules b and c both take effect.

Walkthrough

Scenario: A content provider uses a query analysis rule with the term weight feature in their OpenSearch application instance. They discover an issue with their online search results and decide to use the intervention feature.

Issue: A user searches for the query data permission management. The rewritten query is default:'permission' RANK default:'data' RANK default:'management'. However, the user intended for "data" to have higher relevance than "permission".

Diagnosis: The built-in term weight feature caused this issue. This can be resolved using a term weight intervention.

Solution: Create a new term weight intervention dictionary and apply it to the query analysis rule that is used in the online environment.

Procedure:

1. In the console, go to Feature Extensions > Dictionary Management and create a term weight dictionary.

Enter a Name, select Term Weight for the dictionary type, and click Save.

2. In the new intervention dictionary, add an intervention entry. Enter the query, select System Built-in Analyzer/Custom Analyzer for the analyzer type based on the type configured in query analysis, and set the weights for the tokenized terms.

If the index configured in query analysis uses a custom model analyzer, set Analyzer Type to Custom Model Analyzer, set Instance to Ha3 Engine, and select the corresponding Application Name. Ensure that the selected analyzer is the same as the one configured in query analysis.

3. In the query analysis interface, apply the newly created term weight intervention dictionary to a query analysis rule in a test environment to verify the results.

4. Test the search to verify that the results meet your expectations. The weight of "data" is increased:

Create a term weight intervention dictionary using an SDK

Java SDK: Maven dependencies:

<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>aliyun-java-sdk-opensearch</artifactId>
    <version>0.7.0</version>
</dependency>
<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>aliyun-java-sdk-core</artifactId>
    <version>4.5.0</version>
</dependency>

Java SDK demo:

public class TestTermWeightingInQueryProcessor {
    private static DefaultAcsClient client;
    public static void main(String[] args) throws Exception {
        String regionId = "cn-hangzhou"; // The region ID.
        IClientProfile profile = DefaultProfile.getProfile(regionId, "{ak}", "{secret}");
        DefaultProfile.addEndpoint(regionId, regionId, "Opensearch", "opensearch." + regionId + ".aliyuncs.com");
        DefaultAcsClient client = new DefaultAcsClient(profile);
        String dictionaryName = "dictionary_qp_name"; // The name of the dictionary and query processor to create.
        String appName = "app_name"; // The name of the application to use.
        int versionId = 1234;  // The application version ID.
//        System.out.println("List intervention dictionaries");
//        listInterventionDictionaries();
        Thread.sleep(10000);
        System.out.println("Create intervention dictionary: " + dictionaryName);
        createDictionary(dictionaryName);
//
//        Thread.sleep(10000);
//        System.out.println("Describe intervention dictionary");
//        describeInterventionDictionary(dictionaryName);
//
//        Thread.sleep(10000);
//        System.out.println("List intervention dictionary entries before");
//        listEntries(dictionaryName);
        Thread.sleep(10000);
        System.out.println("Post dictionary entries added");
        postEntries(dictionaryName, "add");
        Thread.sleep(10000);
        System.out.println("List intervention dictionary entries after add");
        listEntries(dictionaryName);
        Thread.sleep(10000);
        System.out.print("Set intervention dictionary to qp");
        setQueryProcessor(appName, versionId, dictionaryName);
        // Use with caution. Set the current query processor as the default only after you confirm that it meets expectations on the search test page.
        Thread.sleep(10000);
        System.out.println("Set default query processor");
        setDefaultQueryProcessor(appName, versionId, dictionaryName);
//        Thread.sleep(10000);
//        System.out.println("Delete dictionary");
//        deleteDictionary(dictionaryName);
    }
    public static void listInterventionDictionaries() throws ClientException {
        ListInterventionDictionariesRequest listInterventionDictionariesRequest = new ListInterventionDictionariesRequest();
        listInterventionDictionariesRequest.setPageSize(50);
        HttpResponse response = client.doAction(listInterventionDictionariesRequest);
        System.out.println(response.getHttpContentString());
    }
    public static void createDictionary(String dictionaryName) throws UnsupportedEncodingException, ClientException {
        CreateInterventionDictionaryRequest request = new CreateInterventionDictionaryRequest();
        String body = "{\"name\": \"" + dictionaryName + "\", \"type\": \"term_weighting\"}";
        request.setHttpContent(body.getBytes("UTF-8"), "UTF-8", FormatType.JSON);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void describeInterventionDictionary(String dictionaryName) throws ClientException {
        DescribeInterventionDictionaryRequest request = new DescribeInterventionDictionaryRequest();
        request.setName(dictionaryName);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void listEntries(String dictionaryName) throws ClientException {
        ListInterventionDictionaryEntriesRequest request = new ListInterventionDictionaryEntriesRequest();
        request.setName(dictionaryName);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void postEntries(String dictionaryName, String cmd) throws UnsupportedEncodingException, ClientException {
        PushInterventionDictionaryEntriesRequest request = new PushInterventionDictionaryEntriesRequest();
        request.setName(dictionaryName);
        // Modify the intervention entry data. The weights for High, Medium, and Low are 7, 4, and 1.
        String body = "[{\n" +
                "  \"word\": \"data permission management\",\n" +
                "  \"cmd\": \"" + cmd + "\",\n" +
                "  \"tokens\": [\n" +
                "    {\n" +
                "      \"token\": \"data\",\n" +
                "      \"weight\": 7\n" +
                "    },\n" +
                "    {\n" +
                "      \"token\": \"permission\",\n" +
                "      \"weight\": 4\n" +
                "    },\n" +
                "    {\n" +
                "      \"token\": \"management\",\n" +
                "      \"weight\": 1\n" +
                "    }\n" +
                "  ]\n" +
                "}]";
        request.setHttpContent(body.getBytes("UTF-8"), "UTF-8", FormatType.JSON);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void deleteDictionary(String dictionaryName) throws ClientException {
        RemoveInterventionDictionaryRequest request = new RemoveInterventionDictionaryRequest();
        request.setName(dictionaryName);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void setQueryProcessor(String appName, int versionId, String dictionaryName) throws UnsupportedEncodingException, ClientException {
        CreateQueryProcessorRequest request = new CreateQueryProcessorRequest();
        request.setAppGroupIdentity(appName);
        request.setAppId(versionId);
        
        String body = "{\"name\":\""+  dictionaryName +"\",\"domain\":\"GENERAL\",\"indexes\":[\"default\"],\"processors\":[{\"name\":\"term_weighting\",\"useSystemDictionary\":true, \"interventionDictionary\":\""+dictionaryName+"\"}]}";
        request.setHttpContent(body.getBytes("UTF-8"), "UTF-8", FormatType.JSON);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void setDefaultQueryProcessor(String appName, int versionId, String dictionaryName) throws UnsupportedEncodingException, ClientException {
        ModifyQueryProcessorRequest request = new ModifyQueryProcessorRequest();
        request.setAppGroupIdentity(appName);
        request.setAppId(versionId);
        request.setName(dictionaryName);
        String body = "{\"active\":true}";
        request.setHttpContent(body.getBytes("UTF-8"), "UTF-8", FormatType.JSON);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void deleteQueryProcessor(String appName, int versionId, String dictionaryName) throws ClientException {
        RemoveQueryProcessorRequest request = new RemoveQueryProcessorRequest();
        request.setAppGroupIdentity(appName);
        request.setAppId(versionId);
        request.setName(dictionaryName);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
    public static void listQueryProcessors(String appName, int versionId) throws ClientException {
        ListQueryProcessorsRequest request = new ListQueryProcessorsRequest();
        request.setAppGroupIdentity(appName);
        request.setAppId(versionId);
        HttpResponse response = client.doAction(request);
        System.out.println(response.getHttpContentString());
    }
}

Limits

• You can create a maximum of 20 term weight intervention dictionaries.

• Each term weight intervention dictionary can contain a maximum of 500 intervention entries.

• If a query contains English words, you must use all lowercase letters.

• A query in an intervention entry can be tokenized into a maximum of 30 terms.