Text - custom analyzers

Tokenization is a fundamental and important component of a search engine. The tokenization results directly affect search performance. Because business scenarios vary, the same phrase can have different semantics in different services and contexts. This means that the expected tokenization results also differ. For this reason, OpenSearch provides basic analyzers for general use. It also provides analyzers for specific fields, such as an E-commerce analyzer for the E-commerce realm.

To better meet your business needs, OpenSearch lets you create custom analyzers. You can build them on the system's basic analyzers and add intervention entries. You can select the corresponding analyzer for the index fields of your application. This lets you influence the tokenization results for indexing and queries and ensures the quality of your search results.

You can manage intervention entries with the secondary tokenization feature.

Enabling secondary tokenization means that after custom tokenization, the results are tokenized again. Disabling secondary tokenization preserves the results of your custom tokenization.

For example, if you use the General-purpose Chinese analyzer on the term "OpenSearch", the result of enabling secondary tokenization is:

The result of disabling secondary tokenization is:

• The entries in a custom analyzer are a combination of all entries from the base analyzer and any manually added entries. Manually added intervention entries have a higher priority than the entries in the default analyzer.

• You can create a maximum of 20 custom analyzers in the new console.

• A single custom analyzer can contain a maximum of 1,000 intervention entries.

• In each entry, the key can be up to 10 characters long, and the value can be up to 32 characters long. One character can be one Chinese character or one English letter.

• Entry content cannot contain uppercase letters (A-Z), full-width symbols (\uff01 - \uff5e), or Chinese punctuation marks.

• For semantic chunking intervention entries, the key and value must be the same after you remove spaces. For example:

  Incorrect entry=>Erroneous entry
  Correct entry=>Correct entry

  In the first entry, the content of the key and value are different after you remove the spaces. Therefore, the entry is not valid.

• The key cannot contain spaces. For example:

  In correct entry=>In correct entry
  CorrectEntry=>Correct entry

  The key in the first entry contains a space (" "). Therefore, the entry is not valid.

• The content of a key cannot be one of the resulting tokens from the value of another entry in the same intervention dictionary. For example:

  custom_tokenizer=>custom tokenizer
  tokenizer
  tokenization

  The key of the second entry, "tokenizer", is one of the resulting tokens in the value of the first entry. Therefore, the second entry is not valid. However, the third entry is valid.

Flow summary

Create a custom analyzer → Make offline changes → Rebuild the index → View the effect of the custom analyzer

Procedure

1. In the OpenSearch console, go to the homepage. In the navigation pane on the left, choose Search Configuration Center > Retrieval Configuration > Analyzer Management, and then click Create.

2. Create an analyzer. Specify the analyzer name and select an analyzer type.

3. Add an intervention entry. Enter the query and the desired tokenization result. This example uses "nuomi" (glutinous rice) and enables secondary tokenization.

Note: Separate tokens with a space. For example: "nuomi" ==> "nuo mi".

4. Test the tokenization to see the result of the intervention entry after the analyzer takes effect.

• 4.1. Enter "nuomi" in the test text box.

• 4.2. Compare the tokenization results from multiple custom analyzers.

5. After you test the tokenization, go back to Basic Configuration under Retrieval Configuration to apply changes to the online application.

Note: "Make offline changes" here means generating an offline application based on your current configuration. The online application is not affected.

6. In the list of application fields, find the Analysis Method column and select the custom analyzer for the corresponding index.

7. Wait for the index to be rebuilt for the changes to take effect.

Custom analyzer effect demo

Take a document that contains "nuomi" (glutinous rice) as an example. If you use the General-purpose Chinese tokenizer, the result is unexpected. A search for "mi" (rice) cannot retrieve documents that contain "nuomi" (glutinous rice), "xiaomi" (millet), or "dami" (rice), as shown in the following figure: If you follow the procedure to add the "test_zw" custom analyzer and modify the application schema, the resulting terms match the intervention after the index is rebuilt, as shown in the following figure:

Notes

• In the new console, you can add entries to existing custom analyzers. If an application already references a custom analyzer, you must rebuild the index for the new entries to take effect. To make the changes take effect immediately, you can re-upload the documents that have tokenization issues. This action triggers an index rebuild for the new intervention entries.

• The query for a custom analyzer cannot exceed 10 characters in length.

• The query for a custom analyzer cannot contain uppercase letters, full-width symbols, or Chinese punctuation.

• The custom tokenization result cannot contain uppercase letters, full-width symbols, or Chinese punctuation.

• If you disable the secondary tokenization switch, the system strictly follows your intervention settings and does not perform further tokenization. If you enable it, the system performs further tokenization on your intervention results.

• You cannot delete a custom analyzer that is referenced by an application.