Custom analyzers

Overview

Analysis directly impacts search performance. Because the semantics of a phrase can differ across business scenarios, different contexts require different analysis behavior. OpenSearch provides both general-purpose and domain-specific analyzers, such as an e-commerce analyzer.

OpenSearch lets you create a custom analyzer by combining a built-in analyzer with intervention entries. You can then apply the custom analyzer to the index fields of your application to control analysis results at both indexing and query time.

Intervention entries

You can control intervention entry behavior by enabling or disabling secondary analysis.

When secondary analysis is enabled, the analyzer further segments each intervention entry result. When disabled, the intervention result is treated as a final output and is not segmented further.

For example, if the entry is 'Open Search' and the analyzer is 'General Chinese', the result of enabling secondary analysis is:

"Open Search" is segmented into two entries: Open and Search.

With secondary analysis disabled, the result is:

'OpenSearch' is treated as a single entry, OpenSearch, without further segmentation.

Usage notes

• A custom analyzer includes all entries from its base analyzer, plus your custom intervention entries. The intervention entries override the base analyzer's default entries.

• You can create a maximum of 20 custom analyzers.

• Each custom analyzer can contain a maximum of 1,000 intervention entries.

• An entry's query cannot exceed 10 characters (Chinese or English), and its value cannot exceed 32 characters.

• Entries cannot contain uppercase letters (A-Z), full-width characters (\uff01 to \uff5e), or Chinese punctuation.

• For semantic segmentation intervention entries, the query and the value must be identical after spaces are removed from the value. For example:

  invalidkey=>wrong key
  validkey=>valid key

  The first entry is invalid because its query invalidkey and value wrong key (which becomes wrongkey after removing spaces) are not identical.

• When adding an intervention entry, use spaces to separate multiple terms in the analysis result. Do not use commas. Commas are treated as part of the term content, not as delimiters, which causes validation to fail. For example:

  applehuawei=>apple huawei (Correct, separated by a space)
  applehuawei=>apple,huawei (Incorrect, the comma is treated as a content character)

  In the second entry, the comma is treated as a literal character, making the value apple,huawei after space removal, which does not match the query applehuawei. Validation fails.

• The query of an entry cannot contain spaces. For example:

  invalid key=>an invalid key
  validkey=>a valid key

  The first entry is invalid because its query, invalid key, contains a space.

• The query of an entry cannot be a substring of the value of another entry within the same intervention dictionary. For example:

  customanalyzer=>custom analyzer
  analyzer
  token

  The second entry, with the query analyzer, is invalid because its query is part of the value (custom analyzer) of the first entry. The third entry, token, is valid.

Procedure

Workflow overview

Create a custom analyzer -> Perform an offline modification -> Reindex -> Verify the results

Steps

1. In the left-side navigation pane of the OpenSearch console, go to Search Algorithm Center > Retrieval Configuration > Analyzer Management, and then click Create.

2. Create an analyzer, define a name, and select a type:

In the dialog box that appears, for Analyzer Type, select Built-in Analyzer or Custom Model Analyzer, and from the Analyzer dropdown list, select a specific analyzer, such as Chinese - General Analysis.

3. Add an intervention entry, enter the query (for example, "sticky rice") and the analysis result, and select secondary analysis:

On the Entry Management page, click Add. In the Add Intervention Entry dialog box, enter a Query and the Analysis Result, enable Secondary Analysis, and click Save. Note: Separate words with spaces. Example: "sticky rice" ==> "sticky rice".

4. Test the analysis to verify the result of the intervention entry:

In the analyzer list on the Analyzer Management page, find the target custom analyzer and click Analysis Test in the Actions column.

• 4.1. In the test text, enter "sticky rice".

After testing with the test_zw analyzer, the analysis result shows that "sticky rice" is split into the two entries sticky and rice.

• 4.2. Compare the analysis results from multiple custom analyzers.

The comparison results show that test_zw segments "sticky rice" into sticky and rice, whereas kevintest2 keeps sticky rice as a complete entry.

5. After testing the analysis effect, return to Basic Configuration under Retrieval Configuration to apply the changes to your application:

In the Analysis Configuration area, click the Offline Modification button.

Note: An offline modification generates a new offline application version based on your current configuration. Your live application is not affected.

6. In the list of index fields, select your custom analyzer from the Analysis Method column for the relevant index field.

7. The changes take effect after reindexing is complete.

Demonstrating the custom analyzer's effect

For example, when a document contains "glutinous rice", the "Chinese - General" analyzer may produce unexpected results: a search for "rice" fails to retrieve documents containing "glutinous rice", "millet", or "rice". After you add the "test_zw" custom analyzer, modify the application structure, and reindex as described above, the resulting terms match your custom analysis.

Additional notes

• You can add entries to a custom analyzer that is already in use by an application, but you must reindex for the new entries to take effect. To apply the changes to specific documents, re-upload them. This action reindexes only those documents, applying the new intervention entries.

• The query of an entry cannot exceed 10 characters.

• The query of an entry cannot contain uppercase letters, full-width characters, or Chinese punctuation.

• The analysis result of an entry cannot contain uppercase letters, full-width characters, or Chinese punctuation.

• If you disable secondary analysis, OpenSearch strictly adheres to your intervention results and performs no further segmentation. If enabled, intervention results may be segmented further.

• To use the Industry - E-commerce General Analyzer as the base for a custom analyzer, your application must be of the Industry-specific Enhanced Edition.

• You cannot delete a custom analyzer that is in use.