AliNLP word segmentation plug-in (analysis-aliws)-Elasticsearch(ES)-阿里云帮助中心

The AliNLP tokenizer plug-in (analysis-aliws) is a built-in system plug-in for Alibaba Cloud Elasticsearch (ES). After you install the plug-in, Alibaba Cloud ES integrates the corresponding analyzer and tokenizer for document analysis and search. You can also use the dictionary configuration feature to hot-update dictionaries.

Plug-in overview

After you install the analysis-aliws plug-in, Alibaba Cloud ES includes the following analyzer and tokenizer by default. You can use them to search documents and use the dictionary configuration feature to update the tokenizer dictionary.

Analyzer: The aliws analyzer retains function words, function-word phrases, and symbols. It also automatically converts all uppercase English characters to lowercase.
Tokenizer: aliws_tokenizer
For more information, see Search documents by using the aliws analyzer and Configure dictionaries.
If the analysis-aliws plug-in does not return the expected results, you can troubleshoot the issue by using Test the analyzer and Test the tokenizer.
To build a custom tokenizer, see Build a custom tokenizer.

Prerequisites

The analysis-aliws plug-in is installed. The plug-in is not installed by default. For installation instructions, see Install and remove a built-in plug-in.

Installing or uninstalling a plug-in triggers a rolling restart of the cluster. As nodes restart one at a time, the cluster status changes from green to yellow. The cluster remains available, but the load on healthy nodes increases and performance might be affected. Perform this operation during off-peak hours.

Limits

The memory size of data nodes in your Elasticsearch cluster must be 8 GiB or higher. If the memory size of the data nodes does not meet this requirement, upgrade the cluster. For more information, see Upgrade the configuration of a cluster.
Elasticsearch V5.X clusters, Elasticsearch V8.X clusters, and Elasticsearch Kernel-enhanced Edition clusters do not support the analysis-aliws plug-in. You can check whether your cluster supports the plug-in in the Elasticsearch console.

Search documents by using the aliws analyzer

Log on to the Kibana console of the target Alibaba Cloud Elasticsearch instance and follow the on-screen instructions to open the Kibana home page.
For instructions, see Log on to the Kibana console.
Note
This topic uses Alibaba Cloud Elasticsearch 6.7.0 as an example. Operations might vary slightly in other versions. Use the actual console interface as the reference.
In the left-side navigation pane, click Dev Tools.

In Console, run one of the following commands to create an index.

Versions earlier than 7.0

PUT /index
{
   "mappings": {
        "fulltext": {
            "properties": {
                "content": {
                    "type": "text",
                    "analyzer": "aliws"
                }
            }
        }
    }
}

Version 7.0 or later

PUT /index
{
  "mappings": {
    "properties": {
        "content": {
            "type": "text",
            "analyzer": "aliws"
          }
      }
  }
}

The preceding examples create an index named index with the type fulltext (or _doc in version 7.x). The index contains a content property of the text type and uses the aliws analyzer.

If the command runs successfully, the following result is returned:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "index"
}

Run the following command to add a document.
Important
The following command applies only to Elasticsearch versions earlier than 7.0. In version 7.0 or later, change fulltext to _doc.
```
POST /index/fulltext/1
{
  "content": "I like go to school."
}
```
The preceding command adds a document named 1 and sets the content field to I like go to school.
If the command runs successfully, the following result is returned:
```
{
  "_index": "index",
  "_type": "fulltext",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}
```
Run the following command to search for documents.

Important

The following command applies only to Elasticsearch versions earlier than 7.0. In version 7.0 or later, change fulltext to _doc.

GET /index/fulltext/_search
{
  "query": {
    "match": {
      "content": "school"
    }
  }
}

The preceding command uses the aliws analyzer to search all documents of the fulltext type for documents whose content field contains school.

If the command runs successfully, the following result is returned:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "fulltext",
        "_id": "2",
        "_score": 0.2876821,
        "_source": {
          "content": "I like go to school."
        }
      }
    ]
  }
}

Note

If the results from the analysis-aliws plug-in do not meet expectations, use Test the analyzer and Test the tokenizer below to troubleshoot the issue.

Configure dictionaries

The analysis-aliws plug-in supports dictionary configuration. You can upload a custom dictionary file named aliws_ext_dict.txt. After you upload the file, all nodes in the cluster automatically load it and hot-update the dictionary without a cluster restart.

Note

After you install the analysis-aliws plug-in, no dictionary file is provided. You must upload a custom dictionary file manually.
Before you upload a dictionary file, make sure that the file meets the following requirements:
- File name: It must be aliws_ext_dict.txt.
- File format: It must use UTF-8 encoding.
- Content: Each line must contain one term with no leading or trailing whitespace. Use UNIX or Linux line feeds (\n). If you create the file in Windows, use the dos2unix tool on a Linux machine to convert the dictionary file before you upload it.

Log on to the Alibaba Cloud Elasticsearch console.
In the left-side navigation pane, click Elasticsearch Clusters.

Open the target instance.

 a. In the menu bar, select the resource group and region.

 b. Click the target instance ID in <b class="xdita-conref" data-key="logstash.pipeline.slide.form.esInstanceId.title" data-project="elasticsearch">Elasticsearch Cluster</b>.

In the left-side navigation pane, choose Configuration and Management > Plug-ins.
In Built-in Plug-ins, find the analysis-aliws plug-in. In its Actions column, click Configure Dictionary.
At the bottom of the Configure Dictionary page, click Configure.
Select a method to upload the dictionary file, and then upload it:
- TXT File: Click Upload TXT File and select a local dictionary file to upload.
- Add OSS File: Enter the Bucket name and file name, and then click Add.
  Make sure that the Bucket and Alibaba Cloud Elasticsearch instance are in the same region. If the contents of the source OSS file change, upload the dictionary file again for the changes to take effect. Automatic synchronization is not supported.
Note
The analysis-aliws plug-in supports only one dictionary file. To update the dictionary file, click the aliws_ext_dict.txt file's X icon to delete the uploaded dictionary file, and then upload a new one.
Click Save.
Saving does not restart the cluster. Instead, it triggers a cluster change that applies the uploaded dictionary file. This process takes about 10 minutes.

Note

To download an uploaded dictionary file, click the corresponding download icon before configuration.

Test the analyzer

Run the following command to test the aliws analyzer:

GET _analyze
{
  "text": "I like go to school.",
  "analyzer": "aliws"
}

If the command runs successfully, the following result is returned:

{
  "tokens" : [
    {
      "token" : "i",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "like",
      "start_offset" : 2,
      "end_offset" : 6,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "go",
      "start_offset" : 7,
      "end_offset" : 9,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "school",
      "start_offset" : 13,
      "end_offset" : 19,
      "type" : "word",
      "position" : 8
    }
  ]
}

Test the tokenizer

Run the following command to test the aliws_tokenizer tokenizer:

GET _analyze
{
  "text": "I like go to school.",
  "tokenizer": "aliws_tokenizer"
}

If the command runs successfully, the following result is returned:

{
  "tokens" : [
    {
      "token" : "I",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : " ",
      "start_offset" : 1,
      "end_offset" : 2,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "like",
      "start_offset" : 2,
      "end_offset" : 6,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : " ",
      "start_offset" : 6,
      "end_offset" : 7,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "go",
      "start_offset" : 7,
      "end_offset" : 9,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : " ",
      "start_offset" : 9,
      "end_offset" : 10,
      "type" : "word",
      "position" : 5
    },
    {
      "token" : "to",
      "start_offset" : 10,
      "end_offset" : 12,
      "type" : "word",
      "position" : 6
    },
    {
      "token" : " ",
      "start_offset" : 12,
      "end_offset" : 13,
      "type" : "word",
      "position" : 7
    },
    {
      "token" : "school",
      "start_offset" : 13,
      "end_offset" : 19,
      "type" : "word",
      "position" : 8
    },
    {
      "token" : ".",
      "start_offset" : 19,
      "end_offset" : 20,
      "type" : "word",
      "position" : 9
    }
  ]
}

Build a custom tokenizer

After the analysis-aliws plug-in tokenizes text, it applies filters, including stemmer, lowercase, porter_stem, and stop. To use these filters in a custom tokenizer, add the analysis-aliws plug-in tokenizer aliws_tokenizer to the custom tokenizer, and add filter configurations based on your business requirements. The following example shows how. Add the stopwords required by your business in stopwords.

PUT my-index-000001
{
  "settings": {
    "analysis": {
     "filter": {
      "my_stop": {
       "type": "stop",
       "stopwords": [
        " ",
        ",",
        ".",
        "　",
        "a",
        "of"
       ]
      }
     },
     "analyzer": {
      "my_custom_analyzer": {
       "type": "custom",
       "tokenizer": "aliws_tokenizer",
       "filter": [
        "lowercase",
        "porter_stem",
        "my_stop"
       ]
      }
     }
    }
    }
}

Note

If you do not need filters, you can remove the filter configuration.

aliws_tokenizer supports building a custom tokenizer by using synonyms. Its configuration is the same as that for the IK tokenizer. For more information, see Use synonyms.

FAQ

How do I configure the analysis-aliws plug-in? What is the format of the dictionary file for this plug-in?
What are the differences among Elasticsearch synonyms, IK tokens, and AliNLP tokens?
If I use the rolling update method to update dictionaries that are dynamically loaded from OSS and the dictionaries stored in OSS are updated, will the dictionaries on all nodes in my Elasticsearch cluster be automatically updated?
When you use the aliws analyzer to tokenize words, the final letter might be missing. For example, tokenizing iPhone and Chinese returns Iphon and chines, with the final e removed.
- Cause: The aliws analyzer applies a stemming filter after tokenization, which removes the final e.
- Solutions: Run the following command to create a custom analysis configuration named my_custom_analyzer without the filter section.
```
PUT my-index1
{
    "settings": {
        "number_of_shards": 1,
        "analysis": {
            "analyzer": {
                "my_custom_analyzer": {
                    "type": "custom",
                    "tokenizer": "aliws_tokenizer"
                }
            }
        }
    }
}
```
- Verification: Run the following command to verify whether the tokenization results meet expectations.
```
GET my-index1/_analyze
{
    "analyzer": "my_custom_analyzer",
    "text": ["iphone"]
}
```

References

To view the plug-ins supported by Alibaba Cloud ES, see Overview of plug-ins.
To install a built-in plug-in by calling an API operation, see InstallSystemPlugin.
To update the dictionary file of the analysis-aliws plug-in by calling an API operation, see UpdateAliwsDict.
To obtain the list of plug-ins for a specified Alibaba Cloud ES instance by calling an API operation, see ListPlugins.