How to create, update, get, and delete indexes, 如何创建、修改、查询、删除索引-Simple Log Service(SLS)-阿里云帮助中心

An index is an inverted storage structure that maps keywords to logical pointers referencing actual data, enabling fast lookups by keyword. You must configure indexes before you can query and analyze log data. This topic provides sample code for creating, modifying, querying, and deleting indexes by using Simple Log Service SDK for Python.

Prerequisites

Simple Log Service SDK for Python is initialized.

Simple Log Service SDK for Python is installed. For more information, see Install Simple Log Service SDK for Python.
A project and a logstore are created in Simple Log Service, and logs are collected. For more information, see Manage projects, Create a basic LogStore, and Data collection.

Precautions

In this example, the public Simple Log Service endpoint for the China (Hangzhou) region is used. Endpoint: https://cn-hangzhou.log.aliyuncs.com.

If you want to access Simple Log Service from other Alibaba Cloud services that reside in the same region as your project, you can use the internal Simple Log Service endpoint, which is https://cn-hangzhou-intranet.log.aliyuncs.com.

For more information about the supported regions and endpoints of Simple Log Service, see Endpoint.

Billing

Logstores in pay-by-ingested-data mode

Indexes occupy storage space. Manage intelligent tiered storage
Reindexing does not incur fees.
Index traffic charges: Billable items for the pay-by-ingested-data mode

Logstores in pay-by-feature mode

Indexes occupy storage space. Manage intelligent tiered storage
Creating an index generates billable index traffic. Billable items for pay-by-feature mode How do I reduce index traffic fees?
Reindexing incurs fees. Billable items and prices are the same as for creating an index.

Raw log

body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

Sample code that is used to create indexes

You can also manage indexes in the Simple Log Service console. For more information, see Create indexes.

The following sample code creates indexes with full-text indexing enabled and field indexing enabled for the request_method and status fields. This example is based on the raw log.

from aliyun.log import LogClient, IndexConfig
import os
# This example retrieves the AccessKey ID and AccessKey secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The service endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a Simple Log Service client.
client = LogClient(endpoint, accessKeyId, accessKey)
# The Project name.
project_name = "ali-test-project"
# The Logstore name.
logstore_name = "ali-test-logstore"
if __name__ == '__main__':
    # Before you create an index, you must plan the configurations for full-text indexing and field indexing. This example enables full-text indexing and field indexing for the request_method and status fields.
    logstore_index = {'line': {
        'token': [',', ' ', "'", '"', ';', '=', '(', ')', '[', ']', '{', '}', '?', '@', '&', '<', '>', '/', ':', '\n',
                  '\t',
                  '\r'], 'caseSensitive': False, 'chn': False}, 'keys': {'request_method': {'type': 'text',
                                                                                 'token': [',', ' ', "'", '"', ';', '=',
                                                                                           '(', ')', '[', ']', '{', '}',
                                                                                           '?', '@', '&', '<', '>', '/',
                                                                                           ':', '\n', '\t', '\r'],
                                                                                 'caseSensitive': False, 'alias': '',
                                                                                 'doc_value': True, 'chn': False},
                                                                         'status': {'type': 'long', 'alias': '',
                                                                                'doc_value': True}},
        'log_reduce': False,
        'max_text_len': 2048}
    print("ready to create index")
    index_config = IndexConfig()
    index_config.from_json(logstore_index)
    client.create_index(project_name, logstore_name, index_config)
    print("create index success ")

Expected results:

ready to create index
create index success

Sample code that is used to modify indexes

You can also manage indexes in the Simple Log Service console. For more information, see Create indexes.

The following sample code modifies indexes by setting caseSensitive to True for the request_method field. Full-text indexing and field indexing for request_method and status remain enabled. This example is based on the raw log.

from aliyun.log import LogClient, IndexConfig
import os
# This example retrieves the AccessKey ID and AccessKey secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The service endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a Simple Log Service client.
client = LogClient(endpoint, accessKeyId, accessKey)
# The Project name.
project_name = "ali-test-project"
# The Logstore name.
logstore_name = "ali-test-logstore"
if __name__ == '__main__':
    # Set the request_method field to be case-sensitive.
    logstore_index = {'line': {
        'token': [',', ' ', "'", '"', ';', '=', '(', ')', '[', ']', '{', '}', '?', '@', '&', '<', '>', '/', ':', '\n',
                  '\t',
                  '\r'], 'caseSensitive': False, 'chn': False}, 'keys': {'request_method': {'type': 'text',
                                                                                 'token': [',', ' ', "'", '"', ';', '=',
                                                                                           '(', ')', '[', ']', '{', '}',
                                                                                           '?', '@', '&', '<', '>', '/',
                                                                                           ':', '\n', '\t', '\r'],
                                                                                 'caseSensitive': True, 'alias': '',
                                                                                 'doc_value': True, 'chn': False},
                                                                         'status': {'type': 'long', 'alias': '',
                                                                                'doc_value': True}},
        'log_reduce': False,
        'max_text_len': 2048}
    print("ready to update index")
    index_config = IndexConfig()
    index_config.from_json(logstore_index)
    client.update_index(project_name, logstore_name, index_config)
    print("update index success ")

Expected results:

ready to update index
update index success

Sample code that is used to query indexes

The following code retrieves the index configuration of a specified Logstore.

from aliyun.log import LogClient, IndexConfig
import os
# This example retrieves the AccessKey ID and AccessKey secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The service endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a Simple Log Service client.
client = LogClient(endpoint, accessKeyId, accessKey)
# The Project name.
project_name = "ali-test-project"
# The Logstore name.
logstore_name = "ali-test-logstore"
if __name__ == '__main__':
    # Get the index configuration.
    print("ready to list index")
    res = client.get_index_config(project_name, logstore_name)
    print("The index config is :%s" % res.get_index_config().to_json())
    print("list index success ")

Expected results:

ready to list index
The index config is :{'line': {'token': [',', ' ', "'", '"', ';', '=', '(', ')', '[', ']', '{', '}', '?', '@', '&', '<', '>', '/', ':', '\n', '\t', '\r'], 'caseSensitive': False, 'chn': False}, 'keys': {'request_method': {'type': 'text', 'token': [',', ' ', "'", '"', ';', '=', '(', ')', '[', ']', '{', '}', '?', '@', '&', '<', '>', '/', ':', '\n', '\t', '\r'], 'caseSensitive': True, 'alias': '', 'doc_value': True, 'chn': False}, 'status': {'type': 'long', 'alias': '', 'doc_value': True}}, 'log_reduce': False, 'max_text_len': 2048}
list index success

Sample code that is used to delete indexes

The following code deletes the index of a specified Logstore.

from aliyun.log import LogClient, IndexConfig
import os
# This example retrieves the AccessKey ID and AccessKey secret from environment variables.
accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')
# The service endpoint for Simple Log Service. This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
endpoint = "cn-hangzhou.log.aliyuncs.com"
# Create a Simple Log Service client.
client = LogClient(endpoint, accessKeyId, accessKey)
# The Project name.
project_name = "ali-test-project"
# The Logstore name.
logstore_name = "ali-test-logstore2"
if __name__ == '__main__':
    # Delete the index.
    print("ready to delete index")
    client.delete_index(project_name, logstore_name)
    print("delete index success ")

Expected results:

ready to delete index
delete index success

References

In addition to its native SDK, SLS also supports the common Alibaba Cloud SDKs. For more information, see Log Service_SDKcenter-Alibaba CloudOpenAPIDeveloper Portal.
To manage indexes in the console, see Create an index.
Simple Log Service CLI.
For more information about index-related API operations, see the following topics: