In a serverless log analysis application, you can visually manage your indexes. You can create, modify, and delete indexes. This topic explains how to create an index, write data to it, and run queries.
Prerequisites
You have created a log analysis application. For detailed steps, see Create a Log Analysis Application.
Index features
Indexes in a log analysis application have the following features:
Indexes offer built-in automatic shard tuning and intelligent rollover. You do not need to manage rollover or aliases. When reading from or writing to an index, specify only the index name.
All fields in an index follow these rules:
Indexes support search but not scoring. Scoring evaluates document relevance.
Search is case-insensitive for indexed fields.
Wildcard queries are not supported for text fields.
The default tokenizer limits each token to a maximum of 255 characters.
If primary key-based writes are disabled, the index does not support read or write operations by ID. Returned IDs have no meaning.
The order of fields—and of fields inside arrays—may change. Avoid relying on field order.
Some field formats may change, such as date fields and numbers enclosed in quotation marks. To avoid format changes:
Time field: Specifies the time format.
Numbers in quotation marks: Do not enclose numbers in quotation marks when writing data.
Numeric double and long fields stored in scientific notation: Avoid storing double or long fields in scientific notation.
Create an index
Log on to the Elasticsearch Serverless console.
In the top menu bar, select the destination region.
NoteCurrently, only the China (Hangzhou), China (Beijing), China (Shanghai), and China (Shenzhen) regions are supported.
In the navigation pane on the left, click Application Management.
On the Application Management page, you can click the name of the target application.
In the navigation pane on the left, you can choose .
You can click Create Index.
In the Create Index panel, you can configure the index.
Category
Project
Description
Basic configuration
Index name
A custom index name.
Field mapping
Manually configure field mappings for the index.
Field name: A custom name for the field in the index data.
Field type: Available data types include boolean, long, double, keyword, text, date, object, and match_only_text. To configure additional field types, you can Switch To JSON Edit Mode to edit them.
NoteFor a full list of supported field types, see Field data types.
For date field formats, see mapping-date-format.
Enable indexing: Enable this option to build an index for the field. The field becomes searchable.
Enable statistics: This setting cannot be modified. Enabling statistics supports statistical analysis on field values but increases storage space.
Tokenizer: For text fields, customize the tokenizer. A tokenizer splits field content into tokens. The default tokenizer is
whitespace'";=()[]{}?@&<>/:\n\t\r#\|,.
Use sample input to auto-generate field mappings.
Click Auto-configure with Sample Input. Enter JSON-formatted sample data in the input box. Click OK. After validation, the platform automatically maps fields to the field mapping table.
Sample JSON data:
{ "bool_field": true, "date_field": "20220126", "double_field": 3.14, "keyword_field": "This is a line of text that does not require tokenization", "long_field": 126, "object_field": { "sub_field": 2022 }, "text_field": "This is a line of text that requires tokenization. Lines longer than 36 characters are inferred as requiring tokenization and are assigned the text type" }Configure field mappings in JSON mode.
In the upper-right corner of the Field Mapping section, click Switch to JSON Edit Mode. Then edit the field mapping in JSON mode.
NoteSettings sync automatically between modes.
JSON example:
{ "settings": { "index": { "apack": { "protection": { "index": { "forbid_id": true } } }, "merge_dynamic_fields": { "enable": false }, "timeseries": { "expiration_days": 7, "advanced_indexing_optimization": { "enabled": true }, "auto_data_organization": { "enabled": true } }, "sort": { "field": [ "date_field" ], "order": [ "desc" ] } } }, "mappings": { "dynamic": "true", "properties": { "bool_field": { "type": "boolean", "doc_values": true, "index": true }, "date_field": { "type": "date", "format": "basic_date", "doc_values": true, "index": true }, "double_field": { "type": "double", "doc_values": true, "index": true }, "keyword_field": { "type": "keyword", "doc_values": true, "index": true }, "long_field": { "type": "long", "doc_values": true, "index": true }, "object_field": { "type": "object", "properties": { "sub_field": { "type": "long", "doc_values": true, "index": true } } }, "text_field": { "type": "text", "analyzer": "lineAnalyzer", "doc_values": false, "index": true, "term_hash_enable": true } } } }
time field
Select a date-type field to record the data timestamp.
NoteYou cannot modify or delete the date field after the index is created.
Data retention period
Limited retention: Retain index data for 1 to 365 days.
Permanent retention: Retain index data indefinitely.
To update the data retention period using an API:
PUT {indexName}/_settings { "index.timeseries.expiration_days": 7 }Write-depth optimization
When enabled, this feature automatically optimizes the index structure. It improves write performance and reduces storage costs. However, it has limits. For example, prefix queries are not supported. For details, see Write-depth optimization.
To enable or disable write-depth optimization using an API:
Enable:
PUT {indexName}/_settings { "index.timeseries.advanced_indexing_optimization.enabled": true }Disable:
PUT {indexName}/_settings { "index.timeseries.advanced_indexing_optimization.enabled": false }
Advanced configuration
Primary key-based writes
When enabled, compute units (CU) and storage usage increase significantly. Document updates may fail or result in duplicate documents.
To enable or disable primary key-based writes using an API:
Enable:
PUT {indexName}/_settings { "index.apack.protection.index.forbid_id": false }Prohibited:
PUT {indexName}/_settings { "index.apack.protection.index.forbid_id": true }
Auto data organization
When enabled, the system runs lossless force merge operations on the index at regular intervals. This improves query performance and optimizes storage space. However, it increases CU usage for writes.
NoteYou can view resource consumption from force merge operations in the monitoring center.
To enable or disable auto data organization using an API:
Enable:
PUT {indexName}/_settings { "index.timeseries.auto_data_organization.enabled": true }Shut down:
PUT {indexName}/_settings { "index.timeseries.auto_data_organization.enabled": false }
Dynamic field configuration
How to handle fields in incoming data that are not defined in the field mapping:
Detect: Automatically add new fields to the field mapping based on their data type.
NoteTo enable dynamic field detection, set the dynamic field value to true.
For new text fields, the default analyzer is lineAnalyzer.
match_phrase queries on default text fields may be truncated. This can lead to inaccurate results.
Merge: Do not add new fields to the field mapping. Merge them into @_all_@.
Ignore: Take no action. Keep the field only in the raw data.
Fail: Throw an error when a new field is detected.
To enable or disable dynamic field merging using an API:
Enable:
PUT {indexName}/_settings { "index.merge_dynamic_fields.enable": true }Disable:
PUT {indexName}/_settings { "index.merge_dynamic_fields.enable": false }
You can click OK.
Write and query index data
You can log on to the Kibana console for your serverless application. For more information, see Use a serverless application with Kibana.
In the upper-left corner of the page, you can click the
icon. Then choose .You can run the following code to manage the index.
View index mapping details:
GET test/_mappingResponse:
{ "test" : { "mappings" : { "dynamic" : "true", "_meta" : { "app_id" : "qning-***", "index_name" : "test" }, "dynamic_templates" : [ { "match_tag" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "analyzer" : "lineAnalyzer", "term_hash_enable" : true, "type" : "match_only_text" } } } ], "properties" : { "bool_field" : { "type" : "boolean" }, "date_field" : { "type" : "date", "format" : "basic_date" }, "double_field" : { "type" : "double" }, "keyword_field" : { "type" : "keyword", "term_hash_enable" : true }, "long_field" : { "type" : "long" }, "object_field" : { "properties" : { "sub_field" : { "type" : "long" } } }, "text_field" : { "type" : "match_only_text", "analyzer" : "lineAnalyzer", "term_hash_enable" : true } } } } }Insert data into the index:
POST test/_bulk { "index" : {} } {"bool_field":true,"date_field":"20230901","double_field":3.14,"keyword_field":"Elasticsearch Serverless","long_field":126,"object_field":{"sub_field":2023},"text_field":"The Elasticsearch Serverless service provides a simple, elastic, and out-of-the-box Elasticsearch experience. You do not need to manage Elasticsearch cluster resources or configurations."}Response:
{ "took" : 20, "errors" : false, "items" : [ { "index" : { "_index" : ".ts@test@20240920170712", "_type" : "_doc", "_id" : "3yewDpIBzu2YeHCV92dK", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1, "status" : 201 } } ] }Query data in the index:
GET test/_search { "query": { "match": { "keyword_field": "Elasticsearch Serverless" } } }Response:
{ "took" : 39, "timed_out" : false, "_shards" : { "total" : 6, "successful" : 6, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "max_score" : 0.2876821, "hits" : [ { "_index" : ".ts@test@20240920170712", "_type" : "_doc", "_id" : "f_J2uxDpIBBqbPspEIglF4", "_score" : 0.2876821, "_source" : { "object_field" : { "sub_field" : 2023 }, "text_field" : "The Elasticsearch Serverless service provides a simple, elastic, and out-of-the-box Elasticsearch experience. You do not need to manage Elasticsearch cluster resources or configurations.", "bool_field" : true, "long_field" : 126, "double_field" : 3.14, "keyword_field" : "Elasticsearch Serverless", "date_field" : "20230901" } } ] } }
Other operations
On the Index List page, you can view, modify, or delete indexes.
Project | Description |
View index configuration | In the Actions column for the target index, click Modify Configuration to view the index settings. |
Modify index configuration | In the Actions column for the target index, click Modify Configuration to update the index settings.
|
Delete an index | In the Actions column for the target index, click Delete to delete the index. |