This topic describes the whitelists for cluster configurations, index configurations, processors, and queries supported by retrieval-augmented generation applications (Version 8.17) in Alibaba Cloud Elasticsearch Serverless.
Usage notes
To ensure service stability, Alibaba Cloud Elasticsearch Serverless (ES Serverless) imposes certain limits on application resources and usage. The whitelists in this topic describe the available features and configurations. However, some whitelisted configurations and features have parameter range constraints. If you exceed these ranges, you must adjust service quotas or submit a ticket to request access.
Cluster configuration whitelist
Cluster configurations define and control the behavior of an entire Elasticsearch cluster to ensure its stability, performance, security, and scalability. The following table lists the cluster configurations that you can modify for retrieval-augmented generation applications (Version 8.17).
Alibaba Cloud Elasticsearch Serverless manages resources as applications and is not based on the concept of clusters. However, you can use cluster-level configurations from native Elasticsearch to adjust the application parameters for Elasticsearch Serverless. The term 'cluster configuration' is used here for consistency with native Elasticsearch. For more information about cluster-level configurations, see the GET/_cluster/settings/include_defaults API.
Configuration item | Description |
Controls whether you must specify an index name when you perform important operations. This prevents the accidental deletion of multiple indexes or large amounts of data. Valid values: Note High-risk operations are those that can cause data loss or make a cluster unavailable, such as deleting an index, shutting down a cluster, and purging a cache.
| |
apack.tenant.index_settings.throw_unsupported_setting | Controls whether an error is reported if an unsupported configuration parameter is used when you create or update an index. Valid values:
|
Controls whether to automatically create indexes. Valid values:
|
Index configuration whitelist
Background information
In Elasticsearch, you can group documents with similar structures into the same index to quickly search and query specific content. The mapping and settings of an index are core operations that define its behavior and structure. Together, they determine how data is stored, queried, and managed.
Mapping: Defines the structure of documents in an index, including the data types of fields (such as
text,integer, anddate) and their behavior (such as whether to tokenize, store original values, or support sorting and aggregation).Settings: Define the global behavior and related constraints at the index level. They control the storage, performance, and availability of the index, along with advanced data-related rules. These configurations directly affect how data is distributed, how searches are executed, and how resources are allocated.
This topic describes the whitelists for field data types and index configurations that are supported by retrieval-augmented generation applications (Version 8.17).
For more information about how to create and configure an index, see the Update index settings API.
Supported field data types
Field data types determine how data is stored and queried in Elasticsearch. The following table lists the field data types supported by retrieval-augmented generation applications (Version 8.17).
Category | Field data type | Description |
Numeric types | An 8-bit signed integer that ranges from -128 to | |
A 16-bit signed integer that ranges from -32,768 to | ||
A 32-bit signed integer that ranges from | ||
A 64-bit signed integer that ranges from | ||
A 16-bit floating-point number with low precision. Use this type to store small floating-point values and save storage space. | ||
A 32-bit single-precision floating-point number. Use this type to store floating-point values with higher precision, such as temperatures or prices. | ||
A 64-bit double-precision floating-point number. Use this type to store high-precision data, such as for scientific or financial computing. | ||
A floating-point value that is compressed for storage to save space. | ||
A 64-bit unsigned integer that ranges from | ||
Range types | Stores a range of integers, including a start value and an end value. Use this type for integer interval queries, such as for an age range. | |
Stores a range of floating-point numbers, including a start value and an end value. Use this type for floating-point interval queries, such as for a temperature range or price range that includes decimals. | ||
Stores a range of long integers, including a start value and an end value. Use this type for large integer interval queries, such as querying for records with an order ID in a specified range. | ||
Stores a range of double-precision floating-point numbers, including a start value and an end value. Use this type for high-precision numerical interval queries, such as for a price fluctuation range in financial data like | ||
Stores a range of IP addresses. Use this type for IP address segment matching, such as blocking IP addresses in a specified range. | ||
Stores a date range, including a start time and an end time. Use this type for time interval queries, such as for an event time or order validity period. | ||
Sub-data types Note Used to handle nested or complex data structures. | Stores a JSON object that contains unstructured nested data, such as user configuration information. | |
A queryable nested object that supports independent indexing. Use this type to store multiple independent sub-objects, such as product reviews or order details. | ||
Text and keyword types | Used for full-text search and supports tokenization. This type is suitable for unstructured content, such as product descriptions or article content. | |
Used for exact matching and is not tokenized. This type is suitable for structured content, such as IDs or email addresses. | ||
Stores a static field that has the same value in every document in an index. For example, a version number or an environment identifier like | ||
Suitable for full-text search scenarios where storage or highlighting is not required. This method saves storage space and improves search performance. | ||
Date and time types | A date and time format that is accurate to the millisecond. Use this type to store time series data, such as log data, or event dates. | |
A date and time format that is accurate to the nanosecond. Use this type to store high-precision timestamps, such as for financial transactions. | ||
Binary type | Directly stores raw binary data, such as a binary log. | |
Boolean type | Stores a Boolean value ( | |
Geolocation types | Stores geographic coordinates (latitude and longitude of a single point). Use this type for location point queries, such as for store coordinates. | |
Stores complex geometric shapes, such as polygons. Use this type for area matching, such as for administrative region boundary queries. | ||
Stores geographic coordinates (latitude and longitude of a single point). Use this type for location point queries. It is often used for foreign key fields, such as associating an order with a delivery address. | ||
IP and network type | Used for storing and querying IPv4 or IPv6 addresses. | |
Vector types | Stores dense vectors. This type is often used in machine learning for things like image vectors or text embeddings. | |
Stores sparse vectors by storing only non-zero values and their indexes. This type is often used for high-dimensional sparse data, such as text features. | ||
Other types | Stores nested objects or arrays. Use this type for unconventional aggregations or storing unstructured data. | |
Performs text-matching queries based on wildcard characters to find text that matches a specific pattern in a field. For example, use | ||
Stores geospatial geometric shapes, such as polygons, lines, and areas. Use this type to handle complex geographic area ranges and spatial relationship queries. | ||
An alias can point to one or more indexes. You can then use a single alias to query data from multiple indexes at once. For example, you can set the alias for indexes like | ||
Controls the concurrency of document updates and data consistency. It is often used for conflict detection to prevent data overwrites or inconsistencies caused by multiple clients modifying the same document simultaneously. | ||
Provides real-time search suggestions. As you type a search query character by character, the system immediately returns relevant, partially matched results. | ||
Automatically converts text content into semantic embeddings through an inference endpoint and intelligently chunks long text for efficient processing of large-scale corpora. By encapsulating the complex vectorization process into a field-level feature, it significantly lowers the technical barrier to building intelligent semantic search applications. |
Supported index configurations
The supported index configurations for retrieval-augmented generation applications (Version 8.17) are divided into mapping limit configurations and general index configurations. You can configure them as needed.
Mapping limit configurations
Index mapping limit configurations are used to restrict the number of field mappings and prevent an excessive number of mapped fields from degrading index performance. The following table lists the mapping limit configurations supported by retrieval-augmented generation applications (Version 8.17).
Configuration item | Description |
Sets the maximum length of a field name to prevent long field names from consuming too much memory. | |
Sets the maximum depth of a | |
Sets the maximum number of | |
Sets the maximum number of nested objects in a single document to prevent documents from containing too many nested objects, which can lead to insufficient memory. The default value is | |
Sets the maximum number of fields in an index to prevent an excessive number of fields from causing index bloat and affecting query performance. The default value is | |
index.mapping.coerce | Specifies whether to coerce the data type of an input value, such as converting the string
|
index.mapping.ignore_malformed | Specifies whether to ignore data that does not conform to the field type. Valid values:
|
General index configurations
General index configurations define the underlying behavior and resource allocation of an index, such as the number of shards and tokenization rules. Proper index configuration helps improve query and write performance, increase resource utilization, and efficiently manage data. The following table lists the general index configurations supported by retrieval-augmented generation applications (Version 8.17).
Category | Configuration item | Description |
Core index configurations | Sets the number of primary shards for an index. This controls data distribution and parallel processing capabilities. The default value is | |
Sets the compression format used to store data. Valid values:
| ||
Sets the interval at which shard data is refreshed from memory to disk. This adjusts write performance and query real-time capabilities. The default value is | ||
Analyzers and pipelines | index.analysis.* | Customizes analyzers, tokenizers, and filters to control the tokenization rules for text fields. |
The default data pipeline used when inserting documents. It is used for system pre-processing of data, such as format conversion or field calculation. | ||
The final processing pipeline after a document is inserted. It is used to ensure that data complies with specifications, such as by filtering encrypted or sensitive fields. | ||
Query and sorting | Defines which fields are matched by default when a query string does not specify a field. For example, it can automatically match the query statement | |
Defines the sorting rules for documents within a shard to accelerate queries, such as sorting by time. | ||
Index blocking | Restricts the types of operations that can be performed on an index, such as write, delete, and metadata access. This is used to temporarily lock an index for maintenance or debugging. | |
Index performance | index.max_adjacency_matrix_filters | Sets the complexity threshold for adjacency matrix queries to prevent memory overflow or performance degradation due to too many or complex conditions. The default value is |
The maximum number of Note
| ||
Sets the maximum result window size for Note
| ||
Sets the maximum difference in length between n-grams in an Ngram tokenizer to prevent index bloat or performance degradation due to improper tokenizer configuration. | ||
Sets the maximum number of refresh listeners for a single index to prevent an excessive number of listeners from affecting refresh performance. | ||
Sets the maximum length of a regular expression or prefix query to prevent long content from affecting query speed. The default value is | ||
Sets the maximum document window size for a rescore operation, which reorders results using a more complex scoring model, to limit the resource consumption of the process. The default value is | ||
Sets the maximum result window size for paged queries to prevent high-overhead paging, such as | ||
Sets the maximum number of script fields in a single search request to limit the resource consumption of script calculations. The default value is | ||
Controls the maximum difference in length between shingles in a Shingle tokenizer to prevent the tokenizer from generating too many combined words. The default value is | ||
Sets the maximum number of conditions that can be added in a single query or aggregation to prevent an overly large condition list from causing insufficient memory and affecting performance. The default value is | ||
index.max_prefix_length | Sets the maximum length of a prefix query to prevent excessively long prefixes from affecting query performance. The default value is | |
index.max_wildcard_length | Sets the maximum length of a wildcard query to limit its complexity and avoid full table scans. The default value is |
Processor whitelist
Before a document is indexed in Elasticsearch, you can use a processor to transform, clean, enrich, or filter its content. The following table lists the Ingest Pipelines Processor whitelist for retrieval-augmented generation applications (Version 8.17).
Processor type | Processor | Description |
Basic data operations | Appends one or more values to a field, which must be an array. | |
Sets the value of a field. If the field does not exist, it is created. If the field exists, its value is overwritten. | ||
Deletes one or more fields. | ||
Renames a field. | ||
Discards the entire document. The document is not indexed. | ||
Forcibly terminates the processor, breaks the pipeline, and returns an error. | ||
Calls another Ingest Pipeline. | ||
Immediately terminates the pipeline execution. Subsequent processors do not run. | ||
Data type conversion and formatting | Converts the data type of a field. | |
Converts a byte size string, such as | ||
Parses a time string and sets it as the value of | ||
Sorts the array values in a field. | ||
String and text processing | Removes leading and trailing whitespace from a string. | |
Splits a string into an array based on a separator. | ||
Merges an array into a string. | ||
Converts a string to uppercase. | ||
Converts a string to lowercase. | ||
Replaces content in a string using a regular expression. | ||
Decodes a URL-encoded string. | ||
Removes HTML tags. | ||
Structured parsing | Extracts fields using simple pattern matching. | |
Parses unstructured text using regular expression patterns. | ||
Extracts fields from a key-value pair string. | ||
Parses a JSON string into an object field. | ||
Parses a CSV string into multiple fields. | ||
Geo and network information processing | Converts latitude and longitude to a geographic grid. This is often used for aggregations. | |
Resolves an IP address to a geographic location. This requires integration with GeoIP. | ||
Determines the direction of network traffic. | ||
Parses a User-Agent string to extract information such as the browser and operating system. | ||
Parses a URI to extract information such as the protocol, host, and path. | ||
Data security | Masks sensitive data in a field, such as hiding the middle digits of a phone number. | |
Generates a unique hash value for the content of a field. This is used to remove duplicates. | ||
Advanced processors | Extracts text from file attachments, such as PDF or Word files. | |
Converts a circular area (centroid and radius) into a geographic shape for geographic queries. | ||
Generates a | ||
Expands a field name with dot notation into a nested object. | ||
Executes a set of processors for each element in an array. | ||
Script and AI processing | Uses the Painless scripting language to write complex logic, such as conditional statements, mathematical operations, and dynamic field generation. Note To use custom scripts, you must submit a ticket to add them to the whitelist. | |
Calls a deployed machine learning model, such as an NLP model, for inference. |
Query whitelist
Query section whitelist
The following table lists the parameters supported by the outermost JSON object of a search request body in retrieval-augmented generation applications (Version 8.17).
Parameter | Description |
retriever | Builds a retrieval-augmented generation (RAG) system. It supports hybrid retrieval (keyword and vector) and simplifies the semantic search process. |
terminate_after | Limits the maximum number of hits. |
min_score | Filters results based on a minimum score. |
_source | Controls whether the |
stored_fields | Returns stored fields. |
query | The main query condition. |
post_filter | Filters results after the query. This does not affect aggregations. |
knn | Performs an approximate nearest neighbor (ANN) search for vectors. |
script_fields | Returns calculated fields from a script. Note To use custom scripts, you must submit a ticket to add them to the whitelist. |
indices_boost | Assigns weights to different indexes. |
aggs / aggregations | Performs aggregation and analysis. |
highlight | Highlights matching content. |
rescore | A rescoring mechanism. |
slice | Querying index segments |
collapse | Merges and removes duplicates from multiple repetitive or similar query conditions. |
pit | Performs a point in time search. |
docvalue_fields | Returns |
fields | Returns specified fields. |
search_after | Performs deep paging. |
runtime_mappings | Defines runtime fields to dynamically calculate field values at query time without re-indexing. This is often used for log parsing, field transformation, and sensitive data masking. Note Some |
Query type whitelist
The following query types are supported by retrieval-augmented generation applications (Version 8.17).
Compound queries
Query type
Description
Combines multiple query clauses. It supports Boolean logic such as
mustandshould.Lowers the score of documents that meet a certain condition, but still returns them.
Gives all matching documents the same score. This is often used in filtering scenarios.
Takes the highest score from multiple queries as the final score to avoid excessively high score accumulation.
Provides full control over the scoring of documents. It can be combined with scripts, randomness, and other factors for scoring.
Full-text queries
Query type
Description
Provides precise control over the position and order of words for complex text pattern matching.
A standard full-text search that tokenizes the input text and then performs a match.
Matches a prefix. This is used in auto-completion scenarios.
Performs a phrase match, which requires words to be in the same order and adjacent.
Performs a phrase match and supports prefix matching for the last word.
Merges multiple fields into a virtual field for searching to improve cross-field relevance.
Executes a
matchquery on multiple fields.Supports complex query syntax, such as
AND,OR,NOT,+, and-.A simplified version of
query_stringwith more user-friendly syntax and better fault tolerance.Optimizes searches on long text by automatically distinguishing between high-frequency and low-frequency keywords.
Geo queries
Query type
Description
Filters geographic data that falls within a circular area defined by a centroid (latitude and longitude) and a distance range.
Shape queries
Query type
Description
Determines whether a geographic shape in a document, such as a polygon or line, has a certain spatial relationship with a specified query shape, such as intersects or contains.
Joining queries
Query type
Description
Used for exact matching of nested objects.
Match all
Query type
Description
Matches all documents in an index.
Span queries
Query type
Description
Matches a span query that is contained within another span query.
Matches a span query and requires it to appear within the first N words of a field.
Enables fuzzy matching for
span_term. It is usually used withwildcard,regexp, orprefix.Matches multiple words and requires them to appear within a certain distance of each other, such as allowing three words in between. The order can be controlled.
Excludes the results of one span query from the results of another span query.
Matches if any of multiple span queries match.
Matches an exact word. It is similar to a
termquery but is used in a span query chain.Matches a span query that is completely within the range of another span query.
Vector queries
Query type
Description
The primary way that Elasticsearch implements dense vector search. It finds the
kmost similar documents by calculating the similarity between vectors, such as cosine similarity.Stores keyword weights to improve computational efficiency.
Expands a query into a sparse vector to improve recall rate.
Specialized queries
Query type
Description
Boosts the score of documents that meet a condition based on a time or geographic distance.
Allows you to use the Painless scripting language to write custom Boolean logic to determine whether a document matches.
NoteTo use custom scripts, you must submit a ticket to add them to the whitelist.
script_scoreis part offunction_scoreand lets you fully customize the_scoreof a document for personalized sorting.NoteTo use custom scripts, you must submit a ticket to add them to the whitelist.
Allows you to manually specify that certain document IDs must appear in the search results and control their positions.
Term-level queries
Query type
Description
Matches documents that contain the field and have a non-
nullvalue.Matches a
termwith a similar spelling, based on the Levenshtein edit distance.Finds documents by their exact
_id.Matches a
termthat starts with a specified prefix.Matches a range of numeric, date, or string values.
Matches a
termusing a regular expression.Matches an exact
term. The match must be exact and is case-sensitive.Matches if the field value is any of the values in a given list.
Supports fuzzy matching with
*(any character) and?(a single character).Hybrid retrieval (Retriever rrf)
Query type
Description
Enables Reciprocal Rank Fusion (RRF) sorting in a search request.
Aggregation query type whitelist
The following aggregation query types are supported by retrieval-augmented generation applications (Version 8.17).
Bucket aggregations
Classification
Query type
Description
Basic types
Groups documents by field value, such as by status or region.
Groups documents by a combination of multiple fields (composite primary key).
Groups documents by numerical interval.
Groups documents by time interval, such as per hour or per day.
Automatically selects an appropriate time interval.
A variable-width histogram used to handle uneven data distribution or many extreme values.
Groups documents by custom numeric range.
Groups documents by custom date range.
Group by IP address segment.
Grouped by IP prefix.
Geospatial types
Groups documents by distance from a point, such as
0-1kmor1-5km.Groups documents by Geohash grid (latitude and longitude grid).
Groups documents by Google S2 or XYZ tile grid (map tiles).
Groups documents by hexagonal grid.
Filtering and sampling types
Creates a bucket that contains only the documents that match the query.
Creates multiple buckets, each corresponding to a filter condition.
Groups all documents with missing fields into one bucket.
Randomly samples a portion of documents for subsequent aggregation.
Ensures that the values of a specific field are not repeated during sampling.
Performs random sampling based on a sample.
Nested types
Performs aggregation within a
nestedobject.Returns from a
nesteddocument to the root document context.Aggregates child documents in a parent-child relationship.
Aggregates parent documents from child documents in a parent-child relationship.
Advanced analysis types
Finds terms with high significance.
Performs significance analysis on a text field.
Finds low-frequency values.
Finds combinations of fields that frequently appear together.
Automatically categorizes unstructured text.
Other types
Builds a cross-matrix of Boolean conditions for debugging filters.
Syntax keywords for nested aggregations.
Buckets documents by a combination of dimensions. It supports deep paging and is used to export full data.
Creates a global bucket that ignores all filters.
Performs efficient aggregation on time series data.
Metrics aggregations
Classification
Query type
Description
Basic statistics types
Calculates the average value.
Calculates the sum.
Minimum value.
(maximum)
Counts the number of field values, including duplicates.
Counts the number of unique values.
Advanced statistics types
Returns
count,min,max,avg, andsumin a single response.Adds
sum_of_squares,variance, andstd_deviationto the data returned bystats.Calculates percentiles.
The percentile rank corresponding to a given value.
The median absolute deviation, which measures data dispersion.
Geospatial types
Calculates the bounding box of geographic points.
Calculates the centroid of geographic points.
Connects geographic points to form a line.
The bounding box of Cartesian coordinates.
The centroid of Cartesian coordinates.
Data structure types
Returns the top matching documents within a bucket.
Returns the values of other fields that correspond to the optimal value of a certain metric.
Returns the five values required for a box plot chart:
min,q1,median,q3, andmax.Other types
Calculates the statistical relationships between multiple fields, such as the mean, covariance, and correlation coefficient matrix.
Calculates the growth rate or rate.
Calculates statistics for a text field, such as length and character types.
Performs a
t-test to determine if there is a significant difference between the means of two sets of data.Weighted average value.