A search index is built on an inverted index and column store to solve complex query challenges for big data. It provides features such as queries on non-primary key columns, full-text index, prefix queries, fuzzy queries, boolean queries, nested queries, geo queries, and statistical aggregations such as max, min, count, and sum. In Internet of Vehicles (IoV) scenarios, a search index is primarily used to retrieve vehicles based on any combination of vehicle properties.
For more information about search index, see Search index.
Query scenarios
A search index provides the general-purpose query interface (Search) and the data export interface (ParallelScan).
If you want to sort or aggregate query results, or the query request is sent from an end user, use the Search operation
If you do not need to sort query results and want to return all matched results in an efficient manner, or the data is pulled by a computing environment such as Spark or Presto, use the ParallelScan operation.
Compared with the Search operation, the ParallelScan operation supports all query features but does not provide analytics capabilities such as sorting and aggregation. This way, query speeds are improved by more than five times. You can call the ParallelScan operation to export hundreds of millions of data rows within a minute. The capability to export data can be horizontally scaled without upper limits.
Interface | Description |
A full-featured query interface that supports all features of search index.
| |
A concurrent data export interface. It supports the query features of search index but does not support analysis features such as sorting and statistical aggregation. Compared to the Search interface, ParallelScan provides better performance. The performance (throughput capacity) of a single concurrent task is five times that of the Search interface.
|
Core features
Search indexes support multi-dimensional data queries and common statistical analysis. The following table describes the core features of search indexes.
Feature | Description | References |
Queries based on primary key columns and non-primary key columns | Queries that are based on only primary key columns or prefixes of primary key columns cannot meet requirements in some scenarios. If you want to query data based on a non-primary key column, you can include the column in a search index and perform queries based on the non-primary key column. | Documentation for query methods provided by search indexes, such as Basic query |
Boolean query | Boolean queries are suitable for order scenarios. In order scenarios, a table may contain dozens of fields. It may be difficult to determine how to combine the fields required for queries when you create a table. Even if you can determine how to combine the required fields, hundreds of combinations may be available. If you use a relational database service, you may have to create hundreds of indexes. In addition, if no index is created for a specific combination in advance, you cannot query the required data. You can use Tablestore to create a search index that includes all the fields that may be required and freely combine these fields in queries. Search indexes also support logical operators such as AND, OR, and NOT. | |
Geo query | As mobile devices gain popularity, geographical location data becomes increasingly important. Geographical location data is used in various apps such as social media apps, food delivery apps, sports apps, and Internet of Vehicles (IoV) apps. These apps require query features that support geographical location data. Search indexes support queries based on the following geographical location data:
Tablestore allows you to use these features to query geographical location data without the need to use other database services or search engines. | |
Full-text search | Search indexes can tokenize data to support full-text search. You can use search indexes to sort query results based on only the BM25-based keyword relevance score. If you want to sort query results based on other relevance scores, we recommend that you use search systems. Search indexes support the following tokenization methods: single-word tokenization, delimiter tokenization, minimum semantic unit tokenization, maximum semantic unit tokenization, and fuzzy tokenization. You can select tokenization methods based on your requirements. To highlight specific keywords in the query output, you can enable the highlight feature. | |
KNN vector query | Search indexes provide the KNN vector query feature. You can find the most similar data items in large-scale datasets by using vectors to perform approximate nearest neighbor searches. This feature is suitable for scenarios such as recommendation systems, image and video retrieval, and natural language processing (NLP). | |
Fuzzy query | Search indexes support queries based on wildcards. This feature is similar to the LIKE operator in relational databases. You can specify characters and wildcards such as question marks ( | |
Prefix query | Search indexes support prefix queries. For example, in a query based on the prefix | |
Exists query | An exists query is also called a NULL query or a NULL-value query. This type of query is used in sparse data to determine whether a column of a row exists. | |
Nested query | In addition to a flat structure, the data produced by online applications, such as tagged images, often has a complex and multilayered structure. For example, a database stores a large number of images, and each image has multiple elements such as houses, cars, and people. Each element in an image has a unique weight score. The score is evaluated based on the size and position of an element in an image. Therefore, each image has multiple tags. Each tag has a name and a weight score. You can use nested queries to query data based on the tag conditions. Image tags are stored in the JSON format. Example: Nested queries can be used to query data that has multiple layers of logical relationships. This greatly facilitates the modeling of complex data. For Nested fields in complex data structures such as JSON, you can also enable the highlight feature to precisely locate the required information. | |
Collapse (distinct) | Search indexes support the collapse (distinct) feature. You can use the collapse (distinct) feature to collapse the result set based on a specific column. This way, data of the specified type appears only once in the query results to ensure the diversity of the result types. For example, in e-commerce scenarios, if you search for | |
Sorting | Tablestore provides the sorting feature based on the primary key of the data. If you want to sort data based on other fields, you can use the sorting feature of search indexes. Search indexes support sorting in ascending order or descending order based on one or more conditions. The sort operation is globally performed on all data in a search index. By default, the returned results of a search index are sorted based on the primary key in the data table. |
|
Match all query | You can use search indexes to return the total number of rows that match the query conditions. This feature applies to data verification and data-driven operations.
|
|
Aggregation | Search indexes allow you to perform aggregation operations to obtain the maximum value, minimum value, average value, sum, count, and distinct count of rows, percentiles, and histogram statistics. You can also use search indexes to group results. This allows you to perform lightweight statistical analysis. |
Use a search index
You can use a search index to query data in the following ways.