Engine performance optimization

Lock policy for vector indexes

Configure the lock policy for vector indexes

For business scenarios that require high-performance vector retrieval, you can modify the online table configuration to lock the vector index in memory. This improves retrieval efficiency. You can configure the policy as follows:

1. On the Deployment Management page, click Data Node.

   

2. Under Online Table Configuration, select the target index table and click Edit.

   

3. Under load_config, add the following configuration. For more information, see Index table loading policies.

   {  
           "file_patterns": [
             "/index/${vector_index_name}"
           ],
           "load_strategy": "mmap",
           "load_strategy_param": {
             "slice": 409600,
             "lock": true,
             "interval": 2
           }
         },

4. After you save the changes, the cluster performs a rolling update based on the minimum service ratio specified in the online configuration.

   

When you operate a production cluster, ensure that the minimum service ratio is greater than 0. Also, ensure that the available data nodes can handle all current traffic during the rolling update.

Notes

• Locking a vector index in memory consumes memory resources on the data node. Modify the lock policy only when the data node has sufficient memory.

Impact of data compression on performance

Recommendations:

• Do not enable compression for frequently accessed indexes, fields, or properties.

• Do not enable compression for hot spot fields.

• Enabling compression significantly affects query performance. Perform a grayscale test on a test cluster to evaluate the impact of data compression on performance metrics, such as query latency and cluster load. Modify the production cluster only after you confirm that the business impact is minimal.

Optimization methods after enabling data compression:

Although enabling compression significantly impacts performance, you can use index table loading policies to partially offset the performance impact. You can configure the policy as follows:

In the cache policy, add the following parameter to load_config > load_strategy_param:

"cache_decompress_file": true

This parameter caches compressed data to reduce the performance impact of enabling data compression.