Vector analysis performance testing

AnalyticDB for PostgreSQL specifications

ECS specifications

Step 1: Configure the connection

Edit ann_benchmarks/algorithms/adbpg/module.py and fill in your instance details:

# Internal endpoint of the AnalyticDB for PostgreSQL instance
self._host = 'gp-bp10ofhzg2z****-master.gpdb.rds.aliyuncs.com'

# Port number
self._port = 5432

# Database name
self._dbname = '<database_name>'

# Database account name
self._user = '<user_name>'

# Database account password
self._password = '<YOUR_PASSWORD>'

Step 2: Configure index and search parameters

Edit ann_benchmarks/algorithms/adbpg/config.yml with your target dataset and index mode.

Index creation parameters (arg_groups) — for details on creating a vector index, see Create a vector index.

Search parameters (query_args):

Tuning for 95% recall rate

The test targets a recall rate of ≥95% with top-10 results. When recall falls short, adjust parameters in this order:

1. Increase ef_search first — this is the most direct lever for recall rate and has a predictable effect.

2. If recall is still insufficient after increasing ef_search, increase max_scan_points.

3. Once recall meets the target, reduce max_scan_points incrementally to maximize QPS while keeping recall at ≥95%.

AnalyticDB for PostgreSQL provides the following reference configurations for each test dataset, tuned to achieve ≥95% recall:

# GIST 960
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 120}]
        query_args: [[ {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 1}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 3200, pq_amp: 10, parallel: 10}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 15}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 20}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 25}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 30}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 50}]]

# Deep 96
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 12}]
        query_args: [[ {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 1}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 10}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 15}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 20}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 25}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 30}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 50}]]

# Cohere 768
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 96}]
        query_args: [[ {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 1}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 10}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 15}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 20}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 25}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 30}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 50}]]

# Dbpedia 1536
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 192}]
        query_args: [[ {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 1}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 10}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 15}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 20}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 25}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 30}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 50}]]

Step 3: Test the recall rate

Run the following command to start the recall rate test:

nohup python run.py --algorithm adbpg --dataset <Dataset> --runs 1 --timeout 990000 \
    > annbenchmark_deep.log 2>&1 &

Replace <Dataset> with the dataset parameter from the table above (for example, gist-960-euclidean).

After the test completes, query the results:

python plot.py --dataset <Dataset> --recompute

Sample output:

0:    ADBPG(m=64, ef_construction=600, ef_search=400, max_scan_point=500, pq_amp=10)        recall: 0.963       qps: 126.200
1:   ADBPG(m=64, ef_construction=600, ef_search=400, max_scan_point=1000, pq_amp=10)        recall: 0.992       qps: 122.665

If recall does not reach 95%, adjust ef_search and max_scan_points as described above and re-run the test.

Step 4: Test the search performance

Vector search benchmarks measure two distinct characteristics depending on your use case:

• Latency (parallel: 1): Measures how fast the system responds to a single query. Use this to evaluate real-time search scenarios where low response time is critical.

• Throughput (parallel: 5 to parallel: 50): Measures how many queries the system handles per second under concurrent load. Use this to evaluate batch or multi-user search scenarios.

After confirming that the recall rate meets your target, run the throughput test with the --batch flag:

nohup python run.py --algorithm adbpg --dataset <Dataset> --runs 1 --timeout 990000 \
    --batch > annbenchmark_deep.log 2>&1 &

After the test completes, open annbenchmark_deep.log to review queries per second (QPS), average response time (RT), and P99 RT at each concurrency level:

2023-12-20 17:31:39,297 - INFO - query using 25 parallel
worker 0 cost 9.50 s, qps 315.92, mean rt 0.00317, p99 rt 0.00951
2023-12-20 17:31:49,097 - INFO - QPS: 7653.155
2023-12-20 17:31:49,113 - INFO - query using 30 parallel
worker 0 cost 13.87 s, qps 216.36, mean rt 0.00462, p99 rt 0.04298
2023-12-20 17:32:03,260 - INFO - QPS: 6361.819
2023-12-20 17:32:03,281 - INFO - query using 50 parallel
worker 0 cost 20.78 s, qps 144.36, mean rt 0.00693, p99 rt 0.02735
2023-12-20 17:32:24,385 - INFO - QPS: 7107.920

Choose an index mode

Select the index mode that matches your workload before reviewing per-dataset results:

Index creation time reflects the one-time cost to build the index before the instance serves queries. It does not affect query performance after the index is built, but plan for it when sizing initial data load time, especially for large datasets.

Instance specifications: 8 cores and 32 GB × 2 compute nodes