MS MARCO vector benchmarks
The MS MARCO workflow compares pg_search, pg_vectorsearch, and VectorChord on
113,520,750 rows with 1,024-dimensional vectors. Each engine restores the same
PostgreSQL 18 heap backup (20261102-224214F) and runs on a c8gd.metal-24xl
runner with a 8-CPU / 32-GiB database container on cores 0–7. All engines use
16 GiB shared memory, 8 GiB shared buffers, 8 GiB maintenance_work_mem,
24 GiB effective_cache_size, and 256 MiB work_mem. k6 runs on cores 64–67.
CI overrides the pinned pdbench Compose configuration for every engine to use 8 CPUs, eight global parallel workers, and four workers per parallel query. The maintenance-worker limit is seven; VectorChord retains eight internal build threads. The worker-process pool remains 32. Query-session settings also use four workers for recall calibration and k6. Memory settings remain unchanged.
Result dataset names include -8cpu-32gb, keeping these baselines separate from
previous 64-CPU runs. The resolved Compose configuration is included in the
run diagnostics as runtime-compose.yml. The RunsOn host remains unchanged;
these limits apply to the database container.
Run
Apply a PR label to run one engine:
| Engine | PR label |
|---|---|
pg_search |
benchmark-msmarco-pg_search |
pg_vectorsearch |
benchmark-msmarco-pg_vectorsearch |
vectorchord |
benchmark-msmarco-vectorchord |
vectorchord-prefilter |
benchmark-msmarco-vectorchord-prefilter |
The label is removed when the run starts. Reapply it to rerun. For an on-demand
runner, add benchmark-nonspot before the engine label. Remove it to use Spot.
Daily runs benchmark all three engines with VectorChord prefilter off. Manual runs
can also select vectorchord-prefilter to enable it; this variant has separate
results. Both VectorChord modes use the same unfiltered MS MARCO queries.
Manual runs select one engine:
gh workflow run benchmark-pg_search-benchmarker-msmarco.yml \
--repo paradedb/paradedb --ref main -f engine=vectorchord
Add -f runner_type=on-demand for on-demand runners; Spot is the default.
Set commit to benchmark another pg_search revision. Scheduled and manual runs
on main update each engine’s baseline unless commit is set.
Measurements
All engines use the same first 100 held-out vectors and published ground truth
for unfiltered cosine top-10 and top-100 queries returning _id, title.
pdbench-load finalize builds and vacuums the index. pdbench search tunes
probes for 90%, 95%, and 99% recall. Reached targets run in k6 with one VU,
10 seconds of warmup, and 50 seconds of measurement. Targets that miss recall
have no QPS or latency result.
Plan checks require the expected vector index, and k6 checks the returned neighbor count. Results include QPS, latency, buffer counts, build time, index size, and peak build memory. Each engine has its own artifacts and baseline history.
The TOML profiles contain index and recall settings. The workflow pins tool and extension revisions. Cluster setup, extension installation, build reporting, and k6 profiling use the same actions as HN. VectorChord’s 100,000-list profile uses the memory-saving training settings described below; a completed 113M run with this configuration is still pending.
Diagnostics
Download the engine’s diagnostics artifact for plans, raw reports, k6 dashboards, database logs, and memory traces:
- Build samples:
benchmark-artifacts/load/*/*_load/points.jsonl. The published peak usescontainer_memory, which excludes inactive page cache. Timestamps are UTC and values are bytes. - Query samples:
containers.paradedb.memoryinbenchmarker/results/<operating-point>.json. Timestamps are Unix milliseconds and values are bytes. The collector subtracts cache when available, so these values use a different memory definition from the build peak.
Run the shared Python checks locally:
python3 -m unittest discover -s benchmarks/cohere -p 'test_*.py'
VectorChord comparison profiles
The VectorChord profiles use 100,000 lists for MS MARCO,
hierarchical k-means in 100 dimensions, sampling factor 64, eight build threads,
build.pin=2, residual quantization, and spherical centroids. Probe sweeps are
capped at the list count. Both build and search use the shared 32 GiB container
configuration, with 8 GiB shared buffers and 8 GiB maintenance_work_mem.