mentat scale benchmark: duckdb-quack

Environment (abridged; full detail in env.txt)

instance_type:   c7i.8xlarge
kernel:          6.12.103-129.197.amzn2023.x86_64
cpu_model:       Intel(R) Xeon(R) Platinum 8488C
cpus:            32
sockets/numa:    1 1 
mem_total_gib:   61.8
hugepages:       HugePages_Total:=0 HugePages_Free:=0 Hugepagesize:=2048 
data_fs:         /dev/nvme0n1p1 xfs   150G   14G  137G  10% /
mentat_git:      d34e0177 + duckdb-quack backend
duckdb:          1.5.5
postgres:        n/a

Dataset and PG size vs shared_buffers

s: 984803 datoms (1600 users, 130000 issues, 6492 history updates)
m: duckdb-quack reads a copy of the embedded-built store (as ext-cache did)

bulk_load

backend scale datoms wall s datoms/s ANALYZE s VACUUM s store size note
duckdb-quack s 984,803 67.6 14,565 0.19 GiB

point_lookup

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 3367 2.889 3.649 4.939 6.875 336.7 0 1
duckdb-quack m 9,850,258 1 read 2850 3.454 4.078 4.303 123.795 284.9 0 1

ref_traversal

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 237 42.203 43.332 43.621 45.425 23.7 0 1
duckdb-quack m 9,850,258 1 read 30 500.107 506.938 507.900 507.900 2.0 0 1

aggregate

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 106 94.838 96.119 97.778 99.150 10.5 0 1
duckdb-quack m 9,850,258 1 read 30 1,100 1,121 1,123 1,123 0.9 0 1

predicate_scan

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 280 35.317 39.007 40.108 41.413 28.0 0 1
duckdb-quack m 9,850,258 1 read 30 380.790 1,837 2,408 2,408 1.0 0 1

pull

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 3585 2.753 3.486 3.659 4.856 358.5 0 1
duckdb-quack m 9,850,258 1 read 1978 5.001 7.322 8.885 12.701 197.8 0 1

as_of

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 113 89.270 90.072 90.795 91.744 11.2 0 1
duckdb-quack m 9,850,258 1 read 30 1,103 1,120 1,135 1,135 0.9 0 1

since

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 1247 7.962 8.525 11.379 12.633 124.6 0 1
duckdb-quack m 9,850,258 1 read 191 52.415 53.935 57.554 58.838 19.0 0 1

input_bindings

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 2829 3.516 4.070 4.217 5.246 282.9 0 1
duckdb-quack m 9,850,258 1 read 1915 5.197 5.611 5.811 10.284 191.5 0 1

write_mixed

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 write 8389 6.780 8.811 17.284 30.712 139.8 0 1
duckdb-quack s 984,803 8 read 16115 44.664 71.155 83.037 125.830 268.4 0 1
duckdb-quack m 9,850,258 1 write 7205 7.806 11.795 15.859 43.360 120.1 0 1
duckdb-quack m 9,850,258 8 read 1092 555.242 1,490 2,042 2,258 17.7 0 1

concurrency_sweep

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb-quack s 984,803 1 read 624 3.838 42.056 42.397 43.665 62.4 0 1
duckdb-quack s 984,803 8 read 4601 3.124 52.983 64.564 66.521 458.8 0 1
duckdb-quack s 984,803 32 read 11092 3.675 73.167 82.188 1,147 1,030 0 1
duckdb-quack s 984,803 64 read 11142 6.407 90.591 1,108 2,537 1,010 0 1
duckdb-quack s 984,803 128 read 11333 9.435 1,056 1,547 5,647 1,014 0 1
duckdb-quack m 9,850,258 1 read 59 4.290 505.275 507.038 507.038 5.8 0 1
duckdb-quack m 9,850,258 8 read 470 4.733 526.820 547.811 562.513 44.8 0 1
duckdb-quack m 9,850,258 32 read 1117 9.364 857.612 903.347 1,070 104.8 0 1
duckdb-quack m 9,850,258 64 read 1169 26.533 1,922 2,084 2,945 104.8 0 1
duckdb-quack m 9,850,258 128 read 1195 49.104 3,505 3,684 4,849 97.2 0 1

sustained

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
duckdb m 9,850,258 1 write 40154 4.787 17.282 29.491 272.002 133.8 0 1
duckdb m 9,850,258 32 read 11134 997.484 2,944 4,264 7,481 37.0 0 1
duckdb-quack m 9,850,258 1 write 15978 15.946 40.401 56.994 458.566 53.3 0 1
duckdb-quack m 9,850,258 32 read 16166 920.525 1,407 1,694 3,891 53.7 0 1

Correctness checks

52 PASS, 0 FAIL, 0 SKIP; 4 backend check runs OK, 0 FAILED (all FAIL/FAILED lines, with annotations, in checks.txt)