Contents

Changelog

All notable changes to pgColumnar are recorded here. The format follows Keep a Changelog. pgColumnar is pre-release; the version marker is 1.0-alpha4, recorded in VERSION. New tables are written in the native on-disk format, PGCN v1. For the forward-looking plan see design/ROADMAP.md; for full history see the git log.

The extension’s default_version is 1.0-alpha4. v1.0-alpha4 is the latest published pre-release. Upgrade scripts from every previously shipped version ship with it (1.0-dev, which the v1.0-alpha tag installed, 1.0-alpha, 1.0-alpha2, and 1.0-alpha3), so a single ALTER EXTENSION pgcolumnar UPDATE reaches 1.0-alpha4 from any of them. Older notes in this file describe default_version as pinned at an earlier version, each true until the next version shipped.

[1.0-alpha4] - 2026-09-17

Fixed

  • docs/features.md promised an encoding choice the writer does not make (#1074).

    The section said “Each chunk takes the encoding that makes it smallest”. It does not. FSST is kept only when it beats the alternative by fsst_min_gain_percent. Both sides are measured after the block codec has run. Storing the FSST codes uncompressed is never one of the options compared. A codec strong enough to shrink the plain text past that margin drops FSST and can write more than none does.

    Reproduced on 200,000 rows of random hex text. Three arms write from one materialised corpus, so every arm sees identical input. The codec is read back from pgcolumnar.column_chunk.block_codec rather than assumed from the GUC:

    arm    page bytes    vs none    block_codec
lz4    10,663,464    99.997%    none x4, lz4 x4
none   10,663,789    100%       none x8
zstd   10,853,334    101.777%   zstd x8

    The descriptor length is what names the mechanism. On the two text columns it is 1870 and 1897 bytes under none. That is the FSST symbol table. Under zstd it is 280, which is FSST gone. lz4 is unaffected because it declined on half the chunks and FSST stayed.

    Only the documentation changes here. The encoding cascade is untouched, and the fix #1074 proposes belongs with its own margin tests rather than beside a release.

  • Two more suite headers told a reader to undo what had already been done (#1088).

    fcfd3e6 fixed this class in hilbert_cluster.sh. Two files still carried it, and every claim in them is false against the tree:

    hilbert_locality.sh   "DELIBERATELY NOT REGISTERED ... makes that arm red
                      until then"          registered; 070's arm PASSES;
                                           harness_selftest 967 + 0
hilbert_curve.sh      "NOT REGISTERED"     registered
                      "src/columnar_curve.c does not exist"
                                           127 lines, links into the .so
                      "070 reports it UNREGISTERED, which is the accurate
                       state"              070 reports the opposite

    hilbert_curve.sh also said registering the suite “belongs in the commit that adds the encoder”. That commit landed.

    hilbert_locality.sh IS THE FILE THE PUBLISHED ADVICE CITES. features.md, how-to.md and best-practices.md quote 1.24x to 2.04x against Z-order, and that range comes from this suite’s pins – 2.0424, 1.6794, 1.4627 at ROWS=200000. A reader arriving from the docs to check the number was told the file is not in the matrix and is expected to redden the selftest.

    THE SWEEP THAT FOUND THEM NEEDED ITS CONTINUATIONS JOINED, which is why the class survived the first pass. hilbert_locality.sh wraps the phrase as makes that arm / # red until then across two comment lines, invisible to a line-oriented grep, and the search reported nothing. Reported by @OffgridwithJD, who checked before concluding the file was clean.

    So the rule for guarding this class is to match on REGISTRATION STATUS, which is structural, and never on the prose, which wraps. The sweep now used joins comment continuations first and compares each claim against the SUITES array.

    Comment only. No check name moves: 3cfc50038951 and bf6154cc7070 before and after. Both suites green on PG17 non-assert, 65 and 184 checks, and harness_selftest unchanged.

  • Hilbert clustering shipped in alpha4 and the pages a user reads never mentioned it (#1043 for the suite header).

    pgcolumnar.cluster_hilbert and pgcolumnar.recluster_hilbert were documented in docs/sql-reference.md and NOWHERE ELSE. features.md, how-to.md and best-practices.md all describe clustering as Z-order and name only cluster and recluster:

    sql-reference.md   cluster_hilbert 5 hits, recluster_hilbert 2
features.md        0
how-to.md          0
best-practices.md  0
user-guide.md      0

    So a reader following the discovery path picked Z-order and never learned the other option existed. It is the headline item of the release whose theme is skipping and layout, with a measured 1.24x to 2.04x advantage on range filters.

    All three pages now carry the verbs, the measured advantage, and the rule for choosing: Z-order for point lookups, Hilbert for ranges, measure when they look close. how-to.md also states that the curve is sticky ON THE SAME KEY, because that is the part which surprises people.

    THE QUALIFIER IS THE WHOLE CLAIM AND THE FIRST DRAFT DROPPED IT. sql-reference.md already said “plain cluster and recluster ON THE SAME KEY maintain that curve”; the paraphrase written for how-to.md said it unconditionally. cluster_inherited_curve requires sort_key_matches, which compares the recorded key position by position, so the column ORDER is part of it:

    cluster_hilbert('h','a','b')   sorted_kind hilbert
recluster('h','a','b')         sorted_kind hilbert
recluster('h','b','a')         sorted_kind zorder

    The page now carries that sequence, because a reader copying the paragraph is exactly who needs it. Caught in review by @OffgridwithJD; the function’s own header calls the condition the thing that keeps “sticky” from meaning “unescapable”.

    AND THE SUITE HEADER TOLD A READER TO UNDO THE FEATURE (#1043). test/hilbert_cluster.sh said the suite is red on purpose, that neither verb exists, and that registering it in the matrix is future work. All three were true when written; 4b66555 carried out every instruction in them and left the paragraphs in place. Verified against the tree: both verbs are defined in pgcolumnar--1.0-alpha4.sql, the suite is registered, and it is green on all five majors.

    A stale instruction is worse than a stale fact, because it tells the next person to undo what was done. The one sentence that is still true is kept, and it is the trap the paragraphs existed to name: a shim renaming the Z-order verbs passes 162 of 181 arms, and S5 is green on it BY CONSTRUCTION because over one column both curves are the identity.

    No check name moves: the sorted name list hashes 72eb1c4e4f42 before and after.

  • META.json named an install script the distribution does not contain, and nothing had ever read it.

    It is the PGXN distribution metadata. It hardcodes the version twice and names the base install script by filename, and no suite, no Makefile rule and no CI step consumed it. It went stale for the whole alpha4 cycle:

    version                    1.0.0-alpha.3     while VERSION said 1.0-alpha4
provides.pgcolumnar.file   pgcolumnar--1.0-alpha3.sql

    THE FILENAME IS THE HALF THAT MATTERS. pgcolumnar--1.0-alpha3.sql does not exist: this repository opens each cycle by RENAMING the base script to the new version. So the published metadata pointed at a file PGXN would not receive, and the two version strings were only the visible symptom. A reader comparing those against VERSION would have called it a cosmetic lag.

    Fixed, and now gated in both harnesses. docs_style.sh already had a section for exactly this class – documents that hardcode a version beside a citation of the file holding it – and META.json is the same shape with a filename added. test/pytest/test_pgxn_metadata.py asserts the same properties through its own parse and its own git archive.

    git archive rather than a filesystem test throughout, because the question is what the DISTRIBUTION contains. An export-ignore can drop a file that is plainly present on disk, which this repository has been bitten by before.

    A fourth arm covers what the third cannot: provides.file is ONE filename, so an export-ignore dropping a different install script would leave it green while ALTER EXTENSION ... UPDATE broke for anyone who installed from PGXN. It is on BOTH sides rather than in python alone, because the pytest guards run in CI while the five-major shell matrix is the release gate, and an arm protecting the upgrade path for published installs belongs in the gate that runs before a tag.

    ITS FIRST SHELL VERSION TRIPPED selftest/080, which is the rule working on arrival: the arm was fine in the pytest twin and only became subject to the no-pipe sweep when it entered the shell harness. The line was

    printf '%s\n' "$_meta_ship" | grep -qx "$_ms"

    a builtin writing a captured string into a reader that exits on its first match, which is #486 exactly. $_meta_ship is the whole git archive | tar -t listing, 4999 bytes on this tree, right at the pipe-buffer boundary where the shape works almost every time. Replaced with a case over newline-sentinelled text, which gives the anchored match grep -x provided and starts no process. The failure direction was a false RED rather than a false green, so noisy rather than dangerous, but it was red in the gate the arm had just been moved into.

    The PGXN form is DERIVED from VERSION rather than hardcoded – 1.0-alpha4 to 1.0.0-alpha.4, and a future 1.0-beta1 to 1.0.0-beta.1 – so the arms do not need editing at the next release. The transform has its own arm, because a derivation is a claim too: were it wrong, every other arm would compare against a wrong expectation.

    Removal proof, both harnesses: restore META.json as it shipped and every substantive arm reddens while every premise stays green. The premises hold because the file still parses and still names a script; it names the wrong one, which is the distinction the arms draw.

    docs_style.sh   3 arms red, premises green
pytest          2 tests red, 9 pass + 2 fail + 0 unrun = 11

    THE TWO COUNTS DIFFER AND THAT IS NOT DRIFT. docs_style.sh splits the version property across two arms with its _mk loop, one for version and one for provides.pgcolumnar.version, while the pytest twin asserts both inside a single test. Three and two are the same three assertions counted by their harness’s own unit. An earlier revision of this entry said “two on each side”, which the table beside it already contradicted (@OffgridwithJD).

    guard_tests re-derived by collection 342 -> 346; cluster_tests unchanged at 410, checked in the same run rather than assumed.

  • One geometry arm accepted a comparison the property does not describe (#1081).

    1081 replaced a count of entry->fileOffset != rg->fileOffset with a per-field

    membership loop, so that dropping any one of the four compared fields reddens by name. The loop matched entry->$_f != rg->$_f OR entry->$_f != natts for every field, because natts is the one field compared against the scan’s own column count rather than against the row group.

    Harmless today – entry->firstRowNumber != natts appears nowhere – and still wrong as a claim: each arm would accept a comparison its own name denies. Three fields now match only the rg form, and natts has its own arm and its own name.

    Reported by @jdatcmd on #1081’s review.

    All four verified by removal on the real suite, each reddening only its own arm:

    firstRowNumber removed   FAIL  a hit re-checks the group's firstRowNumber ...
rowCount removed         FAIL  a hit re-checks the group's rowCount ...
fileOffset removed       FAIL  a hit re-checks the group's fileOffset ...
natts removed            FAIL  a hit re-checks the group's natts against the
                               scan's column count
restored                 26 passed, source byte-identical

    firstRowNumber is the one #1081 shipped unproven. Its mutation did not apply – that line starts (entry-> rather than entry->, so the pattern missed – and the harness’s applied-assertion reported it rather than counting a clean run as a pass.

Added

  • Hilbert clustering: pgcolumnar.cluster_hilbert and pgcolumnar.recluster_hilbert (#889).

    Two verbs rather than a parameter on the existing two. PostgreSQL refuses to extend cluster(regclass, VARIADIC name[]) in either direction: a defaulted parameter cannot precede a VARIADIC one, and an array-plus-kind overload makes the documented cluster('t','a','b') call ambiguous. Both were measured on 18.4. The new verbs match their siblings element for element in argument types, variadic element type, return type and volatility, so a caller switches between them by name alone.

    What the curve buys. Z-order jumps a long way in key space at a bit boundary; a Hilbert curve does not. Keys that are close in the data therefore stay closer in storage, the min/max zone maps over the clustered columns are tighter, and a range filter reads fewer chunk groups. The key is the same width and sorts through the same bytea comparator, so nothing downstream of the sort knows which curve produced it.

    The curve is sticky. sorted_kind is the table’s declared intent, not a property of each call:

    • plain recluster on a Hilbert table over the same key is a no-op returning 0, not a silent conversion back to Z-order;
    • recluster_hilbert on a Z-ordered table over the same columns rewrites it;
    • vacuum_sorted leaves a Hilbert table alone rather than sorting it lexicographically and relabelling it;
    • the maintenance daemon dispatches on the recorded kind, so a Hilbert table is re-clustered with Hilbert instead of being converted on a timer;
    • naming the other verb, or reclustering on a different key, is how a table changes curve.

    Held by test/hilbert_cluster.sh (181 arms) over the SQL surface, the recorded kind, both self-gates and the daemon, and by test/hilbert_curve.sh (184 arms) over the encoder itself.

  • Six more guards counted their callers instead of pinning their property (#1078’s class). Two were blind to the defect they name.

    1078 repaired two arms in native_fetch_projection.sh that compared a whole-file

    grep -c against a literal. A sweep of test/ found twelve sites of that shape; these are six of the remaining ten, in native_fetch_cache.sh and native_fetch_position.sh.

    THE REPAIR DIFFERS PER ARM, BECAUSE THE FAILURE DIRECTION DOES. A count over a call site is blind; a count over a guarded FORM is blind and noisy; and a count is CORRECT where the count is the property. Two arms in the sweep were left alone for that reason – decode_interrupts.sh’s ^#define COLUMNAR_DECODE_INTERRUPT(i) would be a redefinition if it appeared twice, and native_saop_pushdown.sh’s premise is load-bearing for an awk range that would silently concatenate two expressions.

    the entry key    both directions   -> self-referential, keyed N of N
the cid reject   noise only        -> scoped to the function that must contain it
the geometry     blind to 3 of 4   -> membership over all four compared fields
the discard      proxy for "where" -> the two functions named
rank, valOffset  noise only        -> scoped to pgcolumnar_fetch_row

    Measured, every mutation compiling so the suite rebuilds and runs end to end:

    case                        OLD arms            NEW arms
unkeyed group lookup        key=1     PASS      RED  keyed 1 of 2
second keyed lookup         key=2     RED       24 passed
rowCount dropped            geom=1    PASS      RED  rowCount
executor-end discard gone   discard=1 RED       RED  names the function
third discard call          discard=3 RED       24 passed
rank replaced by a walk     rank=0    RED       RED  rank prefix
    Both blindness rows are the case for this change: an unkeyed lookup and a dropped geometry field both leave the old arms green. The two noise rows are what fired on

    1077 and cost a correct PR a red.

    No ledger change: native_fetch_cache and native_fetch_position have zero rows, so they are two of the 249 uncovered suites and no check name here is a ledger key. Verified rather than inherited.

  • The piped-loop sweep reported a clean tree without reading one (#1033).

    selftest/400 proves its detector FIRES – a fixture with a check inside a piped loop gives 1 – and nothing proved it had EXAMINED anything. Without nullglob a wrong $PGC_TESTDIR leaves both globs LITERAL, awk opens no file, grep -c . over no input prints 0, and the arm compares that 0 against 0 and passes:

    PGC_TESTDIR=<a real dir with one offender>   hits=1   detector fires
PGC_TESTDIR=/nonexistent                     hits=0   ARM PASSES, nothing read

    The same file already gets this right 260 lines above, where a different sweep carries [ -e "$_sk_f" ] || continue and a premise: the sweep classified a corpus of check-calling files arm. One sweep was premised and the other was not.

    The population is now counted by what awk actually OPENED – FNR == 1 fires once per file it reads – rather than by ls, so a file that exists and cannot be read is a miss rather than an invisible one, and the premise uses the same mechanism as the detector so the two cannot drift apart. It is RECONCILED against what the globs offered rather than floored at a constant: a literal glob offers 2 words and reads 0.

    Removal proof: pointed at a nonexistent tree, no suite calls a check inside a piped loop still PASSES – which is the defect – while both premises redden.

    Closes the third of #1033’s three gaps; ae008c2 closed the other two.

  • test/pytest/test_projections.py: the multiple-projections DDL, catalog and read path, ported from test/projections.sh (#432). All 75 of its check names, one for one.

    A projection is a second copy of some columns, and every property is about the copy staying honest: it holds the rows the base holds, it loses the rows the base loses, it survives a vacuum that renumbers every row underneath it, and the planner reads it only when it can answer the whole query from it. A wrong projection is a WRONG ANSWER rather than a slow one, because nothing downstream re-checks.

    THE PORT IS STRICTLY STRONGER IN ONE PLACE, and it is worth saying which way. The original’s expect_fail helper runs the statement and passes when it errors AT ALL, so a misspelt table name satisfies every one of its eight refusal arms. The port asserts the SQLSTATE, and every code was measured against this build rather than guessed – 42710, 42703, 22023, 42701, 22023, 42809, 22023, 42704, 42704. The names are the bash suite’s; the assertions are not.

    Three other mechanism changes assert the same property by a stronger means: the EXPLAIN grep becomes a typed JSON field and reads the projection NAME rather than its presence; pgc_set_hash becomes expect.row_set, order-blind by declaration rather than by construction; and the second MVCC session becomes a second connection rather than a background psql on a fifo polled for a token, which removes the wait rather than shortening it.

    Mutation proof, each asserting it applied before its result was believed: removing the

    875 projection-writer reset reddens both directions of the latch (got 105 want 116

    on the mid-transaction add, and the orphan storage the mid-transaction drop leaves); forcing the planner to refuse every projection reddens the covering query and the post-vacuum planner arm with got None want 'pc', which is what proves the port reads the name rather than the presence. Restored, byte-identical: 75 checks, 0 fail.

  • test/pytest/test_sorted_pathkeys.py: the ordered-scan surface, ported from test/sorted_pathkeys.sh (#432). All 110 of its check names, one for one.

    The bash suite pins one decision: when a columnar scan may hand the planner PATHKEYS, the promise that its rows already arrive in a stated order. The planner then drops the Sort above the scan, and nothing downstream re-checks. So a wrong promise is not a slow plan, it is WRONG ROWS.

    The port keeps the original’s three shapes rather than reorganising by feature: a CLAIM arm (the Sort goes), a REFUSAL arm (something made the claim untrue and the Sort comes back), and an ANSWER arm (the rows themselves, against a heap table built from the same data). The ANSWER arm is not a duplicate of the CLAIM arm – dropping the Sort is only correct if the rows arrive sorted anyway, and a plan check alone cannot say whether they did.

    THE ONE ARM THAT NEEDED MORE THAN A PORT is pgcolumnar.parallel_copy, which prepares one transaction per worker. max_prepared_transactions cannot be raised without restarting the postmaster, and the default is 0, so asking for fewer workers does not help. pgc_cluster now sets it where it writes postgresql.conf, at the value lib.sh gives this suite through PGC_EXTRA_CONF. Refusing the arm instead was measured and rejected: it loses three names outright (cannot_run records under the REASON CODE, #1040 phase 0b) AND turns the pytest (cluster tests) job red, because an unrunnable check exits 67 and the job runs pytest under set -euo pipefail. That half exits 0 today with zero unrun, so this file would have been the first to break it.

  • test_docs_cover_the_corpus.py now refuses a NUMBERED SECTION WITH NO BODY.

    The arm above it asks whether each test file is NAMED by a numbered heading. A heading with no body is still a heading, so a section inserted into the gap between another heading and its body leaves both files named and one of them documented under the wrong title – every existing arm green. ## 37. test_iceberg_fdw.py reached main sitting directly above ## 38., with the Iceberg body attached to the userinfo heading. Fixed here, and the section order now matches the bodies.

Changed

  • design/ROADMAP.md’s block-compression entry and alpha4’s plan item record the measurement instead of the claim (#890).

    The entry read as an outstanding task sourced to Zeng et al., VLDB 2024. It was measured on this engine and did not reproduce, and left as-is it would have regenerated the same issue against a different corpus in a year.

    The roadmap carries the five-shape table, the narrow reading of it – the shape the pre-registered rule names as the DECIDER comes out UNDECIDED rather than NET WIN, so “the premise is refuted” is true only of the two most compressible shapes – and the two reasons it retired: no storage tier exists to key a default off, and no shape fires NET COST.

    THE FIRST VERSION OF THIS ENTRY MIXED TWO RUNS IN ONE COLUMN, found in review by @OffgridwithJD. The summary table it was copied from labelled two of its read costs 0.92 (rep) and 0.92 (mix) – shape names from an EARLIER three-shape run, with a different row count and wall-clock rather than instruction counts. Transcribing it dropped the parentheses, which were the only marker that those two values came from somewhere else. Reconciled against the five-shape run row by row, three of the five read costs were wrong:

    shape            C as written     C in the five-shape run
repetitive           0.92                 1.001
text_heavy            --                  0.911
realistic            0.92                 1.001
random_int           1.00                 1.000     matched
incompressible       1.01                 1.010     matched

    Every S value was correct, and no verdict changes: repetitive and text_heavy are NET WIN on S alone, and the three UNDECIDED rows stay UNDECIDED because their C was already below the 1.10 threshold. So the conclusion never depended on the wrong numbers – which is exactly why nothing would have caught them.

    text_heavy’s blank was the costly one. Its real C of 0.911 is the most favourable read cost in the set, so omitting it UNDERSTATED the case the table makes.

    The entry now also carries the conditions – rows, build, counter, and that the codec was asserted from column_chunk.block_codec rather than assumed – plus the limit that an instruction count cannot see an I/O saving at ANY working-set size. #890 learned that when a table went out from an assert build without naming it; the roadmap is where someone goes to reopen this, so the header belongs there more than on the issue.

    Two things kept that would otherwise be lost with it. Dictionary encoding on float columns was a SEPARATE clause of the same roadmap entry, nothing above measures it, and it stays open. And the alpha4 item is marked as having cost nothing in written bytes, so it need not have been alpha work at all – it was scheduled as alpha because the plan assumed it would change the writer, and that assumption was the part that needed checking first.

    Three defects found by the investigation are tracked separately and do not close with it: #1074, #1075, #1076.

Fixed

  • fsst_margin.sh’s low-margin arm looks redundant and is not: it falsifies the descriptor reader.

    The three arms across sections 1 and 2b bracket the FSST decision, and the bracket also falsifies fsst_vectors itself. Nothing said so, and the low-margin arm reads as a weaker duplicate of the margin-90 one, so it is the obvious thing to delete in a tidy-up.

    Measured by evaluating the arms against a stubbed reader rather than argued:

    reader always returns 0    low-margin RED   margin-90 PASS   codec RED
reader always returns >0   low-margin PASS  margin-90 RED    codec PASS
the real tree              PASS             PASS             PASS

    fsst_vectors parses the encoding descriptor by byte offset, get_byte(descriptor, 6 + i * 13), which is the kind of reader that goes silently wrong on a format change and returns a plausible number. With only the margin-90 arm, a reader stuck at 0 reads as “FSST was dropped” and every arm is green.

    AND THE TWO MARGINS ARE THE GUC’S OWN ENDPOINTS. columnar_tableam.c:3269 declares it 5, 0, 99, so the bracket uses the extremes the setting admits rather than an arbitrary low and high. That is what makes each half the strongest form: at margin 0 the keep test is “any compressed win at all keeps FSST”, in the GUC’s own help text, so kept == 0 there would mean FSST never helps on this corpus rather than merely not helping enough. Raising it to 5 to simplify the arm would lose that with nothing to show it went.

    Comment only. No check name moves: the sorted name list hashes 62d1b8a49926 before and after.

    Found by @OffgridwithJD reviewing #1082, which had already merged, and the endpoint reading is theirs too. The table is measured rather than argued, and the range was read from the declaration rather than taken on trust.

  • The arm named “the visibility-only caller decodes nothing” passed on a tree where it decoded (#1077 sweep).

    Two arms in native_fetch_projection.sh counted a CALL SITE, with its argument text, pinned at a literal 1:

    grep -c 'PgColumnarRowIsLive(rel, snap, baseRow)'            "1"
grep -c 'PgColumnarReadRowByNumberCols(rel, snap, baseRow'   "1"

    That asserts a particular call is still written the way it was written, which is not the property either name claims. Plant the regression they exist to catch – a full PgColumnarReadRowByNumber beside the liveness check in the visibility path – and both stay green, because the call they count is still there and the decode added next to it is invisible to them:

    state                          live cols full | OLD1  OLD2 | NEW
clean                             1    1    0 | PASS  PASS | PASS
REGRESSED, full decode added      1    1    1 | PASS  PASS | RED
honest 2nd narrow caller          2    1    0 | PASS  PASS | PASS
empty file                        0    0    0 | RED   RED  | premises RED

    Unlike the allColumns pair repaired in 8e88f42, these fail in ONE direction only. A realistic second caller uses different variable names, so the exact-text count stays at 1 and the old arm is BLIND rather than falsely alarmed. Row 3 is a pass for the old arms by accident, not by correctness.

    THE PROPERTY IS A ZERO. Three entry points exist and only one decodes every column, so the projection path is asserted never to call it. More correct callers of the two narrow entry points move nothing; any full decode moves it off zero – the opposite failure direction from a count pinned at 1.

    Three premises keep the zero from being vacuous: each narrow entry point is called at all, and that PgColumnarReadRowByNumberCols( does not match PgColumnarReadRowByNumber(. That third one DOCUMENTS the assumption rather than providing the guarantee, which the first version of its comment got wrong: a broken prefix would make full >= cols, and cols >= 1 with full == 0 is a contradiction, so the main arm reddens rather than hiding. Corrected in review by @OffgridwithJD.

    Removal proof, against the real suite: md5 c8324ce4f486 -> a2c26702a9cc, full decodes 0 -> 1, 16 passed + 1 failed + 0 unrunnable + 0 skipped = 17, one red naming got [1 full decode(s)] want [0 full decode(s)], all three premises green, restored with an empty diff.

    The fourth and fifth arm in this file repaired for counting a string across a whole file; 8e88f42 did the second and third and the deltuples comment records the first.

  • The docs said pgcolumnar.compression picks a codec. It also picks encodings (#1076), and two things they told users about none are measurably wrong (#1074).

    columnar_encoding.c returns “FSST helps” UNCONDITIONALLY when the codec is none, before it looks at the corpus or the margin:

    if (compressionType == COLUMNAR_COMPRESSION_NONE)
    return true;

    So none is a DIFFERENT cascade, not the same cascade with the codec removed. Nothing in docs/ said so. configuration.md called the setting “default codec for new chunks”, which is exactly the narrow reading that makes the coupling invisible.

    Two user-facing claims fell out of it. administration.md described none as “Lowest write cost, largest size”. Measured on 200,000 rows each, backend instruction counts, zstd against none:

    high-entropy hex text     0.92     zstd CHEAPER to write
repetitive text           0.99     zstd CHEAPER to write

    Neither shape made none the cheaper one, and on the hex shape zstd also produced a table 1.4 percent LARGER. best-practices.md said each chunk “takes the encoding that makes it smallest”, which is wrong twice: the target is the smallest STORED result, so it depends on the codec, and a FSST win below fsst_min_gain_percent is given up on purpose.

    All three pages corrected, with a new Compression and the encoding cascade section carrying the mechanism and both measurements.

    ARMS IN BOTH HARNESSES, and each is a DIFFERENTIAL rather than a bare assertion. “FSST is kept under none” alone is satisfied by any corpus FSST always wins on, so every arm is paired with the same corpus at the same margin WITH a codec, where it is dropped. fsst_margin.sh gains three checks; the independent test/pytest/test_compression_reaches_the_cascade.py builds its own corpus and parses the descriptor itself.

    Removal proof: deleting the early return and rebuilding moves the .so c8e5c790dbac -> 6455728725e5, and reddens exactly the codec arms on both sides while every content invariant stays green:

    fsst_margin.sh    14 passed + 2 failed + 0 unrunnable + 0 skipped = 16
pytest            8 pass + 2 fail + 0 unrun = 10

    The mutant still writes correct rows, which is why the content arms hold: this is a decision changing, not data corruption.

  • The block codec’s buffer was never freed, on either path (#1075).

    flush_one_column compresses the whole encoded region of a column chunk and then leaves the codec’s buffer allocated:

    PgColumnarCompressValueStream(encoded->data, encoded->len, ..., &compData, ...);
if (usedType != COLUMNAR_COMPRESSION_NONE)
{
    finalData = compData; ...
}
if (finalLen > 0)
    appendBinaryStringInfo(chunk, finalData, finalLen);

    BOTH paths leave it dead. When the codec declines, PgColumnarCompressValueStream returns a palloc’d COPY of the raw bytes rather than NULL, by its documented contract, and this caller never reads it – finalData still points at encoded->data. When the codec succeeds, appendBinaryStringInfo has already copied the bytes into chunk.

    It is bounded rather than a leak: flushContext is deleted per row group. What it costs is peak allocation, because it roughly doubles what the flush holds for the encoded region while every other column is still flushing.

    Freed after the append rather than inside either branch, so the success path is covered too. A free placed only in the declined arm is the version that reads as complete and is not.

    MEASURED ON TWO FIXTURES, AND THE FIRST ONE FOUND NOTHING. Peak RSS of the loading backend (VmHWM), lz4, byte-identical input every run:

    200,000 rows, default stripe    baseline 117,178 kB   patched 118,002 kB
600,000 rows, ONE stripe        baseline 326,584 kB   patched 304,979 kB

    The first is +0.70%, the WRONG DIRECTION, and it is recorded because it is the honest half: at that scale the encoded region is a few MB against a 117 MB process and the effect is swamped. The second saves 21.1 MiB, 6.6% of peak, against repetition spreads of 0.14% and 0.27%. The saving is proportional to the row group’s encoded size, and on a default stripe of narrow data it is not observable at all.

    Stored bytes do not move, which is the requirement: 32,055,856 bytes and fingerprint a0959193 identical across every run of both builds.

    NO NEW TEST ARM, deliberately. What changed is peak allocation, and there is no stable way to assert that in CI here – a probe of pg_log_backend_memory_contexts would have to land mid-flush. The correctness requirement is that output does not move, which the existing content suites cover and which was verified by measurement. An arm grepping the source for pfree(codecBuf) would be the exact shape repaired in 8e88f42 and f115d0b.

  • The projection guard fired on a correct caller and stayed green on a wrong one (#1077).

    native_fetch_projection.sh protected one convention – “every column” is an explicit flag, never an absent set – with two arms that counted a string across columnar_reader.c and compared it against a literal 1. That asserts HOW MANY honest callers exist, which is a fact about today’s tree, not the property.

    It errs in BOTH directions, and the second one is why this is a fix rather than a widening. Four source states, the same two arms:

    state        tests guarded flags | OLD arm1  OLD arm2 | NEW
main             1       1     1 | PASS      PASS     | PASS
honest caller    2       2     2 | RED       RED      | PASS
unguarded fn     2       1     2 | RED       PASS     | RED
unguarded inline 2       1     1 | PASS      PASS     | RED

    Row 2 is #1077’s coalescing read: it takes the flag and tests it exactly as the convention demands, and was failed for existing. Row 4 is the defect the arms exist to catch, added inside the existing worker so no new flag is declared – and BOTH arms pass it. arm2 counts the GUARDED form, which is still 1, so a bare bms_is_member(c, needed) beside it is invisible to the arm named “the column test consults that flag rather than a null set”.

    Row 3 shows the one red the old arms do produce is not evidence either: arm1 reddens there for the same reason it reddens on row 2, the count moved. It cannot distinguish a correct caller from a broken one.

    The arms now pin the property against itself – every needed-set membership test consults the flag, guarded N of N – so honest callers move both counts together and an unguarded one moves only the total. Both numbers are printed in the compared strings, so the arm’s message is the reconciliation.

    Two premises were added beside it, because guarded 0 of 0 also satisfies equality: that a membership test exists to guard, and that the flag is declared.

    This is the third arm in this file to be repaired for counting a string across a whole file. The deltuples comment 15 lines above records the first, fixed by scoping; these two were left as whole-file counts and did the same thing again.

  • A parallel custom scan divided its whole run cost by the worker count.

    Core seqscan divides CPU across workers and leaves disk I/O whole. The partial columnar path divided (total - startup) by workers, so an I/O-dominated scan was quoted at half its serial cost with two workers. Measured: serial run 10825, parallel Custom Scan 5412.5 (ratio 2.000). Leaving I/O undivided, the same fixture is 10112.5 (ratio 1.070).

    get_parallel_divisor is static in core; the leader-participation heuristic is reproduced so CPU uses the same divisor a parallel seqscan does.

  • compare_to_bash.py’s corpus arm called a WRAPPED name fabricated. A name too long for one line is written as adjacent literals, and Python joins them at parse time, so the joined name is text the file contains but not text in src can find. The arm exists to catch a reader that CONSTRUCTS a name, so it now collapses the file’s own concatenation and keeps exactly that guarantee: an f-string name still yields a {} template, which the collapse does not rescue. Both directions are asserted.

  • orphan-scan is armed in the matrix runner, so a ledger row naming a deleted check is refused rather than reported (#983, #1015).

    The gate answers “has this run a check the ledger has never seen”. Nothing answered the other half, “does the ledger name a check that no longer exists” – and two rows naming deleted checks sat in the committed ledger from #917 until #983 found them while doing something else. The census counted both. orphan-scan was written for exactly that and had NO CALLER IN THE TREE: tested, and unable to fire on anybody’s change.

    One log per call, not all of them. _by_run returns one entry per LOG, so len(runs) > 1 is true whenever more than one file is passed even when they came from one matrix run. A call shaped like the gate’s $_led_logs is refused by the COUNT, before the before-log/after-log union the message names. So the runner loops.

    --orphans-only, because a skip is not a deletion. By default rc=1 also covers a part that skipped, and that is right: not checked is silence by construction and out of scope, while unprunable is a gap in a part the run claimed and in it. But a skip is box-dependent – part 340 skips only where there is no non-root user to read as – and failing a matrix for that is a gate somebody turns off. The flag changes the QUESTION rather than the severity of an answer, so rc=1 never acquires a second meaning, and the skipped part is still printed.

    Measured before wiring: only part 340 of the 46 has a real check_skip call; 320, 400, 410 and 480 match on comments, a grep-based sweep and printf fixtures. Five majors on one box gave orphans=0, unprunable=0 – but with 108 PASS and 0 SKIP, so the skip path never fired and that run tested nothing about skips. The skipped-part case was forced synthetically instead, and the tool classifies the siblings unprunable rather than orphans.

  • Userinfo in an object-store ENDPOINT was accepted, and the diagnostic told the operator to allow-list it (#995).

    997 closed s3://u:p@bucket/key – userinfo in the URL the caller writes. The

    endpoint the OPERATOR configures was still unguarded, and the bucket guard cannot see it because it is not in the URL at all.

    TWO SHAPES, AND ONLY ONE WAS EVER CAUGHT. Measured on the authority parse:

    http://u:p@host:30829   -> host "u",          port 0       refused, wrong reason
http://user@host:30829  -> host "user@host",  port 30829   port VALID
    The first has a colon inside the userinfo, so the split lands there and the port becomes atoi("p@host:30829") = 0. The second has no such colon: the real port survives, the @ rides along in the host, and the invalid-port refusal never fires.

    995 measured the first and concluded “refused as invalid host or port”, which is

    true of that shape rather than of the code.

    The second shape is why this is a guard and not a message change. Its refusal came from the allow-list, naming the host it could not match, and the hint then said:

    ALTER SYSTEM SET pgcolumnar.objstore_allowed_endpoints = 'user@host'

    A diagnostic that invites widening a security boundary to accommodate a parse bug is worse than a wrong error code. Following it moves the failure from the allow-list to a DNS miss.

    The guard sits BEFORE the scheme and region demands rather than at the authority parse eighty lines later. Placed there it is unreachable whenever no region is configured: measured, every endpoint arm reported requires a region option until it moved. That is the same reasoning the bucket guard carries, and the trap #995 named.

    Arms in test/objstore_userinfo.sh beside the existing ones, with a clean-endpoint control in the same run, and in test/pytest/test_objstore_endpoint_userinfo.py independently.

  • A test file documented as an unnumbered ### section was invisible to every arm that checks TESTS.md (#1024).

    Not in the numbering, so 1..N with no gap never saw it. Not in the contents, so the link arms never saw it. Still NAMED in the document, so the coverage arm was satisfied. test_iceberg_fdw.py shipped that way in #1057 and sat undetected.

    1024’s own report – two PRs each taking the next section number – is no longer

    open: test_the_contents_list_is_numbered_in_order landed after #1023 and catches a duplicate and an inversion in one rule. Planted, it reddens. This change keeps that guard and closes the remaining shape, which is cheaper to hit: the collision needs two PRs in flight, a heading at the wrong level needs one person.

  • A bash suite that unrolls a family as literals graded MISSING against the port that parametrises it (#1045 class 3).

    bash   diff_query "c_int range" ... "c_text range"        11 literals
port   @pytest.mark.parametrize("col", sorted(RANGES))
       expect.row_set(c, h, f"{col} range")               1 template

    The port asserts every one of the 17 properties – RANGES holds the same 11 columns bash unrolls and EQUALITIES the same 6 – but a literal never meets a template, so differential was declared INCOMPLETE for a spelling.

    The grader now resolves the container and emits one CONCRETE name per member, matched literally on both sides. The alternative, widening _template so a literal matches a template, resolves the same 17 and gives up the ability to ever detect them going wrong: c_bytea range would match {} range whether or not the port covers c_bytea. Measured – drop c_bytea from RANGES and the expansion reports it by name, where a widened template cannot.

    THE EXPANSION IS ADDITIVE, and that is a constraint rather than a convenience. Where BOTH sides are templated the template IS the match: 11 of hilbert_locality’s 30 bash names and 10 of hilbert_cluster’s match that way, so replacing the port’s template orphans them. Measured, replacing breaks three green pairs and takes differential to 13 rather than 0.

    IT IS ALSO NOT AN ASSERTION. One parametrised arm is ONE assertion and ELEVEN spellings; counting the spellings made differential report 274 named assertions where the port has 100. The expansion feeds the MISSING calculation and nothing else – it is not counted and it is not listed as extra.

    Refused, each costing a false MISSING at worst: a module constant, because FLOAT_RTOL = 1e-6 renders as 1e-06 and bash carries neither spelling; two or more distinct parametrised columns, because stacked parametrize is a cartesian product and expanding one while holding the other invents names that exist nowhere; and any container that is not literal. Read through _name_argument and nothing else – expanding every f-string instead emits SELECT id, c_int FROM %T as a check name, 96 such in differential alone.

    differential reaches missing: 0 and leaves INCOMPLETE. Eight of the nine graded pairs are now one-for-one; the ninth is blocked on #1040 phase 0b.

  • The grader could not read a port’s own name when a for loop supplied it (#1045 class 2).

    for label, sql in (("allnull column scan",  "SELECT * FROM %T"),
                   ("allnull column count", "SELECT count(a) FROM %T")):
    c, h = p.both(sql)
    expect.row_set(c, h, label)

    _as_names reads a variable as nothing, deliberately: reporting a guessed string is worse than reporting none. So the arm RAN AND PASSED while the grader reported its bash counterpart MISSING. 46 names across 15 sites, 37 of them in differential, whose port is behaviourally complete.

    This is _parametrized_names one level down and gets the same answer: the table is literal, the column is a name, read the column. Read CELL BY CELL rather than row by row, because ast.literal_eval on a whole row raises when any other column holds an f-string – which is differential’s mismatched-collation table, five literal labels beside one interpolated query.

    Only a column actually used as a NAME ARGUMENT is read. A loop variable merely mentioned in the body is not a name, and harvesting it invents properties out of SQL strings; measured, that doubles the names this returns. A table that is not literal – a module constant, a comprehension, a computed label – contributes nothing rather than a guess.

    differential goes from 54 MISSING to 17, and the 17 are a spelling rather than a gap: bash unrolls c_int range through c_text range and c_int eq through c_arr eq as literals where the port parametrises them over RANGES and EQUALITIES, which hold the same 11 and the same 6 columns. That is #1045 class 3, a judgement the tool cannot make, and it stays declared rather than decided by widening _template.

  • pgc_ledger.py merge reported the UNION of majors, which cannot show the defect that has occurred twice (#1048).

    The summary printed a union over rows:

    majors = sorted(set().union(*(v[0] for v in rows.values())) if rows else set())

    A union cannot represent a MINORITY set. Merge rows carrying {18} into a ledger whose rows carry {15,16,17,18,19} and the union does not move, so the line was byte-identical on a correct merge and an incorrect one – the one statistic that could not see the only defect this summary has ever had, and the only one it emitted.

    It happened twice in three hours, to the same person, with a written note in between:

    1041 wrote 12 rows at 18 against 934 uniform ones, caught only by CI’s `suites

    (PG 17)leg; #1042 wrote 8 against 1209, caught by a manualuniq -c. Both times the merge printedmajors … 15, 16, 17, 18, 19`. The operator was not ignoring the output; the output agreed with them.

    It now prints the distribution, names it NOT UNIFORM when there is more than one set, and prints rows N = sum of buckets N beside it:

    majors: NOT UNIFORM -- 2 distinct sets over 5 rows
       3 rows  15;16;17;18;19
       2 rows  18
  rows 5 = sum of buckets printed 5

    The reconciliation counts what was PRINTED, and a SOURCE-TEXT pin holds that, because no behavioural arm can: wherever the display prints every bucket the two sources are equal by construction, so a wording-preserving swap back to sum(dist.values()) passed the whole file. The guarantee rested on a comment until review pointed out that comments rot where arms do not. sum(dist.values()) would equal len(rows) by construction, so it could never catch a bucket lost in the DISPLAY – and truncating the loop drops the MINORITY bucket, the one the summary exists to show, while the line still balances. Found by @OffgridwithJD in review.

    Reporting only. Whether merge should REFUSE a non-uniform result is a live design question and is deliberately not settled by this change.

    test/pytest/TESTS.md also described the ledger as FIVE tab-separated columns and omitted majors from the list, from the day that column landed (#1010) until now.

  • compare_to_bash.py could not read a name a suite passed through its OWN wrapper, and published a bare {} in its place (#1053).

    1051 taught the extractor the recorders lib.sh shares. A suite may also define

    its own, and there are two shapes, only one of which is a gap:

    COMPOSE   check "non-owner refused: ${1%%(*}"     the definition states a
                                                  TEMPLATE naming the property
FORWARD   check_text "$label" "$got" "$want"      the definition states nothing

    A composing wrapper is already read correctly – non-owner refused: {} covers all nine of native_ownership’s call sites, which is why that pair grades one-for-one. A forwarding wrapper’s definition yields the bare template {}, and 17 of those were being published: a “property” with no content, sitting in MISSING where no port can ever assert it, and MATCHING a port name that is entirely one interpolation.

    The grader now derives each suite’s own recorders by the rule that already works for lib.sh – a function forwarding a bare positional into a known recorder’s name slot, transitively, seeded from pgc_record – reads the call sites of the forwarding ones, drops the bare {}, and leaves composers alone. 145 names across 14 suites become readable. sorted_pathkeys alone gains 18, and they are not a random 18: that suite pairs every “plans no Sort” with an “and still answers correctly”, so the grader could see every claim about the PLAN and none about the ANSWER.

    AND IT REFUSES what it cannot read. hilbert_curve.sh defines two helpers taking a newline-separated LIST of names in one argument, so no rule about argument positions can read them; the grader now exits 2 naming both rather than grading the rest. One suite of 264, a true positive, with no pytest twin.

    MEASURED BEFORE BUILDING, and it changed the design: refusing on “the name position is not a bare positional” also refuses every COMPOSING wrapper – 32 suites, including hilbert_cluster, hilbert_locality and native_ownership, three pairs that are COMPLETE today – to fix nothing. Every graded pair is unchanged by what shipped.

  • iceberg_fdw.sh is ported to pytest: the FDW’s partition and metrics pruning (#388, #432).

    74 of the suite’s 76 check names, one for one under compare_to_bash.py, across identity partitions, file metrics, bucket[8], truncate[100], and day/year/month/hour on date, timestamp and timestamptz columns.

    EVERY PRUNING ARM IS PAIRED WITH A CORRECTNESS ARM, which is the design of the bash suite and the reason it matters here: pruning is an optimisation, so a bug in it returns FEWER ROWS rather than a slower plan, and a row count cannot tell the two apart because the right answer to most of these predicates is also small. The oracle is iceberg_scan of the same table under the same predicate – it receives no predicate and opens every file, so it cannot over-prune.

    Three things the port asserts that a count alone would not:

    • a coarse transform must READ the boundary bucket. year(ts) puts every 2020 timestamp in one bucket, so ts > 2021-01-01 has to keep the 2021 file even though the constant is in it. An exact [V,V] rule prunes that file and loses the row.
    • bucket[8](id) and metrics are distinguished rather than conflated. id = 5 keeps one file under bucket pruning and two under metrics alone, because bucket-3’s range [3,7] contains 5. The expected count is 4 for that reason.
    • a date partition the FDW cannot convert must be read in full (#660). The wrong behaviour is to NULL-fill and prune, and its symptom is an empty answer.

    _pruned returns a sentinel rather than 0 when EXPLAIN reports no Files Pruned marker. “Pruned nothing” and “did not say” are the same number and opposite facts, and returning 0 for the second would make every pruning arm pass vacuously the day the FDW stopped reporting.

    The two names not carried are pgc_skip’s refusal names. That is structural: pgc_skip records under the NAME it is given and expect.cannot_run records under the REASON CODE, so a port cannot emit those strings as check names at all (#1040 phase 0b). Declared in INCOMPLETE with that reason rather than worked around by naming a passing premise after a missing dependency.

  • compare_to_bash.py read 6 of differential.sh’s 86 check names and reported the port one-for-one (#1045).

    The extractor’s helper list was every recorder lib.sh names check*. diff_query is a lib.sh wrapper that forwards its $1 into check, so the NAME is in the suite and only the RECORDER is in lib.sh:

    test/differential.sh:116   diff_query "c_uuid range" "SELECT id FROM %T WHERE ..."

    and the grader read none of them. Corpus-wide, five recorders were unread:

    diff_query          name is $1   150 names   11 suites
diff_query_ordered  name is $1     5          3
pgc_skip            name is $2    70         47
pgc_pass            name is $1     9          2
pgc_fail            name is $1    15         11
                                 249 names   70 of 264 suites

    pgc_skip is why the table records a POSITION rather than a membership. Its name is $2 and $1 is a capability, written bare at 68 of its 70 call sites and quoted at the other two, so a pattern keyed to the first quoted argument reads the capability at those two. A wrong name is worse than an absent one: no port can assert test_decoding, so it would be reported MISSING for ever.

    THE DRIFT GUARD #1040 ADDED WAS GREEN THROUGHOUT, AND WAS NOT BROKEN. It derived its population by SPELLING – ^(check(?:_[a-z_]+)?)\(\) – so diff_query was never in the set it ranged over. A guard is worth exactly its population. It now derives the population from what a function DOES, taking the closure from pgc_record, and checks each name’s argument position as well as its membership. Reverting either half reddens it: the spelling population finds 8 recorders where the closure finds 13, and a membership-only guard passes while pgc_skip is read at $1.

    differential moves to INCOMPLETE with missing: 54 in the same change, because the extractor cannot be widened and the pair kept green at once. The port is not missing 54 properties. Two blindnesses were cancelling: the suite records through diff_query, which this change reads, and the port binds most of its own names to a for loop variable, which the grader still cannot read (#1045 class 2). The pair graded missing: 0 on 6 of 86 bash names against 62 of 132 port names – two blind halves cannot disagree. 17 of the 54 are a spelling rather than a gap: bash unrolls c_int range through c_text range and c_int eq through c_arr eq as literals where the port parametrises them over RANGES and EQUALITIES, which hold the same 11 and the same 6 columns.

    Reading a name from the argument that HOLDS it means one pattern per position, and an empty position is not a no-op: (?:) matches everywhere, so the pattern degenerates to “any word, then any quoted string”. On zonemap_boundaries.sh, which has no position-2 helper, it invented six names including $PGC_DB and $(dirname, each of which would be reported as a bash property the port is missing for ever, because no port can assert them. The positions are therefore derived from the table’s own values, so an empty group cannot be constructed, and building one is refused rather than silently returning junk. Found because a reviewer’s stale .pyc left the table mid-mutation: python validates a cached .pyc on source mtime in whole seconds and size, so a same-size edit applied and restored within one second keeps running while the file’s md5 reports clean.

  • skip-loop-arms.py read a one-line function body as everything below it, so a psql wrapper counted as a check recorder (#1042).

    The tool decides which functions record a check by reading each one’s body, and it took that body as everything up to the next brace at column zero:

    test/audit.sh:122   q() { run_pg "$PSQL -c \"$1\""; }

    q’s brace is not at column zero, so its body ran on into the NEXT function’s and swallowed every check call in between. q – a psql wrapper that records nothing – then classified as a RECORDER, and its first argument, SQL text, entered a set of valid check names. The corpus has 199 one-line definitions, so this is the common form and not an edge case. The body now ends at its own closing brace, counted per line.

    IT CHANGED NO VERDICT ON THE TREE AS IT STOOD, which is why it needed an arm and not only a fix. A/B between the two extractors on db74d9e9c: loops 8 / compared 6 / interpolated 1 / armless 1 from both, byte-identical. Two extractors that agree on today’s corpus are indistinguishable from the tool’s own output, so a regression would have been invisible to every existing check. That is a property of today’s corpus and not of the tool.

    THE TOOL NOW SAYS WHEN A BODY NEVER CLOSES. Counting braces is defeated by an unbalanced one inside a quoted string, and the corpus has exactly one – _us_unbound in test/selftest/400-a-check-result-must-be-machine.sh, whose grep -oE '\$\{?...' and tr -d '${' leave the walk unterminated, so its body is 229 lines and runs to end-of-file. That is NOT fixed here: one pathological definition in 909 does not buy a shell lexer, and a lexer is a much larger thing to be wrong about. What is fixed is the silence. _us_unbound is misclassified as a recorder today by the shipped tool AND by this one – the symmetric difference of the two emitter sets is empty – so it predates this change and survives it, and it is reported separately rather than folded in.

    AND HEREDOC BODIES ARE BLANKED BEFORE SCANNING FOR DEFINITIONS, which this file already did for its structure walk and did not do here. The guard for this very defect writes q() { ... } into a fixture as heredoc content, so without it the tool reported the test’s own fixture as a finding in the real tree: unclosed 2, the second being the guard’s own swallower.

    The new selftest part brackets the change in both directions, because no single mutation can redden all three arms:

    revert the body walk         the one-line wrapper arm reddens        1 FAIL
delete the unclosed report   the count premise and the NAMED arm     2 FAIL
report unconditionally       "a clean file is not named" reddens     1 FAIL
                             and the other two stay green

    Each mutation was asserted to still PARSE before its red was believed, and each restore was md5-asserted.

  • test/hilbert_cluster.sh has a pytest twin: the Hilbert clustering SQL surface, graded one-for-one (#432).

    45 collected tests, 184 checks, across the bash suite’s eight arms: the surface and its refusals by SQLSTATE, “it only reorders”, the recorded kind, the self-gate in every direction, the single-column identity, the vacuum_sorted ruling, the daemon, and the enumerations. compare_to_bash.py reports 0 MISSING both as the tool ships today (124 bash checks) and under the widened extractor in #1044 (133). The difference is that suite’s nine check_unrunnable sites, all of which are twinned, so this is the first exercise of the widening on a suite that uses those helpers correctly rather than on the one where they were broken.

    TWO PLACES THE PORT ASSERTS SOMETHING THE ORIGINAL GETS FOR FREE, and they are one class: wherever a port replaces a structural guarantee with a procedural one, it owes an arm the original does not need.

    The bash suite gives the daemon’s naptime and thresholds to the server through PGC_EXTRA_CONF, so they sit in postgresql.conf before the postmaster starts and the suite cannot run without them. pgc_cluster.py has no such hook, so the port sets them with ALTER SYSTEM and a reload, which is available because all three are PGC_SIGHUP and which can silently fail to take effect. It would fail silently in the worst way: at the default naptime the daemon still acts, the poll still sees the tail fold, and every arm still passes while the values the fixture claims to have set were never in force. So the three are read back from the server. The same for max_parallel_workers_per_gather = 0, which the fixture set and nothing read until the parity tool reported the bash suite’s own premise as missing.

    A RELOAD IS NOT A READ, and this one is inherited by any later port that changes postmaster-level state. ALTER SYSTEM SET pgcolumnar.autovacuum = on, then pg_reload_conf(), then SHOW on the same connection returned off: a reload signals the postmaster and a backend already open absorbs it at its next command boundary, and this module runs every statement through one connection by design. The bash suite never meets it because every q is a fresh psql, so a new session here is the port of what the original gets for free rather than a workaround.

    Every conditional check_unrunnable in the bash suite is its own test here. expect.cannot_run() records under the reason code, so two refusals in one test would collapse onto a single UNMET_PRECONDITION record and neither could be told from the other.

    THE DAEMON ARM RECORDS HOW LONG IT WAITED, not only that it succeeded. A fixture one poll from its window and one fourteen from it produce identical greens, so the count is the only thing that distinguishes them and drift toward the bound is otherwise invisible. Measured on all five assert builds, each run on its own:

    PG15  PG16  PG17  PG18  PG19
   1     1     1     1     1     poll(s) of 15, at a 2s naptime

    READ THAT FOR WHAT IT DOES NOT SAY. One poll on every major means the loop NEVER WAITED: the condition was true on the first check each time, so nothing in those runs exercised the waiting at all, and a loop that always succeeds on poll one is indistinguishable from no loop. The distribution is a SINGLE POINT, taken on an idle container, and the case the bound will actually meet is a shared and loaded CI runner – this repository has that divergence recorded elsewhere as 0 in 400 idle runs against 6 in 400 under load. So: one poll on each of five majors on an idle container; the bound of 5 is four above the only value ever observed; no loaded measurement exists. A reader who meets a red at six is the first person to see the loop do its job, rather than someone looking at a regression.

    The count is printed by the fixture, and pytest captures fixture stdout on a PASSING run, so -s is what surfaces it; on a failing run the arm’s own message carries the number, which is where it is needed.

    This is the one part of the change with a single source of evidence: @jdatcmd reviewed the rest but has no PostgreSQL on their host, said so rather than offering a reading of the loop, and the five-major measurement above is the only one that exists.

    What the port does NOT buy is stated in the file: over one column the Hilbert index and the Morton index are both the identity, so the single-column arm is green on a relabelled Z-order implementation by construction. Four arms refuse one, and no others.

  • An unrunnable record names the check it stands in for, so a refused check keeps one ledger key instead of two (#1040).

    check_unrunnable NAME REASON DETAIL gives one check the honesty pgc_skip gives a whole suite: the check did not run, and the reader is told which. That only works if the record carries the name the check uses when it DOES run. Two of the four sites in one loop in hilbert_locality.sh carried a shortened name:

    :574  check_unrunnable "box $box: groups read, Z-order"
:597  check_num        "box $box: groups read over $PLACEMENTS placements, Z-order"

    while the other two matched their runnable twins exactly. So the property had two ledger keys and which one appeared depended on whether that box’s two partitions came out different that day – the key was a function of the data. hilbert_locality is not a ledger-covered suite, so this was a wrong key waiting to be seeded rather than a wrong number in check_ledger.tsv.

    The convention is already near-universal, and that is what made the lapse invisible: 23 of the 25 check_unrunnable call sites in test/*.sh carry the runnable name (hilbert_cluster 9 of 9, projection_rewrite 11 of 11), so a shorter name in a new refusal branch reads as ordinary. Both halves of a two-branch site are rarely read together.

    A new selftest part asserts the property over the whole corpus, driving .github/scripts/unrunnable-arm-names.py. The tool DERIVES three things a list of it got wrong first: which functions record, from the pgc_record call in their body; which argument is the name, because pgc_skip records "$2" and reading argument one takes the capability where the check is called arrow support is present; and what counts as a refusal, from the verdict rather than the helper’s spelling. check_skip is deliberately not swept – a skipped arm has no runnable counterpart by construction, so 21 of its 23 call sites have no twin and always will.

    The part carries a positive control because the guard’s steady state is zero and a broken sweep reports zero too: it drives the real tool over a fixture whose refusal branch names something the file never records, and over a control where the names agree.

    This also closes the only pair compare_to_bash.py fails once its extractor is widened to all eight lib.sh helpers (#1040): with the two names fixed, all seven ported suites grade one-for-one under the widened extractor, with no change to the port and none to pgc_vacuity.py.

    The sweep’s exit code is the verdict. It was a flat zero in the first version, so the tool printed two MISMATCH lines and reported success. The part gates on the parsed output and was never fooled, which is exactly why the exit code needed its own arms rather than an observation: the next caller is the one that trusts $?, and a gate that cannot fail is a trap whether or not today’s only caller steps in it. Three arms, because pinning the non-zero side alone passes on a tool that always exits 1, and the clean side alone passes on a corpus with nothing to find – so the clean run uses a second fixture directory that HAS a refusal site, with a premise asserting it. Reported by @jdatcmd.

  • The census re-derivation printed in check_ledger_budget.txt reads the wrong field and returns zero (#1040).

    awk -F'\t' '$4=="never"' test/check_ledger.tsv | wc -l     -> 0
awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l     -> 1189 on db74d9e

    1010 inserted the majors a row claims as field 4, moving the last-red to field 5, and

    the recipe stayed on field 4. The entry for #1010 in this file states that check_ledger_budget.txt carries the corrected form; it did not. The gate is unaffected – it computes the census itself and never runs this command – so the harm is a reviewer re-deriving the number by the printed method, getting 0, and correcting a budget that was right. Fixed here rather than filed because this change moves that very number, and a wrong recipe beside a number nobody can check is worse than no recipe.

  • allow_empty documented a rule the code did not enforce (#1031).

    Expect.rows documents the argument as taking “a REASON, not a flag”. The sentence even gives the rationale: the hatch should cost more to type than the honest assertion. One line below it sits a truthiness test:

    if _empty(got) and _empty(want) and not allow_empty:

    So allow_empty=True satisfied it and carried nothing, and the hatch cost LESS than the assertion rather than more. Measured before the refusal: allow_empty=True and allow_empty=1 both passed, 3 passed. row_set forwards the argument, so it inherited the hole and now has its own arm asserting it does not route around the refusal.

    THE CHECK FIRES WHENEVER THE ARGUMENT IS GIVEN, not only when both sides turn out to be empty. Otherwise a flag form in a test whose sides happen to be non-empty passes today and refuses on the day the data changes. That is the worst moment to learn it.

    TWO LIVE SITES USED THE FLAG FORM AND BOTH WERE SUBSTANTIVELY CORRECT. Each had its population premise on the line above. test_check_records.py even stated the argument in a comment: “an empty offender list is the answer to both, and only one of them is good news.”

    So this is not a defect hiding behind the hatch. The cost fell on the next reader. The hatch exists so every empty-on-both-sides comparison carries its justification where an audit of allow_empty= can read it, and half of them carried none. Both now do.

    Pinned in test_guards_pinned.py, which exists so a refusal is asserted by its message rather than by “something failed”. That file was built after a census found 12 of 17 guards deletable with the suite still green.

  • The gate refuses two checks that share one ledger key, instead of printing a note about them (#982).

    AND THE SAME CLASS ONE LEVEL DOWN, found while building the arm for the first. read_ledger did rows[(f[0], f[1], f[2])] = [...], so a duplicated key in the TRACKED file collapsed silently and the last line won. Measured on a two-line fixture, both orders:

    never first, then 2026-09-01   survivor last_red='2026-09-01'
2026-09-01 first, then never   survivor last_red='never'      the red is GONE

    Line order decided whether a recorded red observation survived. A merge that keeps both sides of a changed row turns ever red back into never. That is what #918 and #925 exist to prevent, arriving from the opposite direction.

    NOTHING ELSE COULD CATCH IT, and bounding the census cannot. check_ledger_budget.txt says checks_never_observed_red is a CENSUS and must not become a ceiling, because every new check enters as never and bounding it deadlocks. The gate compares the budget’s number with the ledger’s, and both come from the same dict, so they agree either way. Measured: with the budget regenerated alongside, an erased red passes the gate at rc=0.

    It is refused as an INTEGRITY FAILURE (rc=2) rather than a gate verdict, beside the other inputs that do not parse. A ledger that cannot be trusted is not a gate result.

    So the two halves are the same shape at two levels. A SET hid two checks in one run; a DICT hid two rows in one file. The question that found both is what the input canonicalises before the guard sees it.

    AND A THIRD, IN THE ARM THAT POLICES THE OTHER TRACKED KEY-VALUE FILE. test_harness_deps.py read expected_tests.txt with nums[f[0]] = int(f[1]), so a duplicated key collapsed and the last line won, exactly as read_ledger did. Measured on one fixture read both ways:

    the line form   names guard_tests as duplicated
the dict form   sees two keys and keeps 280, the LAST line

    THIS IS NOT HYPOTHETICAL. Three PRs were open at once, each moving guard_tests, and resolving all three keep-both produced three of those lines. ci.yml reads the value with awk '$1=="guard_tests"{print $2}', which prints one line per match. So WANT becomes multi-line, test -n "$WANT" still passes, and the flag refuses it at exit 4:

    pytest: error: argument --pgc-expect-tests: invalid int value: '284\n280\n283'

    It FAILS CLOSED, so this is legibility rather than a hole. What exit 4 does not say is that a line is duplicated. Two arms now say it. The removal proof on the real file reddens the new arm and leaves the pre-existing one green, which is the point.

    Keep-both is right for a changelog and wrong for a key-value file. Nothing in the tree said so.

    cluster_tests 205 -> 207, not guard_tests: test_harness_deps.py DEFINES NO_CLUSTER and is not in it. I got that wrong first and the mechanism caught it, which is the argument for the mechanism.

    A ledger row is keyed on (suite, part, name), so two checks with the same name in one part share a row. Nothing is mis-recorded while both pass. The hazard is exact: when one goes red the row records ever red, and its namesake inherits a red observation nothing attacked. checks_never_observed_red then falls by one for a check nobody attacked, and that census is what #918 and #925 exist to make trustworthy.

    merge has detected this since #982 was filed, and returns 0. That is how three of them sat in one part of selftest/400 for a day. The instance was fixed by c3b13aed; this is the mechanism, which that issue called the more valuable half.

    THE GATE COULD NOT SEE IT AT ALL, and the reason is worth recording. cmd_gate builds its records as sorted({(s, p, n, m) for ...}). A set collapses the duplicate before any arm can count it. So the same canonicalisation that makes the rest of the gate correct made this one class unreachable. The count now comes from the raw records through _by_run.

    PER LOG, because one check observed in two logs is two RUNS of it. That is the normal case and the way the ledger accumulates evidence at all. Only a repeat inside one log is a collision. An arm pins the distinction, because written over the logs together the refusal would reject every multi-day merge.

    EVERY SUITE, DELIBERATELY UNLIKE THE NEW-CHECK REFUSAL. That one is restricted to suites the ledger covers because it cannot know which of an uncovered suite’s checks are new. This one needs no history: two records, one key, one log is decidable from the log alone. The ledger covers four suites of 253. Copying the restriction would close the class in four places only. The next collision would then sit in one of the other 249 until that suite was seeded.

    MEASURED BEFORE WIDENING IT, because a gate that reddens 250 unmeasured suites is a gate somebody turns off. A full PG 18 matrix run with the refusal armed for every suite:

    suites that ran                       247   (6 skipped, 0 incomplete)
matrix verdict                        ALL VERSIONS PASSED, RC=0
shared ledger keys found                0

    Check names are static, so one major’s matrix measures this class completely rather than sampling it. A real harness_selftest log says the same end to end. Run against the COMMITTED ledger and budget, the gate returns 0 with no shared-key line, over 934 records and 934 distinct keys.

  • projection_privilege.sh has a pytest twin, and both halves now attribute a refusal by SQLSTATE instead of by error text (#432, #562, #563).

    The bash suite decided four things by matching the message, and two of them were load-bearing rather than decorative. Its own comment says why: both ACL layers raise 42501, so a bare “refused” stays true if the SQL REVOKE is deleted and the C check catches it instead. Measured there: with the REVOKE removed the suite still passed 14 of 14.

    The fixture is what separates them, not the wording. Called with a projection name that does not exist, on a table the role may read, a caller stopped by the SQL grant never runs the body and gets 42501; one that gets past the grant reaches the lookup and gets 42704. The refusal is attributed by what the code REACHED. RLS is a third code again, 0A000 from ERRCODE_FEATURE_NOT_SUPPORTED, which is a different SQLSTATE class from either ACL refusal.

    Two orderings that src/columnar_projection.c and src/columnar_vacuum.c assert had no test in either harness: the base ACL is checked before the projection is looked up, so a caller with no SELECT cannot learn whether a projection exists on a table it may not read; and the ACL is checked before RLS, so a caller with no privilege is not told the table has row-level security enabled. The second is a correction the source records being made in review.

    Five mutations, each asserted to apply at both call sites. Moving the ACL check below the projection lookup but above its raise reddens nothing, correctly – 42501 still wins. Moving it below the raise reddens both ordering arms, and the shell half prints the disclosure: got [42704] want [42501].

  • compare_to_bash.py reads the assertion’s NAME (#432, #897).

    The parity tool decides whether a port is one-for-one with its bash suite, which is

    432’s definition of done, and it was reading the wrong argument. The python side was

    matched with expect\.\w+\([^)]*?"([^"]+)", whose lazy [^)]*? stops at the FIRST quoted argument. For expect.num(got, 1, NAME) that is the name, so the tool looked correct. For expect.sqlstate(err, "42501", NAME) it is 42501.

    Every SQLSTATE assertion was therefore read as the literal 42501, reported as an “extra” name the bash suite does not have, while the real property was reported MISSING. #432’s ports are exactly the ones replacing a grep on a message with a SQLSTATE assertion, so the tool went blind in proportion to the work being done well.

    It parses with ast now and takes the last string argument, resolving f-strings to templates, both arms of a conditional, and the name column of a @pytest.mark.parametrize. Bash interpolations reduce to the same template, including $1, which is the commonest one in a check name and which the first version of the reducer missed because its pattern required a letter after the dollar.

    Measured over every pair in the tree: 61 bash properties reported missing, now 0. 34 were never missing. The rest were real and are closed here: stats_privilege had invented a name for a property the bash suite already named, and zonemap_boundaries was missing its backend alive premise outright. Neither was visible while the tool was reporting the wrong string.

  • A UNIQUE-constraint check passed on any psql failure, and a recursive sweep passed on a tree it never read (#1033).

    native_recluster.sh decided “unique still enforced” from psql’s exit code:

    psql ... -c "INSERT INTO n VALUES (1, 0, 0, 'x');" >/dev/null 2>&1 && echo no || echo yes

    || echo yes cannot tell a unique violation from any other failure, and >/dev/null 2>&1 discards the message so nothing else can either. Measured on the identical expression: a missing table, a wrong port, psql absent from PATH and a syntax error all produced yes.

    THE CONTRAST, RUN END TO END on PG 18 rather than argued. The same mutation – point the probe at a table that does not exist – against both versions of the arm:

    main's arm    PASS  "unique still enforced"       suite PASSED, rc=0, 12 passed
this arm      FAIL  got [42P01] want [23505]      suite FAILED, rc=1

    And the arm is observable in the direction that matters. Replace the unique index with a plain one and the duplicate INSERT succeeds. The arm goes red there too, so it tests the constraint rather than merely the SQLSTATE plumbing.

    The arm now reads the SQLSTATE. 23505 is ERRCODE_UNIQUE_VIOLATION and comes from the index; 42P01, 42601 and a connection failure do not. arrow_import.sh and audit.sh already read SQLSTATE this way, so this is the convention rather than a new one.

    SECOND, A SWEEP WITH NO POPULATION. local_open_race_free.sh asserted that a removed helper leaves no trace:

    "$(grep -rc 'PgColumnarRejectNonRegularFile' "$SRC" | awk -F: '{s+=$2} END{print s+0}')" "0"

    grep -rc prints one file:count line per file and prints NOTHING for a path it cannot open, and END{print s+0} then manufactures the 0 the check wants. The three arms above it read $OBJ, not $SRC, so nothing established that $SRC was a source tree. Measured:

    SRC=src            files=56  premise=yes  arm=0   both agree
SRC=/no/such/tree  files=0   premise=no   arm=0   the premise catches it, the arm alone passes

    A premise now counts the lines the recursive grep emitted, which is exactly what the arm’s awk sums over.

    NEITHER SUITE NEEDS A LEDGER ROW, which is why these two and not a third. check_ledger.tsv holds zero rows for native_recluster and local_open_race_free, so the gate cannot refuse a new check there.

    A third site has the same shape and is left alone: selftest/400’s tree-wide piped-loop sweep. It lives in harness_selftest, which the ledger covers with 934 rows. Its detector IS premised, since :549 proves it fires on a planted offence, and only its glob population is not. So it belongs to a change that carries the five-major ledger merge.

    Verified on PG 18 in the container: local_open_race_free.sh PASSED, 11 checks; native_recluster.sh PASSED, 12 checks; both mutations red; main’s arm green under the same mutation.

  • The vacuity inventory named one entry twice, and no arm could fail on it (#432).

    VACUITY_MODES.md section 3.4 carried the same six-id bullet twice, verbatim, on two lines each. It is deleted, and three arms in test_docs_cover_the_corpus.py now refuse a duplicated entry.

    WHY NOTHING CAUGHT IT, measured rather than guessed. Every count this document states is checked. Every one of those checks is blind to this by construction: _named_modes_in builds set(MODE_ID.findall(chunk)) per section, so each total is over distinct ids. Measured with the second copy present and then deleted:

    with the duplicate      (28, 44, 72)
without the duplicate   (28, 44, 72)

    So test_the_mode_inventory_states_its_own_totals_correctly, the prose-totals arm and the sum arm were all green with a duplicated entry in the file. Deduping ids is right. A total must not move because a line was pasted twice. The cost falls on the reader instead: one group of open modes reads as two. So the new arm is about entries, and the counting rule is unchanged.

    MY OWN FIRST SWEEP MISSED IT, which decided the unit the rule uses. An adjacent duplicate-LINE sweep over every document in the directory reported nothing. The duplicate is a two-line bullet, so line 1 of the first copy and line 1 of the second are not adjacent. Re-keyed on the bullet ENTRY, the same sweep found it, and found exactly one tree-wide. Four sweeps here have now failed by keying on how something is written rather than on what it contains. So the unit is named in the code, and both fixture arms use a two-line bullet rather than a one-line one.

    The rule carries its false-positive budget as an arm. The inventory legitimately repeats short bullets, so entries under a 40-character floor are not compared. The budget is measured at the boundary: the same bullet passes below the floor and is refused above it.

  • The section that checks for stale documents carried a stale count (#432).

    TESTS.md section 6 said “the five fixture arms”. Five was correct at 3d6e1216 (2026-09-09) and counted the arms taking tmp_path; there are eight of those now and thirteen fixture arms in total. Nothing read the number, so it went stale in the document whose whole subject is documents going stale.

    It is removed rather than corrected. The paragraph above it already decided that for the same reason: test_the_document_states_no_totals_for_a_merge_to_get_wrong exists to keep a totals line OUT. A count in prose that no arm reads is a claim waiting to go wrong. The arms themselves are listed in the table above, where a reader can count them.

  • TESTS.md’s contents list was out of numeric order and no arm could see it (#1026).

    After #1023 merged, the document read:

    TOC       ... 29, 31, 30      and 29, 31, 30, 32 once the next section arrived
sections  ... 29, 30, 31      contiguous and correct

    1023’s contents entry for section 30 landed after 31. Resolving this PR’s conflict in

    that same region meant choosing an order, so the fix lands here: TOC and sections are both 1..32 with no gap and no inversion.

    WHY NOTHING CAUGHT IT. test_docs_cover_the_corpus.py already sweeps every contents-list link and asserts it reaches a heading. Both orders resolve, so that arm is green either way. Measured by restoring the broken order under the new arm:

    the new arm      FAIL  got '[(29, 31), (31, 30), (30, 32)]' want 'none'
the link arms    1 passed

    So the link sweep is not a weaker version of this rule; it answers a different question, and a shuffled contents list was outside both.

    Two arms. The first reads TESTS.md. It requires the contents numbers and the section numbers each to count 1..N with no gap, and one contents entry per section.

    The second is the removal proof on a fixture. It uses the 29, 31, 30 shape that actually shipped rather than a single swap. It also names an omitted entry apart from an inversion: a missing entry gives [(1, 3)] and a shuffle gives [(1, 3), (3, 2)].

    THE GAP RULE CATCHES THE COLLISION TOO. Three open PRs each claimed a section number another had taken, which is the cause rather than a coincidence. A number used twice leaves a gap in the section sequence, so 1..N with no gap reddens on a duplicate and on an omission with one rule.

    guard_tests 282 -> 284, re-derived by collection.

  • native_ownership.sh has a pytest twin, and it asserts the SQLSTATE (#432).

    Nine maintenance and DDL functions, each refused to a non-owner with 42501 rather than with a grep for must be owner. CLAUDE.md already states the rule: 42501 comes only from aclcheck_error, and the refusal is aclcheck_error(ACLCHECK_NOT_OWNER, ...) at columnar_vacuum.c:189 and :205. A text grep passes whatever code the server attached.

    It also stops conflating refusal with login: the bash suite runs each call as a role that must be able to connect, so a role that could not log in fails the arm for a reason unrelated to ownership. SET ROLE changes the effective user without authenticating.

    A third arm states the ordering the bash comment asserts in prose: the check fires before the work, so a non-owner is refused for a projection that does not exist.

    THE PREMISE ARM CORRECTED ITS OWN DOCSTRING. Every refusal carries premise: alice reaches the table, because each test runs in a private schema. Measured by removing the grant, alice gets 42P01 relation does not exist rather than the 42501 I had claimed: an unqualified name resolves through search_path and an unusable schema is skipped, so the arms fail rather than falsely pass. The false-pass case needs a QUALIFIED reference, which raises 42501 for the schema.

    AND THE PARITY TOOL CANNOT GRADE THIS PAIR. Both sides build names at runtime, so compare_to_bash.py reports PORT IS INCOMPLETE for a complete port. Measured: 81 of 253 suites carry at least one interpolated check name. That bounds how much of

    432’s parity the tool can certify, and it is a false red rather than a false green.

  • stats_privilege.sh has a pytest twin, and it asserts the SQLSTATE (#432).

    The bash suite decides the refusal with a grep on the message. CLAUDE.md names the rule: 42501 comes only from aclcheck_error, while a grep for “permission denied” is also satisfied by other refusals.

    WHICH ARMS ARE LOOSE, MEASURED RATHER THAN ASSERTED, because two rounds of review narrowed this twice. The defect arm at :82 uses permission denied for table, which the schema message does NOT match – so that arm is not confusable, and an earlier version of this entry claiming otherwise was wrong. The two BARE greps are at :68 and :70, and both are premises:

    :68  premise: the no-privilege role cannot read it by ordinary SQL
:70  premise: the catalog tables are NOT readable by these roles

    So the risk is a premise satisfied for the wrong reason, which weakens what the suite rests on, rather than a defect slipping through. The port asserts 42501 on the premises and on the refusal, and that the refusal names the table.

    Real logins rather than SET ROLE, because session-opening is a property this suite tests and SET ROLE would assert it away.

    SCOPE, MEASURED AND THEN CORRECTED. Across the corpus, 50 suites already assert a refusal by SQLSTATE and 3 assert both. FIVE assert by text with no SQLSTATE anywhere: native_ownership, stats_privilege, projection_privilege, rls_direct_storage and import_export_privilege. Two are now ported; the class closes at five, not at the whole corpus.

    My first sweep said four. It missed import_export_privilege.sh:76 because the flag class grep -[qic]*i? does not cover the E in grep -qiE, so the line never matched and a fifth member stayed invisible while the output looked complete. Found by @OffgridwithJD. Four sweeps in one day across two sessions have now failed this way, each keyed on how something was NAMED or SPELLED rather than on content: key on content, and when a sweep returns a tidy number, grep for one known-present member and check the sweep found it.

    A HELPER TURNED A DRIVER DETAIL INTO A PRODUCT CLAIM. psycopg3 returns the FIRST statement’s result for a multi-statement execute, so SET search_path ...; SELECT hands back the SET’s empty result. The first version collapsed that into a 0 and the arm reported “the OWNER cannot read its stats”. Every call site now asserts the error is None rather than folding it into a value.

  • The docs gate checks that a markdown table is still a table (#1026).

    docs_style.sh enforced seven rules over every user-facing page. Sentence length, the idiom list, em and en dashes, prose double-hyphens, conflict markers, the nav entry, and every VERSION citation. ALL SEVEN ARE ABOUT PROSE. So a table that had stopped being a table passed the gate whose purpose is keeping those pages readable.

    The measured case. A note and a second table were spliced into the middle of configuration.md’s set_options argument table. That left six of the nine arguments as a headerless block, and docs_style.sh passed with 14 checks.

    It was the second splice in one day. The first gave the GUC table no blank lines, which made an awk RS='' guard read two GUC rows as one record and pass on main. Both are the same fact: a markdown table is a contiguous run of | lines, and a blank line is structural.

    Fences are tracked BY LINE rather than stripped with a regex. The regex form already in that file is fine for counting, but it loses line numbers and it breaks on an unclosed fence. State-tracking under-reports there instead, which is the safe direction.

    MEASURED BEFORE LANDING, which is what a static guard here owes. 0 across the gate’s own scope, and 5 elsewhere in the tree. All five are REAL rather than false positives: 3 in test/pytest/TESTS.md and 2 in a design document, none of which the gate covers.

  • The mutation ledger records WHICH MAJORS each check exists on (#1010).

    suite<TAB>part<TAB>name<TAB>majors<TAB>last-red<TAB>mutations

rows            1179 -> 1197   (carried 1177, new 20, dropped 2)
checks_never_observed_red   1171 -> 1189
ever red           8 ->    8   (all carried, with their dates and one mutation)
major sets      1197 x "15;16;17;18;19"
covered         differential, harness_selftest, native_join_runtime_filter,
                native_join_vector_agg
suites_not_covered          249 (unchanged)

    Every row claims all five majors, because all four covered suites are major-invariant: 934, 204, 46 and 13 records, identical on 15/16/17/18/19 in a five-major matrix. So the census moves only by the twenty arms this change adds, and the diff is one field per line.

    The 20 new rows are part 410’s arms about the majors field; the 2 dropped are the rows whose checks this change RENAMED (every committed row has five fields became six). The migration refuses to drop a row carrying history, for the reason orphan-scan refuses it – the catalogue of what has been seen red is the thing no run can recreate – so a drop is only available for a row with nothing to lose, and the name has to be typed.

    A check’s existence depends on the major, so a ledger that cannot say where a check exists cannot tell a deleted check from one that never ran here. test/analyze_differential.sh:61 emits ONE record on PG15-17 and a suite’s worth on PG18+; test/fk_referencing.sh:287 emits DIFFERENT CHECK NAMES in its two branches.

    THE MAJOR IS A FIELD, NOT PART OF THE KEY, and that is the design rather than an implementation detail. Measured on a full matrix at 4d7c75ae, 252 suites on PG15 and PG18: 6367 of 6472 checks are identical on both majors and 105 exist on exactly one. A (major, check) key would hold 6472 x 5 = 32,360 rows to express those 105 – about 247 duplicate rows for every row that differs, each a second copy of one observation. Keeping the key at (suite, part, name) also keeps checks_never_observed_red counting CHECKS: under a pair key it would count pairs, and “5800 checks” in a tree holding 1150 of them is a number that lies by its own name.

    The field is a sorted ;-separated SET – 15;18 – and it ACCUMULATES. A plain assignment was measured doing the wrong thing to last-red on #918: merging a PG15 log after a PG18 log must not make the check stop existing on 18, because the order somebody merges logs in is not a fact about the code.

    NO WILDCARD. “Every major observed” would change meaning the day a major joins the matrix, inheriting a claim nothing measured.

    unknown is a token in the set like any other, and the common one: harness_selftest never references PGC_MAJOR, so every record it emits carries it.

    WHAT THIS FIXES is orphan-scan, not gate. The gate refuses a check in the LOG the ledger has not seen, and a PG18-only check does not appear in a PG15 log, so it stayed correct by never being asked. orphan-scan asks the opposite question, and a PG18-only row is exactly what a deleted check looks like on PG15. It was saved only by the SKIP rule – analyze_differential emits a check_skip so its part was unprunable, while fk_referencing emits check and has no SKIP at all, so once that suite is seeded a PG15 run would have called its two PG17+ checks deleted. The fourth category already said the true thing, so the scope gains an intersection and nothing else: no new category and no grandfather rule.

    The gate also cannot refuse a check on a major it holds no rows for, for the same reason it cannot in a suite it has never seen – otherwise adding PG20 would redden every check at once, which is a gate somebody turns off. It tightens the moment one run on that major is merged, and it says out loud when it is not enforcing.

  • The docs name 1024 as the floor for stripe_row_limit (#1017).

    A vector is a fixed 1024 values, so a row group smaller than one never fills it and the chunk-shared FSST symbol table is not built. Measured on 200,000 rows of a text column, compression = none, against 12,800,000 raw bytes, two identical passes:

    stripe_row_limit 1000   0 FSST tables    13,625,000   106.4% of raw
stripe_row_limit 1200   166 of 167        6,998,031    54.7% of raw
stripe_row_limit 2000   100 of 100        6,990,641    54.6% of raw

    At 1000 the column costs more than storing the bytes uncompressed. THE ACCEPTED MINIMUM IS 1000, enforced in set_options, so the most aggressive legal setting is the one that pays this – and docs/administration.md tells a reader to LOWER this setting for point-lookup-heavy tables, which is the path in. The warning now sits in that block rather than in a reference table.

    Documentation only. The minimum is unchanged: whether to raise it, make FSST work below a vector, or warn at set_options is still open on #1017.

    The chunk-group limit does not affect this, and that is now measured rather than assumed: chunk_group_row_limit at its floor of 100 stores 7,086,080 bytes, the same byte count as 1024 and 10000. Measured by @OffgridwithJD.

    THE GUARD WAS BORN GREEN TWICE BEFORE IT WORKED, and the measurement is why it does now. A blank-line block reader passed on main, because configuration.md’s GUC table has no blank lines and stripe_row_limit’s row shares a block with chunk_group_row_limit’s “fixed 1024-value vectors”. A three-line proximity window passed for the same reason. One line naming both is 0 on all three pages on main, and it is also a claim about the prose: the floor has to be stated in a sentence.

    The two harnesses disagreed while that was being found – the awk arm used paragraph mode and passed on main for two pages, the python twin split on blank lines and did not – which is the argument for keeping both halves, paid back the day it was written.

    AND ONE LINE SAYS NOTHING ABOUT WHERE. Moving the line out of the advice block to the end of administration.md, 402 lines away, left the arm passing while its name claimed the floor was stated beside the advice. Reported by @OffgridwithJD. The arm now asserts the SECTION: the floor and the lowering advice must sit under one ## heading. A heading is a declared boundary, which is what the paragraph reader lacked.

  • A check record names the PostgreSQL major it was observed under (#1010).

    RESULT<TAB>suite<TAB>part<TAB>name<TAB>verdict<TAB>major<TAB>reason

    A CHECK’S EXISTENCE DEPENDS ON THE MAJOR, so a record that does not name one identifies a check only partly. test/analyze_differential.sh:61 emits ONE record on PG15-17 and N on PG18+; test/fk_referencing.sh:287 emits DIFFERENT CHECK NAMES in its two branches, so the two majors' key sets are disjoint. Measured on a full matrix at 4d7c75ae: analyze_differential, analyze_function, native_repack and pg19_vacuum_options each emit exactly 1 record on PG15 against a suite’s worth on PG18.

    A grep for PGC_MAJOR does not find all of them, which is why this is a field rather than a convention: test/native_repack.sh:48, test/pg19_vacuum_options.sh:34 and test/native_dml.sh:61 gate on server_version_num and never mention it.

    Before this the tool could only learn the major from whoever invoked it, which is the --date not-a-date failure one field over: a PG15 log merged as PG18 is misattributed and nothing in the log can contradict it. The major is VALIDATED rather than stored – eighteen, 18.2, pg18 and an empty field are each refused – because a major decides which checks can exist.

    unknown is the one non-numeric value, and it is lib.sh’s own word for a field the harness did not set (it already uses it for an unset suite and part). It is a real case rather than a courtesy: PGC_MAJOR is set in pgc_setup, and 14 suites need no cluster so never call it. Measured on a full pg18 matrix, 544 of 6753 records carry it – audit, concurrency, decode_interrupts, hilbert_curve, objstore_stash_recovery, phase2-phase6, smoke, unique_conc, update_conc, wal_envelope.

    It is ORDER-DEPENDENT in a suite that sources parts into one shell: a record emitted before the first pgc_setup says unknown and one after it names the major. harness_selftest is that shape, 10 of its 46 parts call pgc_setup, and on pg18 all 916 of its records named the major – so the first setup precedes the first record today, and a part added ahead of it would change that.

    The ledger itself is unchanged in SHAPE: it still keys on (suite, part, name) and discards the major. Keying on it is #1010’s second step, and it needs a migration this change does not.

    checks_never_observed_red   1155 -> 1164
suites_not_covered          249 (unchanged; no suite was seeded)

    The nine rows are part 400’s new arms: four emitter arms driving pgc_record with PGC_MAJOR set to two different values and unset, one that the reason still follows the major, and five reconciler arms – four invalid majors and the unknown control that keeps them from passing because the reconciler started refusing everything. RE-DERIVED from the committed file by the derivation the budget file states, not computed from 1155:

    awk -F'\t' '$4=="never"' test/check_ledger.tsv | wc -l

  • Ungrouped vectorized aggregate over a unique-key inner Hash Join (#752).

    The fold used to require a single base relation, so a star-schema join dropped it. A unique dimension is a filter of the fact table, so the fold can keep running. Duplicate-key dimensions, LEFT joins, extra Join Filters, and grouped aggregation over a join still use core Agg. A Join Filter besides the hash clause is not a membership test, so the fold refuses it. pgcolumnar.enable_ungrouped_vector_agg stays off by default.

    checks_never_observed_red   1155 -> 1162
covered                     native_join_vector_agg, 13 checks, last-red 2026-09-12
awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l

    Field 5 because the entry above inserted the majors as field 4. At the moment this change landed on its own it was field 4; both ship in the same release, so the form here is the one that works on the shipped tree.

  • A pytest cluster that will not start now says why (#1016).

    pg_ctl prints “Examine the log output.” and nothing examined it, so a cluster that failed to start produced fifty identical errors naming the COMMAND and not one naming the cause – measured on a GitHub runner, fifty pg_ctl: could not start server and the reason sitting in a file nobody read. lib.sh has had pgc_start_log_report since #537 for exactly this; the pytest harness had no equivalent, and the two are meant to be parallel in functionality. It reports the FATAL lines with their line numbers, then a tail, and says so explicitly when it found neither – silence reads as “nothing to say”, which was #537’s whole complaint. Written on this side rather than called across the boundary, because the harnesses stay independent.

  • The pytest tests that need a cluster now run in CI, and both pytest jobs assert how many tests they collected (#1016).

    166 collected tests in 9 files, counted on this tree: 93 when the gap was filed, plus

    1012’s test_join_vector_agg.py and #1020’s test_differential.py, which landed into the

    ungated half while this change was in review. That is the argument for the change rather than a detail about it – the half was growing faster than it was being gated.

    ci.yml had one pytest job, pytest-guards, and it installs psycopg deliberately NOT – that absence is what proves those files need no database. nightly.yml mentions pytest zero times. run_all_versions.sh mentions it zero times and must, because the two harnesses stay independent and the shell runner invoking pytest is the cross-harness call the project forbids. So 8 files and 93 test functions, 26% of the corpus, ran nowhere: green when somebody ran them by hand, silent when they stopped.

    They were never broken. Measured on pg18a with the driver present, at the time the gap was filed: 99 collected, 289 checks, 289 pass, 40 seconds. Nothing ran them.

    --pgc-expect-tests is now passed by BOTH jobs, from test/pytest/expected_tests.txt. The flag existed and nothing used it. What it closes is narrower than “pytest passed with no tests” and worse: a file list that resolves to real files and collects FEWER tests than it should. Measured, dropping one file from the guard list:

    unarmed   rc=0   "255 passed"          17 tests gone, nothing said
armed     rc=4   "collected 255 test(s) but expected 272"

    A nonexistent path already fails on its own, so that was not the hole. A valid-but-short list was.

    THE NUMBERS ARE IN A TRACKED FILE, not in the workflow and not in an environment variable, for the reason check_ledger_budget.txt gives about its own: a change to one is then a diff a reviewer sees, sitting next to the test that moved it. PGC_SKIP_TIMING is the precedent for the other choice – set in two workflow files, suppressing whole suites for months, with no diff ever showing it.

    The SPLIT is derived from NO_CLUSTER in test_harness_deps.py, in both jobs, rather than written out again: two copies of which file needs a database is a thing that goes stale silently. test -n guards every derived value, because an empty read would omit the flag and fail OPEN.

    test/pytest/README.md recorded the old reason and it had gone stale twice over: it said CI would have to install from requirements-test.txt first, which pytest-guards already does, and it proposed registering the run in SUITES, which is the cross-harness invocation the independence rule forbids. A second CI job was always the mechanism.

  • The mutation ledger covers a third suite: differential, 204 checks (#752).

    suites_not_covered          250 -> 249
checks_never_observed_red   951 -> 1155
covered                     harness_selftest, native_join_runtime_filter, differential

    BOTH NUMBERS ARE DERIVED FROM THE FILES, never computed from the old ones, and this change is its own argument for that rule. Written against an earlier base the same seed produced 913 -> 1117; #983 then landed forty rows and pruned two, and the census became 1155. Carrying 1117 forward would have been arithmetic that was true when it was written and false when it shipped.

    CORRECTION, and the wrong version is left visible because the sentence was an INSTRUCTION. This entry first said the census is grep -c over the ledger. It is not. The gate compares the budget against the count of rows whose last-red is never:

    awk -F'\t' '$4=="never"' test/check_ledger.tsv | wc -l

    SECOND CORRECTION, by the same rule that kept the first one visible: #1010 inserted the majors a row claims as field 4, so on the shipped tree the last-red is field 5 and the command above reads a major where it expects a date. The form that works is $5=="never", and check_ledger_budget.txt carries it. What the number COUNTS is unchanged – the key is still (suite, part, name), so a row is still one check – which is the reason the major is a set in a field rather than part of the key.

    A plain row count agrees with that only while nothing has ever been seen red, which is true of this tree today (1155 rows, 1155 never, 0 ever red) and stops being true the first time a check is attacked – the event the ledger exists to record. So the number here was right by luck and the method was wrong, in an entry whose whole subject is derive rather than carry. Reported by @OffgridwithJD. The correct derivation now sits in check_ledger_budget.txt beside the number it governs, so the next person does not have to find it in a changelog.

    The ceiling is the registered list minus the ledger’s own suites.

    WHY THIS SUITE, measured rather than chosen by taste. It is the heap-versus-columnar differential correctness suite, so a check that cannot fail there is a wrong answer nobody sees. 16 of the last 300 commits touch it, so the gate will fire. It runs in 19 seconds, and no open change touches it.

    THE RISK THAT DECIDED IT WAS STABILITY ACROSS RUNS, because a suite whose checks move between runs churns the ledger and fires the gate on nothing. Two consecutive runs on PG17: 204 records, 204 distinct names, zero duplicate keys, and the two sets identical in NAME AND IN VERDICT – the second half matters because a flipped verdict churns the last observed red column while the keys stay still.

    Proof that seeding changed behaviour, re-run against this base:

    after seeding, a log with one unseen differential check    rc=1, REFUSED
before seeding, the same log against main's ledger         rc=0, not refused
after seeding, the real log                                rc=0, no false red

    The middle row is the point: the gate refuses an unseen check only in a suite it covers, so before this those 204 checks were invisible to it. The third stops the first from being bought with a gate that refuses everything.

    The tax is the gate working: a change adding a check to differential now needs the ledger regenerated in the same commit, which is a reviewable diff.

  • test/selftest/470 now has the pytest half it shipped without (#994).

    998 added the shell part and no pytest twin, against the owner’s rule that a test

    living in one harness is not finished. test_skip_loop_arms.py is that half, and it is not a port: the shell part runs the sweep over the real corpus and asserts no mismatch survives, which measures the TREE; this one drives the same tool over planted trees whose right answer is known, which measures the INSTRUMENT.

    The distinction is the reason for writing it rather than a justification after the fact. A classifier that filed every site as armless would report zero mismatches, and the shell part’s population premises would still pass on whatever loops remained. Six tests, seventeen checks: the clean site is compared rather than quietly skipped, a renamed sibling is caught, an armless branch and an interpolated one are each counted as themselves, and compared + armless + interpolated == loops so nothing falls out of the report.

    Three removal proofs, each mutation asserted to apply before the run. Stopping the mismatch report reddens the rename arms; comparing the interpolated site literally reddens the other two, because removing the tool’s refusal to compare manufactures a FALSE mismatch on a correct site.

    The third decided the control’s shape. Its first version compared two trees, one clean and one drifted, and a classifier that reports a mismatch naming the WRONG LINE passes that: 0/1 either way, measured on the mutant. So the control plants both loops in one file and pins the drifted loop’s own line, which is the only form that can tell “it found my bug” from “it found something”. Raised in review by @OffgridwithJD.

    A fourth proof closed a blindness the first three shared. Returning 2 from the tool’s main() while still printing correct counters left all six tests green: every one of them reads stdout and none noticed the tool had become unusable. The shell half caught it through its TOOL FAILED fallback, and there it was a PREMISE that failed while the headline arm stayed green. So the exit code is now asserted once in the shared helper, and under the same mutation all six fail. Reported by @OffgridwithJD, whose own first probe of it was invalid and said so: sys.exit(2) appended after the __main__ guard applied cleanly and moved nothing. Asserting a mutation APPLIED is not asserting the behaviour MOVED.

    And the partition arm, compared + armless + interpolated == loops, is the strongest line in the file and the cheapest to satisfy wrongly: a classifier filing everything as armless satisfies it perfectly. It is load-bearing only because the per-bucket arms sit beside it, and that is now written where a reader will find it rather than left to be worked out.

  • The pytest harness reports its own check totals, and the record stream is reconciled against what arrived (#937, third phase).

    A run now ends with checks run: N and an accounting: line counted from the records, so the harness states what it did rather than leaving it to be inferred from pytest’s test count. Five assertions across two tests is five.

    The reconciliation compares two quantities that arrive by different routes: what the recorder held, read in the process that ran the test, and what arrived, read back off the report after it was built. Under -n the second route crosses a process boundary.

    Reconciling the count against the records would have been vacuous, because the second phase made the count len(records) on purpose. Partitioning the records into verdict buckets and summing them is the same trap. Both compare a value with its own definition.

    A record that does not arrive, or that carries a verdict outside the closed set, refuses the run and names the test. Both refusals are proven by injecting the failure from a conftest, because no code in the tree drops a record and an arm that waits for a real defect is not evidence the check can fail.

    It is a transport check and not a completeness check. A record created after the report was built is invisible to it, because both quantities come from one read of the recorder at one instant. That is a limit rather than an oversight: the totals are built from what arrived, and under -n the controller has no recorder to consult. An arm asserts the limit so it cannot be claimed away.

  • Every counted assertion in the pytest harness produces a record, and the count is derived from them (#937, first phase).

    The shell harness makes counting and recording the same call, so no path can do either alone, and reconciles the totals afterwards. The pytest half reaches the same property through Python rather than through the shell’s format.

    It reaches it more strongly, because Python can remove the possibility instead of policing it. The count is not a second variable kept in step with the records; it is len(self._records), a property with no setter. A count that cannot be written cannot drift from the stream it counts.

    Measured before this: _counted() at 15 call sites, the counter incremented by one line and read by one, and zero per-assertion records.

    The record is an object on the recorder, not a formatted line. The shell’s record is tab separated, so pgc_record has to strip tabs and newlines out of a check name. There is no separator here to smuggle, and a name carrying both is asserted to round-trip byte-identical, so the class of defect cannot return silently if these ever become a line.

    cannot_run() records UNRUN rather than a pass. An assertion that declined to run is an outcome like any other.

    A refused assertion leaves no record: a VacuityError means the assertion never ran, so the stream is outcomes rather than attempts.

    Still to come in #937: the verdict resolved from the outcome, and a session reconciliation that can fail.

  • A serial inner Hash Join can push the build-side keys into a direct columnar scan (#752).

    The coordinator wraps core Hash Join. It does not keep a HashPath from joinrel->pathlist. It builds a private path, drains the build side into a tuplestore, then lets Hash replay that spool. The scan skips chunk groups outside the conservative key interval when types and collations match. It also rejects non-matching rows with a Bloom filter of those keys, using the same saturation cap as on-disk bloom filters.

    The path is serial and INNER only. LEFT, SEMI, ANTI, CROSS, parallel, and projection-backed outers are refused. pgcolumnar.enable_join_runtime_filter is off by default until the skip is measured on the join fixture.

    EXPLAIN (ANALYZE) reports Runtime Filter Groups Removed and Runtime Filter Rows Rejected. Those counters are dedicated. They are not InstrCountFiltered1.

  • Every check result is machine-readable, and counting a check is the same operation as recording it (#917).

    check, check_num and check_text printed PASS or FAIL and nothing else, so proving that a mutation reddened one NAMED check meant grepping prose and retyping the result. That is also how a reverted guard once reported plain green while the check count fell from 190 to 186: the suite passed, and the only evidence anything had changed was a number nobody compared.

    lib.sh had eleven places that bumped PGC_CHECKS, each with its own outcome line beside it, which is eleven chances to add a twelfth and forget the line. projections.sh did exactly that with an expect_fail at ten call sites, for as long as it existed. There is now one, pgc_record, so a helper cannot report an outcome without being counted and cannot be counted without reporting one. Every human line is byte-identical; 3,762 call sites is past what a careful refactor can be trusted on, so both harnesses pin the exact strings.

    Each record names the suite, the part, the check, the verdict and the reason. The part matters because harness_selftest sources 40-odd parts into one shell and phrases its premises to be copied, so a key of suite and name is a key of check NAMES rather than of checks: 583 records give 579 distinct pairs against 582 distinct triples. It is derived from BASH_SOURCE, not from a convention.

    A skipped wall-clock check is a fourth counted outcome. Under PGC_SKIP_TIMING, check_timing and check_ratio_needs_quiet_machine printed a SKIP line a reader sees while leaving the count at zero and emitting no record, in branches no arm reached. checks run: now reports the checks a suite encountered rather than the ones it evaluated, and the summary reconciles four counters against it. A suite that skipped every check reports SKIPPED rather than PASSED, which the old zero-check condition caught only by accident.

    The matrix reconciles each suite’s records against the count its log states, and names the cause rather than the arithmetic: more records than counted is a check that ran in a subshell, fewer is a counter bumped outside pgc_record.

  • A ledger of which checks have ever been seen red, and under what (#918).

    Nothing recorded it. That is the gap that let 39 checks across 35 suites ship unable to fail, three of them inside the suite whose whole purpose is to stop exactly that: the gate answered “did anything print FAIL” and had never answered “could anything print FAIL”.

    It records that a named check WAS OBSERVED RED in a recorded run. It does not claim the check is proven able to fail, which needs a named mutation applied deliberately; conflating the two would put a claim in the ledger that nothing measured. It is fed by every real failure, not only by deliberate mutation runs.

    run_all_versions.sh runs the gate before it removes the build directory, which is the only place a matrix run can reach every suite’s log. What the gate refuses is a check the committed ledger has never seen, in a suite the ledger covers. Regenerating the ledger is the intended fix and a reviewable diff.

    The two tracked numbers are different kinds of thing, and the first design treated both as ceilings and deadlocked. checks_never_observed_red is a CENSUS: every new check enters as never, so bounding it means every added check breaks the gate and the only way to land one is to raise a number the design says may only fall. It shipped that way once, at 614 rows, 614 never, ceiling 614. suites_not_covered IS a ceiling, because adding a check to a covered suite does not move it, and the gate refuses to see it raised above its previously committed value rather than leaving that to review.

    The row is keyed on suite, part and check name. The part matters because harness_selftest sources 40-odd parts into one shell and phrases its premises to be copied, so a name-only key is a key of check NAMES: 583 records give 579 distinct pairs against 582 distinct triples. A rename is detected and named rather than silently resetting a check’s history to never, and the mutation column accumulates a set rather than keeping only the most recent attack.

    Bad input fails closed. An unreadable log, an empty one, and a record missing its verdict each returned success before, which is worse than no gate because it certifies.

  • A write that wrote no rows no longer passes as a fixture that built something (#432).

    INSERT ... SELECT ... WHERE false writes nothing and raises nothing. psycopg reports INSERT 0 0 with a rowcount of 0, and nobody in the pytest corpus read either field – so the fixture a test meant to build did not exist, and every assertion below it compared two empty things. That is insert-wrote-no-rows in test/pytest/VACUITY_MODES.md, which now lists it as refused rather than as a gap.

    THE COMMAND TAG DECIDES, NOT THE ROW COUNT. SELECT 0 and INSERT 0 0 both carry rowcount == 0, so a guard keyed on the count alone would refuse every test whose last statement was a SELECT over an empty result. statusmessage is the server’s own tag, so this guard never parses SQL. Measured on PG 18 against a pgcolumnar table: DDL reports CREATE TABLE, SET or TRUNCATE TABLE with rowcount -1, an INSERT ... WHERE false reports INSERT 0 0 with 0, and UPDATE 0 and DELETE 0 likewise.

    A deliberate zero stays writable. A DELETE that must match nothing is a real negative control, and expect.wrote(cur, 0, name) says so: it compares the count and marks the write as named. An unnamed zero fails the test, and naming a count does not excuse a wrong one. The refusal runs in the CALL phase rather than a teardown fixture, because #931 measured that a teardown guard reports the test it guards as PASSED and fails separately.

    TWO PROPERTIES, TWO FILES, PROVEN SEPARATELY. The classifier and the refusal live in test/pytest/test_writes_wrote_rows.py, which needs no database: a stub cursor carrying the two measured fields exercises them exactly. Whether the connection the tests actually use is watched is a different claim that no driver-free arm can make, and test_the_connection_the_tests_use_is_watched makes it through a real INSERT ... WHERE false, on both conn.execute and a cursor the connection handed out – 24 sites and 42 sites in the corpus respectively, so a proxy watching only the connection would leave most of it unwatched.

    The split is load-bearing, measured rather than asserted. Unwiring the connection proxy in conftest.py and changing nothing else leaves the driver-free file at 10 passed and reds the wiring arm alone. That is #917’s defect in miniature – its pytest twin tested a function’s body and left the runner’s CALL to it uncovered, so removing the call kept that half green – and it is why these arms are in two files.

    Prove-by-removal, five mutations, each applied by exact string match and the file asserted to still parse: no refusal -> 2 arms red; the command tag ignored -> 2 red and the wiring arm red; every statement treated as a write -> 4 red; the connection not wrapped -> 0 driver-free red and the wiring arm red; the acknowledgement not recorded -> 1 red. Control: 10 passed driver-free, 9 passed with a cluster.

    Full corpus with a cluster: 246 passed, and none of the 12 write sites already in the corpus reddened – the false-positive budget this guard needed before it could ship.

  • A broad pytest.raises must name a SQLSTATE, and the block must hold one statement (#432).

    pytest.raises(psycopg.Error) claims that one of 254 SQLSTATEs arrived, across 42 SQLSTATE classes, counted against psycopg 3.3.5. It does not claim even that much. Measured on this tree before the guard landed, this reported 1 passed, exit 0:

    with pytest.raises(psycopg.Error):
    conn = psycopg.connect("host=/nonexistent-socket-dir dbname=pgc")
    conn.execute("SELECT pgc_definitely_no_such_function()")
expect.num(1, 1, "the server rejected the call")

    What satisfied it was OperationalError with sqlstate None: the connect failed, nothing reached a server, and the statement under test never executed.

    pgc_vacuity.py now refuses two shapes at collection time, found by walking the ast rather than matching lines, so an offending file does not collect at all. A raises over Error, DatabaseError, Exception or BaseException must bind the exception and pin its SQLSTATE, and any raises block must hold exactly one top-level statement. expect.sqlstate(exc.value, "42883", name) is the honest form the refusal points at; it compares a typed field, refuses a two-character SQLSTATE class as the prefix claim it is, and refuses an empty set of codes so the tuple escape hatch cannot become the hole.

    The family list is bound inside the scan rather than at module level. Every conftest.py under test/pytest/ is imported before collection, so a module-level tuple is writable from the corpus the rule polices – import pgc_vacuity then pgc_vacuity.<the tuple> = () – after which the scan reports zero offences for ever and the suite is green with the guard off and nothing saying so. An arm writes three spellings of the name onto the module and requires the refusal to still arrive.

    This CLOSES raises-too-broad, which moves to VACUITY_MODES.md section 2, and only NARROWS raises-catches-setup, which stays in section 3.4. The statement rule counts TOP-LEVEL statements, so two shapes still walk past it, each being one statement that performs the setup inside the block: a call to a helper, and a compound statement such as a for holding the setup and the statement under test. Both are measured at 1 passed, exit 0, zero offences, and both have an arm asserting the scan reports nothing on them, so the residual is a measurement rather than a sentence. A recursive statement count would catch them and would also refuse a legitimate single-statement loop; what would close the mode is a claim about which statement raised.

    Parsed rather than grepped, because the suite writes the forbidden shape inside a pytester.makepyfile string in every arm. Over test/pytest/*.py the ast finds 5 pytest.raises call sites and reports 0 offences, while a pytest.raises( line regex matches 35 lines, 30 of them inside a string literal or a comment. Swapping ast.parse for that regex makes the layer refuse its own test suite with 22 invented offences and exit 4, which is mutation 11 of 11 in the removal proof.

    THE ARMS LIVE IN ONE HARNESS. test/pytest/test_raises_sqlstate.py carries all of them, through pytester and through the scan directly. An earlier version of this change also shipped a shell mirror, test/selftest/440-a-raises-must-name-a-sqlstate.sh, which checked the scan by GREPPING ITS SOURCE: 44 of its 55 checks were grep -c against the function’s text and it invoked python3 zero times. Reviewing it, @linuxhikerpm measured three faithful neuterings – False and prefixed, nothing renamed, every pinned substring left in place – and all three left that part at 55 passed while the scan went blind. A text pin catches a rewrite or a deletion; it cannot catch False and, which is how a guard actually dies.

    The mirror is gone, and the second reason is the one that settles it: the shell harness and the pytest corpus are parallel in functionality and do not drive each other. A shell part whose whole subject is another harness’s source text is a dependency rather than a parallel guard – it asserts against an implementation instead of against the product. So the neutering proof is now two arms that copy the layer, disable one condition faithfully, and require the copy to go blind while still containing the text a grep arm would have pinned.

    THREE DEFECTS @linuxhikerpm FOUND IN THE GUARD ITSELF, each reproduced before it was fixed. A bare exc.value.sqlstate, and a code = exc.value.sqlstate never read, both satisfied the pin while asserting nothing – the rule counted any attribute named sqlstate anywhere in the body. It now requires the read to reach a CALL, following one hop of assignment so the honest code = ... / expect.text(code, ...) form is not refused. And pytest.raises(expected_exception=...) escaped BOTH rules, because the class was read from call.args[0] and the item was skipped before it was recorded, which made the statement rule silently conditional on the class being positional while the documentation stated it unconditionally.

  • The harness guards run in the gate, without a database, and which files that is gets DECIDED rather than listed (#432).

    conftest.py imported psycopg at module scope, and conftest is imported before every run, so a database driver was a hard requirement of COLLECTING the whole pytest corpus – including every test that never opens a connection. Deferring that one import into the two fixtures that connect lets the guard-testing half run where the gate runs: .github/workflows/ci.yml gains a pytest-guards job with no database, no build, an interpreter and two pinned packages. The job derives its file list from NO_CLUSTER in test/pytest/test_harness_deps.py and its pins from requirements-test.txt, so neither is a second copy, and it asserts psycopg is ABSENT before running – otherwise the tests would pass for the ordinary reason and prove nothing about the coupling.

    NO_CLUSTER is now decided, not declaimed. The only arm over it asked whether the files it named EXIST, which is one direction of a membership claim, and the missing direction is the one that loses coverage: a database-free test file that nobody adds to the list is simply absent from the job, every arm stays green and nothing says so. It had already happened twice – test_build_refusal.py and test_layer.py both need no database and neither was listed. The property is now computed from the corpus by an ast walk and required to equal the list in both directions, so the job runs every database-free file rather than the four somebody remembered.

    An ast walk rather than a line regex, because three shapes here defeat a grep: a file may name the driver in a docstring, discuss a cluster fixture in prose, or build another test as a string for pytester. And needing a database is not importing the driver – a test reaches a cluster through a FIXTURE and may import nothing – so a file is cluster-bound if it imports the driver at module scope, if any test or fixture in it requests a fixture that reaches a cluster, or if it drives a cluster-bound file as a subprocess. The connecting fixtures are read off conftest.py rather than named in the classifier.

    Every rule was proved by removal: eight mutations, each asserted to have applied and restored byte-exact, each reddening a named arm. Two of them survived the first pass and found real gaps – a closure over each file’s own fixture graph that changed no classification, removed as dead, and a prose filter with no killing arm, which now has one. The classification of all twelve corpus files was exercised against reality: each of the six called database-free passes with import psycopg shimmed to raise and no usable pg_config, and each of the six called cluster-bound fails, for the driver or for pg_config and nothing else.

    Because the corpus is not in SUITES and the job runs only the database-free files, none of those arms runs in the gate, so test/selftest/350-the-pytest-corpus-must-be.sh runs the membership decision through the module’s command line, with its own fixture corpus to fail against. A guard that does not run is a comment.

  • Exact zone-map boundary coverage now lives in matching shell and pytest tests (#831).

    The <= and >= arms put the constant exactly at a row-group minimum or maximum and compare returned rows with a heap twin. The <, >, and = mirrors assert groups removed with bloom disabled, so a conservative pruning regression cannot hide behind a correct answer. Each of the five one-token strategy mutations was proved to fail its corresponding assertion.

  • The source fingerprint has one implementation, in Python, and both harnesses call it.

    test/lib.sh and test/pytest/pgc_cluster.py each carried their own answer to “what was this binary built from”. In one day the pair produced four defects, two in each copy, and not one was found by whoever wrote that copy: the Python side never walked objstore/*.c, then mixed in each file’s bare name so src/module.c and objstore/module.c were interchangeable, then omitted each build directory’s Makefile; the shell side hashed xargs -0 cat | md5sum, a stream with no per-file boundaries, so moving bytes between two files left the hash unchanged while the source no longer compiled. The Python docstring asserted parity with the shell throughout all four. It was false when written and stayed false through two rounds of fixing.

    test/pgc_fingerprint.py is now the only implementation. It is Python rather than shell because the portable half should be the one that survives: bash is largely a GNU thing, while Python is present on FreeBSD and Windows where bash is not. It uses the standard library only and runs on the system interpreter, never the pytest virtualenv, so a freshness gate cannot depend on the test dependencies of a harness it gates.

    It is also faster. The shell forked md5sum once per file; the module starts one interpreter:

    shell, forking md5sum per file      239 ms per call
the module                           26 ms per call
across 261 suites, twice each        124 s  ->  13 s

    Unifying them closed a fifth defect that neither implementation had been suspected of. sort -z orders by LOCALE COLLATION and nothing in the harness pinned a locale, so one tree fingerprinted two ways depending on the machine:

    LC_ALL=C             6d122a7158d5
LC_ALL=en_US.UTF-8   0b59bd75fa4f

    en_US.UTF-8 is a common desktop default, so a developer could stamp a tree and have CI read it back and call the binary stale. The module sorts bytes, which is what LC_ALL=C produced and what every stamp already on disk was written with, so no existing stamp is invalidated.

  • The test harness refuses to measure a binary that was not built from the source under test.

    test/run_all_versions.sh builds and installs once per major and then runs every suite with PGC_SKIP_BUILD=1, so no suite could tell whether the binary it measured came from this tree. The controller now records a fingerprint of the build inputs after a successful install, and pgc_setup checks it in every suite, whether that suite built or skipped. A mismatch is fatal and names both fingerprints.

    A second check covers the other half. make install does not reload anything: shared_preload_libraries maps the library at postmaster start, so a reinstall under a running server leaves the backends executing older code than the file on disk. The .so mtime is compared against pg_postmaster_start_time(), and the message says to restart rather than only that something is wrong.

    Neither check fails when it cannot answer. Someone who ran make install by hand has no stamp, so that case prints freshness UNVERIFIED and says which question went unanswered, rather than printing nothing and letting a reader assume it passed.

    pgc_freshness_verdict and pgc_running_binary_verdict are pure functions of strings, so test/selftest/340-the-binary-must-be-built-from.sh drives them directly: 27 arms including both empty inputs, a non-numeric epoch, equal timestamps on the boundary, sensitivity to each fingerprint input class, and the three the review added – the derived build-directory set, the running binary check driven against a live cluster, and devloop writing the stamp. harness_selftest goes from 261 checks to 288, both measured.

  • IN (...) and = ANY(array) now test each listed value against row-group zone maps and bloom filters instead of reducing the list to its [min,max] hull (#752).

    A scattered list whose hull spans a table could previously read every row group. The set is now one internal predicate whose element tests are a disjunction, while the outer predicate list remains a conjunction. The executor still rechecks exact membership, so pruning remains conservative.

    Per-element evaluation is capped at 128 non-NULL entries. Larger lists retain the bounded two-key hull because exact vector refinement otherwise costs elements times rows.

  • test/projection_rewrite.sh, 84 checks. Nothing in the tree asserted that a projection answers after a rewrite, which is why this was silent.

    Every arm compares a pgc_set_hash of read_projection against the base table rather than checking that the call did not raise, so a projection re-recorded EMPTY fails – which matters because the correct end state after a bare TRUNCATE is an empty projection that answers. Every arm also asserts what its operation DID (REWROTE, NOOP or FAILED) and reports its properties as UNMET_PRECONDITION rather than as passes when it did not: an operation that failed or no-opped leaves the storage id unchanged and read_projection answering, which is indistinguishable from a path that handles projections correctly. Three arms carry pgcolumnar.vacuum, vacuum_sorted and cluster, which already re-record for themselves, so a future fix moved into the table-AM callback reddens here instead of double-recording.

Changed

  • Serial join runtime filter is on by default (#752).

    Three measured cases, which is the argument. The docs being ready is why the guidance exists, not why the default is right:

    clustered on the join key   19 of 20 chunk groups removed, 1 read
scattered                   0 groups removed, but Bloom rejects more
                            than 15,000 of 19,800 non-matches
build side too large        the Bloom disables itself

    The third is why this is safe everywhere. A filter that is on for every user needs an answer for the case where it helps nothing, and turning itself off is that answer. Group skip still needs the fact table clustered on the join key, and a scattered fact table still cannot drop groups. SET the GUC off to compare.

    NOT MEASURED: the overhead of a filter that is on, not saturated, and rejecting almost nothing, which is a join where nearly every row matches. Saturation bounds the pathological end and the scattered case bounds the middle, so this is reasoned rather than measured. It is the gap to close if the default is ever doubted.

Fixed

  • The standing parity arm graded a hand-written list, and nothing enforced it (#432, #1046).

    test_the_ported_suites_in_this_tree_are_graded_one_for_one grades the pairs it is GIVEN. A pair that existed and was not given to it was not graded, and nothing said so: the arm passed, grading the ones it knew about, and reported a clean verdict for a tree it had not fully looked at. Absent-from-the-list and no-gap-found produced the same green.

    Latent throughout – the declared set happened to equal the tree, so nothing was ever silently ungraded. Found by @OffgridwithJD, whose eighth port would not have been graded by it, and it would have gone live the moment a ninth landed undeclared.

    COMPLETE and INCOMPLETE are declared at module scope and asserted BOTH ways, the shape SHELL_REFERENCES already uses: a pair in the tree that is not declared reddens with the stem named, and a declared stem whose pair has been deleted reddens too. A pair is derived as test_<stem>.py beside test/<stem>.sh, so the 24 pytest files with no suite stay out without a second exemption list.

    It forbids one thing, and that was the decision rather than an oversight. An incomplete pair could previously land declaring nothing; now it must carry a stem and a reason. The escape hatch is attached rather than the case forbidden, and the cost today is zero – 8 pairs exist, 8 are declared, INCOMPLETE starts empty. Named by @OffgridwithJD, who pointed out that “forbids nothing that was allowed before” was the comfortable phrasing.

    Five mutations, each asserted to apply by md5 and the tree restored after:

    drop a stem                            every pair in the tree is declared
declare a stem with no files           every declared stem is a pair that exists
move a stem to INCOMPLETE, thin reason every incomplete pair says why
a stem in both lists                   no stem is both complete and incomplete
move a stem to INCOMPLETE, full reason ALL GREEN -- the hatch works

    The last is the one that matters: it demonstrates the escape hatch is usable, rather than only that the guard fires.

    AND IT DID NOT, FIRST TIME. The cardinality premise below was added AFTER that mutation was run and the mutation was never re-run, so “the hatch works” was true of a file that no longer existed. The premise counted len(COMPLETE) against a hard-coded 8, so moving one stem to INCOMPLETE took it to 7 and reddened the suite: a pair could not be declared incomplete without going red, which is the one thing this change exists to allow. Caught by @OffgridwithJD reviewing, not by the author re-running.

    The premise now counts the DECLARED TOTAL against the pairs that exist on disk, both derived. That keeps the guard – an emptied COMPLETE still reddens, because 0 does not equal 8 – permits the hatch, and removes a hard-coded number that would have needed an edit the first time a ninth pair landed, which is the budget shape #982 argues against.

    All five mutations were then re-run against the file that ships. Three of them redden the accounting premise as well as their named arm, which is correct rather than noise: a dropped stem, a phantom stem and a double-declared stem each genuinely break the accounting, and the named arm fires beside it to say which.

    Hoisting the list made an existing guard fire, which is the sweep working. The standing arm’s loop now iterates a module-level name rather than a literal, so test_loop_coverage_premise.py demanded a cardinality premise – an empty COMPLETE would leave every arm unrun and the verdict comparison trivially equal.

  • compare_to_bash.py read five of the eight check helpers lib.sh defines (#432, #1040).

    The bash side of the parity tool matched check(?:_num|_ratio|_text|_timing)?, so check_unrunnable, check_skip and check_ratio_needs_quiet_machine were invisible. A property asserted through one of the three was never reported MISSING and could not move rc, which means a pair could grade one-for-one on the strength of the grader’s blind spot. hilbert_locality was exactly that, and #1041 closed the two properties it was hiding.

    It never drifted out of date: 0cbf574 introduced that pattern, and check_unrunnable already had 21 call sites that day. Same defect shape as the python side in #1036 and #1038 – a rule true of most of a class taken for a property of the class – sitting on the other side of the same tool for five days.

    The helper list is hand-written, so it is pinned the way _NAME_ARG is: an arm re-derives it from lib.sh’s DEFINITIONS and fails with the helper named. Two more arms state what the tool does not cover – the four suite-local helpers (check_structure, check_reconstruct, check_split_happened, check_float), none of whose suites has a pytest twin, and the pattern shape that used to make a prefix unreadable.

    The longest-first ordering is NOT what makes that work, and the code said otherwise until it was measured. Python’s re backtracks across alternatives, so a pure reorder reads both names identically. What the old pattern could not do was read check_ratio_needs_quiet_machine at all: it matches check_ratio, wants whitespace, finds _needs, backtracks to the empty option, wants whitespace after check, and fails. Measured on a fixture holding both, the old form reads ['short'] and this one reads ['short', 'long']. The arm pins the pattern shape; the ordering is readability.

    The population is check or check_<something>, not check[a-z_]*: the loose form also matches checks_in in decode_interrupts.sh, a counting utility that returns a number and records nothing.

  • compare_to_bash.py read the wrong argument for the four helpers whose name is not last (#432, #1036).

    The previous fix replaced “the first quoted argument” with “the last argument”. That is true of 14 of Expect’s 18 helpers, but it is a property of most of them rather than of the class, and the last argument is a real string in each of the other four – so a wrong name looked exactly like a right one:

    call last argument the name it records
    refusal(result, name, *patterns) a message PATTERN name, argument 1
    cannot_run(reason, detail="") the DETAIL of one run reason, argument 0
    plan_marker(plan, key, name=None) a plan KEY the name= keyword only
    plan_node(plan, ..., name=None) a field of the NODE the name= keyword only

    refusal is the worst: the real name goes MISSING and a fragment of an error message arrives as an EXTRA, which is two false entries from one call.

    Measured over every pair in the tree at 73e8e3d, with the table as the only variable: 68 extras, now 67. Two were false – a plan_marker key and a cannot_run detail, both on hilbert_locality – and UNMET_PRECONDITION appeared in their place, the reason code cannot_run really records. No pair’s verdict moved, because rc is driven by MISSING and extras never moved it. That is why nothing caught this: the tool reported a plausible list, and only the list was evidence either way.

    UNMET_PRECONDITION is reported as an extra only because the tool cannot see the bash side of it. hilbert_locality.sh:574 and three lines after it check that property with check_unrunnable, which the bash extractor’s check(_num|_ratio|_text|_timing)? does not match. Widening it by that one alternative and nothing else takes that pair from rc=0 missing=0 to rc=1 missing=2, every other pair unchanged. Filed as #1040: it is a port’s worth of work, not a tool fix, and this change is only what made it visible.

    The positions live in a _NAME_ARG table, because the tool is deliberately standalone (ast, re, sys) and importing Expect to ask would pull in pytest. A hand-written derived value goes stale, so it is pinned: a drift guard reads the real signatures out of pgc_vacuity.py, recomputes every entry, and fails with the helper named. That guard first passed over a missing refusal entry – name IS its last DECLARED parameter, since *patterns is not – and now accounts for the vararg.

    A second coincidence sat inside the clause that fixed the first, found by @OffgridwithJD in review. -1 is a claim about the CALL SITE while the guard reads the SIGNATURE, and they agree only while no optional parameter sits after the name: expect.rows(got, want, "THE NAME", "the reason") read the reason, and expect.plan_marker(plan, "key", "THE NAME") read nothing at all, dropping a name silently. Latent rather than live – no call site passes a trailing optional positionally – but #1037 makes allow_empty a reason string, which is exactly that argument. Closed in the signatures rather than in the reader: rows, row_set, plan_marker and plan_node take everything after the name as keyword-only, so the wrong call is now a TypeError. No call site changed; all four already used keywords.

  • A BOGUS-verdict ledger record is refused by naming the verdict, not by field count (#1013).

  • The star-schema join how-to names clustering on the join key (#752).

    Group skip was already measured: 19 of 20 groups when the keys are local, 0 of 20 when they cycle. The page named the GUC and not that discriminator. A fact table that is not clustered on the join key still holds every key in every group, so the filter cannot skip. The fold over a join is a later slice.

  • A skipped arm records under the name it would have used, so a skipped arm and a deleted one are no longer indistinguishable (#994).

    Four check_skip calls stood in for 19 named arms under a name none of those arms has. When the condition failed, those 19 produced no record at all: a reader could not tell which arms did not run, and the ledger could not tell a skipped arm from a deleted one, because a skipped arm’s row has no matching record exactly as a removed check’s would.

    The convention was already in the tree, 30 lines below one of the offenders: skip under each arm’s own name, in a loop.

    The issue counted 17. It is 19. Two arms in sorted_pathkeys.sh go through ansp, which records under its first argument, and a sweep that looked for check did not see them.

    One arm’s name interpolated the very variable whose emptiness causes its own skip, so the skip would have recorded a key no real run emits. That name is now stable and the collation it names moves into the display, which is not the key.

    Six sites, not four, and 19 arms. Two arms go through ansp, which records under its first argument. Two sites hold their arms in the then branch with the skip in the else, which a classifier looking only forward reads as having no arms at all.

    340 also skipped five arms behind a branch whose comment said they had already been skipped above. Above had skipped the three premises, not these five, so on a box with no non-root user five arms produced no record.

    A new selftest part asserts every skip loop names exactly the arms its sibling branch would emit. The loop duplicates those names, so a rename desynchronises them silently and the skip records under a name nothing emits, which is the failure this change exists to remove. That is not hypothetical: writing this, a name from another open PR’s rename went into the loop, and the comparison is what caught it.

    The part reports what it did not compare. One loop’s sibling arm is generated by a loop of its own, so a literal comparison would be wrong in both directions. That loop is the one this change repairs, and it is not covered. A total of zero mismatches would otherwise read as a corpus in agreement.

    This is the precondition for arming the orphan guard in #983. Until a skipped arm records under its own name, absence cannot mean removal.

  • A conftest can no longer switch a vacuity rule off by rebinding a name the layer reads (#924).

    pytest imports conftest.py from the directory it is policing, into the policing interpreter, before collection. Every module-level name in pgc_vacuity.py is therefore writable by the code it judges.

    958 closed the datum one exploit used. The three scans that read such data

    are module-level names one frame further out, and each was a two-line conftest away from being a no-op. Measured with the pinned runner:

  GUARD               no conftest    with the scan rebound to a no-op
  order collapse      REFUSED rc=4   PASSED rc=0
  broad except        REFUSED rc=4   PASSED rc=0
  raises not pinned   REFUSED rc=4   PASSED rc=0

Plugging a fourth name would reopen this again: the transitive closure from the eight hooks is 31 of the module’s 47 names, ast among them. So the layer snapshots its own bindings at import, holds the snapshot in a closure, and refuses a run in which any of them changed. Names added later are covered without being listed anywhere.

No allowlist is needed: the module contains no global statement, so every module-level binding is constant after import.

The bindings are restored before the refusal is raised. pytester runs its inner session in-process on the same module object, so without that an inner conftest’s rebind stays made for every test that follows.

This is a cost guard, not a lock. Reaching into the hook’s __defaults__ still reaches the closure. The criterion is that silencing a rule must cost more than stating a reason, which expect.cannot_run(REASON, detail) does.

The shell harness needs no equivalent, and not because bash is simpler: its policing runs in a different process from the code it polices. selftest/260 reads lib.sh with grep and awk and never sources the file it judges.

  • The sentinel sweep no longer excludes an assertion by accident of naming (#938).

    _comparisons() selected on the first two parameter names, so wrote(cur, want, name) sat outside because its first parameter is not called got. That happens to be the right answer for a cursor, and it would also have been the answer for a future comparison whose first parameter was left. Exclusion is now a positive match on the kind of the left operand, and inputs == selected + excluded fails when a method matches neither rule. A list of method names is not the fix.

  • A loop that never ran asserted nothing, and half of that was already refused by a mechanism nobody had recorded covered it (#432).

    assert-inside-a-loop-over-zero-rows in VACUITY_MODES.md 3.5 is two shapes.

    Already refused. When a loop’s body holds a test’s ONLY counted assertions, a zero-trip loop leaves the count at 0 and pytest_runtest_call raises VacuityError. Measured on a planted test rather than read off the hook: the zero-trip case fails with “made no counted assertion” and the same loop with one row passes.

    Was open. When the test ALSO asserts outside the loop, the count is non-zero, the test passes, and the loop’s assertions simply never ran. That is the shape a query returning no rows produces, and the shape a glob matching nothing produces.

    test_loop_coverage_premise.py requires a cardinality premise for exactly that shape. THE POPULATION, measured over the whole corpus before the arm was written: 20 loops non-empty by construction and so unable to be zero-trip, 0 in the already-refused shape, and 2 at risk – both of which already carried a premise. The arm is green on arrival, which is the point rather than a weakness: the property was true and nothing was holding it there, so what this catches is the third one.

    IT NARROWS RATHER THAN CLOSES, and 3.5 names the residual. The honest requirement is a premise bounding the cardinality of THIS iterable; what is enforced is a counted assertion outside the loop taking len(...) of something. test_harness_deps.py’s loop iterates sorted(found) while its premise bounds len(files), because found is built from files in a preceding loop – so a rule demanding the names match would reject correct code, which is how a guard gets switched off. The residual is a loop whose premise bounds the wrong collection: a reviewer catches it, a sweep does not.

    Section 5 gains entry 6, struck and anchored to its mode id, which is the first use of the rule the previous commit added.

  • Section 5 of VACUITY_MODES.md, the “what to add next” list, is checked (#432).

    1a, 2 and 3 are all compared against the mode ids on disk. Section 5 was prose, and it was wrong: entry 1 still said “the constant exists and nothing writes it” long after query_error() existed and test_failed_query_sentinel.py carried ten arms over it, including the producer’s uniqueness and the constant’s non-uniqueness that is the reason the producer exists.

    That is the most expensive place in the document for a stale sentence, because its only reader is someone about to build something. The near-miss one document over is what it costs: a bad enumeration of test/selftest/340 made an existing block look like a coverage gap, and the duplicate was written and proven to discriminate before the duplication was noticed.

    What is checkable is the anchor, not the work. Entry 1’s work landed under four test names, none of them the one the entry proposed, so asking whether the NAMED test exists would have passed and said nothing. Two arms hold the list instead:

    • every entry names at least one mode id, so it is tied to the inventory at all. Entry 1 named none, which is exactly how it stayed wrong.
    • no UN-STRUCK entry names an id section 2 already claims as refused. An entry whose id has reached section 2 is done by the document’s own accounting, whatever the test ended up being called.

    Entry 1 is struck and anchored to error-swallowed-to-empty, and it records what the entry got wrong rather than replacing it: both halves of “the constant exists and nothing writes it” were false – two sites already minted sentinels by hand, and the missing thing was the REFUSAL in four of the five comparisons.

    Fixing it made the second arm vacuous on this document, because with every entry struck there is nothing left to refuse. The planted fixture beside it is therefore the whole of its evidence, and the document says so rather than leaving it implied: a two-entry fixture where moving the open entry’s id into section 2 must be named, with the clean control beside it.

    The totals do not move – 28 refused, 44 not refused, 72 named – because the mode was already counted as refused. Only the list that tells the next person what to do was wrong.

    And 3.6 now records measured populations, so the next entry is chosen on evidence. Section 5’s new rule is that an entry must name a mode id, which makes WHICH id worth measuring. Four of 3.6’s were counted by AST scan over the whole corpus: truthy-cursor-from-execute has 7 sites and all are benign – every one is x = cur.execute(...), idiomatic in psycopg3, and zero branch on it, which is the dangerous form; empty-query-string-succeeds has 0; the multistatement mode has 1, a setup that fetches nothing with a count(*) premise right after it; and the server-cursor rowcount mode has 2, both legitimate. So all four are PROSPECTIVE, and a guard for any of them would be insurance rather than a closure. Recorded because a refusal with no population has not refused anything, and 1a counts section 2 as “refused today”.

    One instrument defect of mine, caught by that fixture: the parser matched the SECTION HEADING as an entry. re.split(r"^## ") leaves a chunk beginning “5. What to add next”, which the numbered-item pattern also matches – inventing an entry 5 that is the title and colliding with the real entry 5. The fixture reported 3 entries in a two-entry document, which is how it was found.

  • The pytest corpus no longer drives the shell harness for properties the shell harness already holds, and the inventory of what remains is down to four calls, none of them debt (#432).

    CONTEXT.md’s rule: the two harnesses are parallel in FUNCTIONALITY and independent in CALL. test_build_refusal.py drove test/lib.sh for seven calls over pgc_write_source_stamp, pgc_source_stamp_path, pgc_freshness_report and pgc_freshness_verdict – four PURE SHELL functions, so the python arms were a second measurement of someone else’s subject, agreeing with it by construction.

    Five arms removed and not one needed porting. test/selftest/340 already held every property they asserted, and more of it in each case:

    the python arm what 340 already had
    the stamp writer reports failure the same, plus a premise that the stamp really was not written
    two installations of one major do not share a stamp the same, plus pkglibdir keying and two unreadable pg_configs
    the report names each file and states how many the same two arms
    an empty manifest says so rather than printing nothing the same arm
    a failed digest is unknown, never a false stale the same, over TWO unreadable files, plus the premise below

    340’s version of the last one carries a premise the pytest twin did not state: that the unprivileged read AGREES with the privileged one while nothing is denied. Without it the arms measure the user switch rather than the permission denial.

    The first measurement of 340 was wrong and nearly cost a duplicate. Enumerating its checks with grep -cE '^check "' gave 81. The real number is 89: 340 has indented check calls inside an if and a for, and the eight the sweep missed are exactly the unreadable-source block. On that bad count one python arm looked like a genuine gap, and a duplicate of it was written – and proven to discriminate against a mutation – before the duplication was noticed.

    The tell was a duplicate check name. pgc_ledger.py reported “duplicate check name in one run, so one ledger row covers 2”, and the first response was to rename the new check. The right response to a name that already exists is to ask WHY it exists. Renaming it hid the only evidence that the work was unnecessary. A check sweep has to be anchored at ^[[:space:]]*, not at column 0.

    Two mutations were run against the duplicate before it was discarded, and the first was a no-op for a reason worth keeping: pgc_source_fingerprint returns empty because the MODULE prints nothing and exits 0, not because the wrapper’s rc != 0 branch fires. Mutating that branch changes nothing on this path. The property lives in test/pgc_fingerprint.py, and the wrapper’s contribution is only that it does not substitute a value for the module’s empty answer.

    What remains in test_build_refusal.py is four calls in two arms, both named: test_the_two_fingerprint_implementations_cover_the_same_inputs, which is the one permitted cross-reference because the property IS the relationship between the two implementations, and a historical-parity arm whose fixture is its own. _sh_fp_as is deleted with its last caller.

    TESTS.md loses the rows naming the deleted arms, and three prose passages that named them are rewritten to say where the property lives now rather than left pointing at arms that do not exist. A backticked name is a claim that it exists.

  • A count grep never produced no longer reads as “present” (#929).

    922 replaced roughly 28 producer | grep -q PAT tests with

    [ "$(grep -c PAT ... || true)" != 0 ], which fixed a real EPIPE race (#486). The replacement answered present where the original answered absent whenever grep produced no stdout, and a pattern that does not compile is the way to get there:

    grep -cE '[' file   ->  stdout is []   (empty, not "0")
[ "" != 0 ]         ->  TRUE           (a STRING comparison: "" is not "0")

    So the test reported the pattern present for a question it never managed to ask.

    Every pattern in the tree is valid today, so no site was wrong. The hazard is the DIRECTION of the next edit: a premise arm phrased to want present – and most are, because a premise asserts the fixture really is in the state the test needs – turns GREEN when its pattern stops compiling. It passes BECAUSE the instrument broke, which is the failure this harness spends most of its effort refusing. The old form failed red.

    23 sites compared a count as a string; they now compare numerically, which is behaviour-preserving in every case that is not broken. Measured:

    input                      [ "$n" != 0 ]   [ "$n" -ne 0 ]   stderr
a real count: 0            false           false            no
a real count: 3            true            true             no
EMPTY (grep usage error)    TRUE            false           YES

    Both spellings failed the same way. = 0 is an ABSENCE claim, and on an empty value it is false – which does not assert absence, and is the safe direction once it is loud. All 23 sites pass exactly one input to grep, so the value is always a bare number and a numeric comparison cannot be confused by file:count output.

    THE POPULATION RECONCILES, and the first version of this entry did not. It said “58 sites, of which 20 string-compared and 32 numeric” – and 20 + 32 is 52. The 58 came from a broad grep and the 20 from the sweep’s own narrower one, so two instruments were reported as one measurement. With the sweep’s pattern corrected:

    on main (cfe1fde9)   58 inputs = 23 string-compared + 35 numeric
after this change    58 inputs =  0 string-compared + 58 numeric

    test/selftest/440-a-count-grep-never-produced.sh holds the arms and a heredoc-aware sweep requiring zero string comparisons on a grep -c, so the class is closed rather than the 23 instances.

    THE SWEEP’S FIRST PATTERN COULD NOT SEE THREE OF THEM. [^)]* stopped at the first ), which is inside the GREP PATTERN rather than at the end of the substitution, so any pattern containing a parenthesis hid its own site: sorted_mark_rename.sh:183 and sorted_pathkeys.sh:53 and :456, each spelling grep -cE '^ *(->)? *(Incremental )?Sort'. The guard and its population came out of the same regex, so the guard agreed with the count by construction – which is why one plant now carries a parenthesis and differs from the plain one in nothing else. Reported by @jdatcmd. The sweep skips comments as well as heredocs: a flat grep counts the paragraph that documents the idiom, which is how a guard comes to flag its own explanation.

  • test/harness_selftest.sh can no longer exit 0 having evaluated nothing (#934).

    Handed a pg_config the box does not have, it printed four lines, never reached its summary, and exited 0. Measured on main:

    /usr/local/pg18a/bin/pg_config    rc=0  1246 lines  checks run: 610
/usr/local/pgNOPE/bin/pg_config   rc=0     4 lines  no summary at all

    A caller cannot tell the second from the first. It is not hypothetical: it cost a whole mutation round, because the control and the mutated arm both reported rc=0 with zero FAIL lines – which reads exactly like “the mutation changed nothing”, the conclusion the run existed to test. The log being 4 lines instead of 1246 was the only thing that gave it away. The default argument is /usr/local/pg17/bin/pg_config, which the audit container does not have, so the wrong invocation is the easy one to make.

    TWO CAUSES, and the second is the one that generalises. _bindir was assigned from a command that had failed, so every later PATH was wrong; and part 010, which is SOURCED, then took its own skip path and called exit 0 – which exits the DRIVER rather than the part.

    The driver now refuses the argument before sourcing anything, on two predicates because one is not enough: a pg_config can exist and be executable and still answer nothing, which is the shape that produced the empty _bindir. Both of part 010’s skip paths now exit 66, the status lib.sh calls PGC_EXIT_SKIPPED, paired with the SKIPPED (ran no checks) line the runners already require beside it.

    test/selftest/430-the-self-test-must-not-report.sh holds the arms, including a heredoc-aware sweep requiring that no part exits 0 – closing the class rather than the two instances. The sweep has to be heredoc-aware because the parts generate fixture scripts that legitimately end in exit 0: it sees 2 sites before this change and 0 after, where a flat grep -c 'exit 0' sees 13 before and 24 after – the flat count moves because the new part’s own fixtures add ten, for a reason that has nothing to do with the defect. It sees exit 0 and a bare exit; it cannot decide exit $? or exit "$rc", and each of those is zero in the tree today.

    After: rc=2 and a named refusal for a missing pg_config, for a path that is not a directory, and for one that answers nothing; 604 passed + 0 failed + 0 unrunnable with a real one.

    The part’s own comment at line 3 already said why this mattered – “a quiet skip means the guard stops being tested that run without anyone noticing” – four lines above the first exit 0.

  • The pytest corpus reaches into the shell harness in three files instead of four, and the inventory that records it is now a mechanism rather than prose (#432).

    CONTEXT.md’s rule: the two harnesses are parallel in functionality and independent in implementation. A pytest test that drives test/lib.sh is the first measurement wearing a Python wrapper – it agrees with the shell by construction and can never report it wrong – so the coupling turns a twin into a mirror.

    test_build_refusal.py was the largest item on that inventory at 36 cross-harness calls. 22 of them are gone. Their subject was test/pgc_fingerprint.py, which has been the one implementation since #907, and pgc_source_fingerprint and pgc_source_manifest in lib.sh are thin wrappers that shell out to exactly that module – so the path was python -> bash -> lib.sh -> python3 -> the module, and taking the middle two out changed no subject. Measured before converting a single arm: the fingerprint is byte-identical through both paths, the manifest identical line for line, and both agree across LC_ALL=C, C.UTF-8 and en_US.UTF-8.

    Two of those arms need a separate process rather than an in-process call – one reads as an unprivileged user, because root ignores chmod 000, and one varies the locale – and both now use the module’s own CLI, which is the entry point lib.sh itself uses with lib.sh taken out of the path.

    13 calls remain and only 7 are debt: they drive pgc_write_source_stamp, pgc_source_stamp_path, pgc_freshness_report and pgc_freshness_verdict, which are PURE SHELL rather than wrappers over shared code, so those properties belong to the shell harness. Two are the one permitted cross-reference, named as the rule asks, and two are a historical-parity arm whose fixture is its own.

    SHELL_REFERENCES in test_harness_deps.py makes the inventory falsifiable. It declares which files reach across and by what mechanism, and asserts set equality in both directions: a new file that reaches in reddens, and a file that stops reaching and is left in the list reddens too – which is what stops a record of debt becoming a permanent exemption.

    A file-level guard, which is what the rule asks for and the most it can honestly be: within a flagged file it cannot tell a path joined onto the real tree from the same name joined onto a tmp_path, because both are the string lib.sh and only the dataflow says which.

    The other direction: test/selftest/370 is deleted. It pinned the POSITION of plan_marker’s empty-plan refusal by reading the Python source as text, which is all a shell part can see. It cannot run the function, so it cannot tell a refusal that still fires from one stranded behind an early return – the only thing the position is for. The property moved into test_guards_pinned.py as test_the_empty_plan_refusal_precedes_the_arms_it_protects, which takes the class off the expect fixture so the file still imports nothing. 370’s other three properties were already covered; confirmed by running those arms, not by reading them.

    That arm discriminates – move the refusal to the end of plan_marker and it fails – and it is NOT load-bearing today: with the refusal at the end, plan_marker([], absent=True) still refuses, because plan_marker has no early return. So 370’s stated reason, that the absent arm returns a pass first, describes a shape the function does not have. It is labelled prospective insurance against a refactor that adds one, rather than sold as a live hole.

    CONTEXT.md said seven shell files reach across and the number is three. lib.sh, selftest/030 and selftest/040 matched only pgc_cluster_datadir and pgc_cluster_is_ours, which are shell functions defined in lib.sh – rule 3 of that entry, “a word that merely looks like a filename”, caught for the second time in the entry that states it. selftest/350, 360 and 380 are the real three.

    test/selftest/360 is rewritten rather than deleted, and it now checks more. Eleven of its seventeen arms were text pins on pgc_vacuity.py: that the unrunnable field is written, that something reads it, that the read reaches session.exitstatus, that the override is conditional. Those are gone. What is left is the kind CONTEXT.md permits – a property that IS the relationship between the two harnesses, so it cannot be stated from one side – and there turned out to be THREE such properties where the part checked one.

    The part checked the INCOMPLETE exit code and left two duplications beside it unchecked: the closed list of unrunnable reasons, and the one-line shape an unrunnable check prints. Both are now parsed out of both files and compared, with a drifted fixture for each so the comparison can fail. The reason list is the worse omission of the two, because being closed on both sides is its whole purpose.

    Both deletions were measured first. With session.exitstatus = EXIT_INCOMPLETE made unreachable in pgc_vacuity.py – the defect exactly as it shipped – the four behavioural arms in test_layer.py go from 4 passed to 2 failed, 2 passed. Two is correct rather than partial: the other two assert exit 1 for a run with a real failure and exit 0 for a run with nothing unrunnable, and neither outcome moves. The file was restored and compared byte-for-byte afterwards.

    And the reason the part gave for keeping those pins was true when written and is not now. It said the behavioural arms “need pytest, psycopg and a virtualenv; CI installs none of them”. CI has a pytest-guards job that installs pytest pinned from requirements-test.txt, asserts psycopg is absent, and runs the database-free file list – which contains test_layer.py. A stale justification for keeping coverage in the wrong place is harder to find than a missing check, because nothing reddens.

    Two instrument defects of my own, both caught by the new arms' own premises. The shape parse took the FIRST line matching the print marker, which in pgc_vacuity.py is the DOCSTRING that spells the shape out for a reader; requiring a quote before the marker selects the code in both languages. And the drifted-shape fixture wrote one space where the real shape has two, so the parse found nothing and the comparison was empty-against-real – which “differs”, for the wrong reason. Its premise arm said so.

    test/selftest/380 keeps finding 1 and hands finding 2 to the pytest corpus. The part covers two of @linuxhikerpm’s #897 findings, and they have different subjects. Finding 1 is test/pgc_fingerprint.py, which lib.sh runs with the system interpreter and which is not part of the pytest harness, so checking it is this part’s own business and it stays. Finding 2 is test/pytest/pgc_cluster.py, and the nine arms that read it as text are gone.

    Seven of its twenty-one checks had a cross-harness subject, not twenty-one: the other fourteen read pgc_fingerprint.py, lib.sh, or fixtures the part writes itself. Counting the whole file would have deleted coverage that was never debt.

    make_cluster’s cleanup SHAPE moved to test_build_refusal.py as test_the_cleanup_guard_has_the_shape_the_leak_needs, beside the behavioural arm that provokes a real failed setup. It has to be a source check, and it says so: make_cluster fails exactly one way in the behavioural arm – a missing pg_config – while three more properties decide whether the guard works, and two of those cannot be provoked at all. You cannot deliver SIGINT into initdb reliably, and a cluster that started is one the arm would then have to stop.

    Five mutations say the moved arms discriminate, each asserted to have applied, each leaving the module well-formed, restored byte-for-byte afterwards: BaseException narrowed to Exception, cluster.stop() removed, the bare re-raise turned into pass, a private hashlib.md5 added, and shutil.rmtree(root, …) removed. The two whose effect reaches the filesystem redden the behavioural arm as well; the three that redden only the source arm are exactly the properties the behavioural arm cannot see.

    One of those five did not apply on the first attempt – the cluster.stop() pattern assumed twelve spaces of indentation and the call sits at sixteen, inside a nested try. The harness refused to report a result for it rather than printing a green, which is the only safe behaviour for a mutation that did not land.

    And one thing went wrong that is worth more than the change itself. I ran the selftest before naming the two new pytest arms in TESTS.md, so the run was red on the doc-coverage check – and then merged that log into check_ledger.tsv, whose entire subject is which checks have ever been red. It recorded a red for 350’s “every test file and every test in the corpus is named in TESTS.md”, a check that failed only because my own change was half-finished. Reverted with git checkout HEAD --; there is no second copy to repair from.

    The merge step now refuses a log that is not definitely green: a numeric checks run: line, a floor on it so an aborted run cannot pass, zero FAIL lines, and rc=0. The absence of a FAIL line is not enough, because an aborted run has none either – the same shape as a pending-count that cannot see a job

    test/selftest/350 goes from 50 checks to 5, and two of its rules moved rather than being deleted. Forty-six of the fifty had a subject on the other side of the boundary: they globbed test/pytest/*.py and parsed Python out of it, or swept the markdown in that directory. The pytest corpus asserts all three properties natively in test_docs_cover_the_corpus.py, where the subject is, so the shell copy could only ever agree with it.

    Two of the three rules were implemented on BOTH sides and self-tested only on the shell side – the side that cannot run the corpus it counts. Deleting those fixtures would have left the Python implementation with no fixtures at all, so they went with them:

    • the mode-counting rule’s edges: an id of fewer than three words, an id named twice, stopping at the next heading, section 3’s back-references, and the row reader taking the value cell rather than a digit inside its label
    • the contents-list anchor rule: GitHub’s derivation, the broken link that shipped, and a control beside it

    _named_modes_in(text) and _stated_row(text, label) are new seams, so a fixture can reach the rules at all. The row reader’s fixture makes the label digit and the value DIFFER – label “section 2”, value 9 – because 350’s own fixture had both as 2 and could not see the bug it was written for.

    What stays in 350 is its arms over ci.yml, whose subject is neither harness. A shell part may read the workflow for the same reason it may read the Makefile, and no pytest arm can assert that the gate runs pytest without assuming the thing in question. One arm is added while the part’s subject is being settled: the job must ASSERT psycopg is absent rather than assume it.

    The stale justification, in triplicate

    TESTS.md section 6 said “Nothing runs pytest. Not run_all_versions.sh, not any workflow under .github/”, and concluded that the .sh half was therefore the enforcement. CI has a pytest-guards job: it installs pytest pinned from requirements-test.txt, asserts psycopg is absent, derives the file list from NO_CLUSTER, and runs it.

    That claim was load-bearing in THREE places at once – TESTS.md section 6, selftest/360 and selftest/380 – each asserting the behavioural half could not run in CI. Nothing was wrong in any of them; the reason was, three times, and no arm reddens on a stale justification. All three are corrected.

    Where the two harnesses now stand

    Shell parts holding a reference to anything under test/pytest/: one, and it is deliberate. selftest/360 reads pgc_vacuity.py to compare the three values the two harnesses both write down – the INCOMPLETE exit code, the closed list of unrunnable reasons, and the line an unrunnable check prints. That is CONTEXT.md’s permitted cross-reference: a property that IS the relationship, so it cannot be stated from one side. The inventory said seven; the measured answer was three plus a deleted fourth; the end state is one. which never started.

  • One matrix report no longer gives two answers to “how many suites accounted for their checks” (#928).

    A full PG 17 report printed both, three lines apart:

    population reconciliation: registered=251 | accounted=237, ... | sum=251
of those, 235 accounted for their checks and 7 did not

    237 and 235, both describing suites that accounted for their checks, differing by exactly 2. The population line counted with the WIDE reader, pgc_log_shows_any_accounting; the breakdown line derived from the NARROW one, pgc_log_shows_accounting. The figure a reader acts on is the second, because it is the one phrased as a problem, and it overstated the debt by 2.

    That matters more than a mismatch: the breakdown line exists to stop an overcount, and deriving it from the narrower reader reintroduced a smaller version of the same overcount in the line added to close it.

    THE GAP IS A DIFFERENT MECHANISM, NOT A DEBT. The wide reader also accepts a suite that prints its own checks run: line. Measured on this tree: of the twelve registered suites that never call pgc_summary, exactly two – bench_guards and docs_style – emit a tally of their own, and the other ten keep none. Those two are the 2.

    The headline now comes from the same file the population line counts, and the two mechanisms are broken out beneath it with the own-mechanism suites NAMED rather than counted, so a third adopting its own tally appears without anyone editing a number.

    The comment above the line said “Ten registered suites exit 0 having never called pgc_summary”, which conflated the two populations: twelve never call it, and ten of those keep no tally. Both halves were true of something; neither was true of what it said.

  • A test helper no longer takes a tree it ignores (#933).

    _sh(srcdir, expr) in test/pytest/test_build_refusal.py read as “evaluate one lib.sh expression against a tree”: the docstring said so, nine call sites passed a fixture tree, and the body sourced the module-global SRCDIR – the real source tree – instead. Whatever those arms measured, it was not parameterised by the tree they were handed.

    WHICH READING WAS INTENDED IS A MEASUREMENT. Every caller passes a tree built by _tree_with_module or _tree_with_source, and none contains test/lib.sh:

    fixture tree holds: ['Makefile', 'objstore', 'pgcolumnar.control']
honouring srcdir:   rc=1, "No such file or directory" -- the source fails
sourcing SRCDIR:    rc=0, the function under test runs

    So the parameter could never have worked – honouring it would have made every one of those arms measure a failed source rather than the function under test. The arms mean the real tree, so the parameter was noise that made nine call sites read as something they were not. It is gone, and behaviour is unchanged: the expressions that need the fixture interpolate it themselves.

    A corpus-wide AST scan now requires that no helper takes a parameter it never reads, so the class is closed rather than the instance. Two exclusions, both real: a TEST function’s parameters are pytest fixtures, and requesting one has an effect whether or not the body reads it; and a HOOK’s signature is pytest’s API, where arguments arrive by name, so declaring one you do not read is how a hook says which it wants. Three of the four the scan found were hooks – pytest_collection_modifyitems(config), pytest_xdist_node_collection_finished(node), pytest_sessionfinish(exitstatus) – so only _sh was a defect and the budget was 1, now 0.

  • The vacuity guard’s PLACEMENT is now a checked property, because a guard in a teardown cannot fail the test it guards (#432).

    Measured, the same AssertionError raised from two places: from a pytest_runtest_call wrapper the run reports 1 failed; from a fixture teardown it reports 1 passed, 1 error. pytest has already recorded the call phase as passed, so a teardown refusal arrives as a separate error on the same node-id and the test’s own outcome stays passed. Anything counting passes – --pgc-expect-tests, a CI summary, a human reading “N passed” – sees a pass.

    This layer’s guard was already in the call-phase wrapper, so nothing was broken. What was missing is that nothing said so: moving it into the expect fixture’s teardown was a plausible-looking refactor that would have turned every vacuous test from failed into passed with an error beside it. Two arms in test_runshape.py pin the placement, and the second is the control – without it the first passes whatever phase the guard is in, because “a vacuous test fails” is equally true of a correct guard and of no guard at all next to an unrelated failure. Proved by moving the guard into the teardown: the first arm reddens.

    guard-as-teardown-fixture-still-reports-passed is NARROWED rather than closed, and VACUITY_MODES.md 3.7 says which half. The half closed is this layer’s own guard placement. The half still open is the general shape: a guard anyone adds later in a teardown still cannot fail its test, and nothing refuses that.

  • A failed query is no longer comparable with another failed query (#432).

    error-swallowed-to-empty: two queries raise, a helper turns each into the same value, and they compare equal. The test is green and has asserted nothing about either query. test/lib.sh closed this by PRODUCING the sentinel with a sequence number per failure – res="QUERY_ERROR.$seq" – so two failures can never compare equal.

    The pytest port had the constant QUERY_ERROR = "QUERY_ERROR", a comment claiming it was “unique per occurrence” – which is false of a constant – and a refusal in exactly one assertion. Measured before the fix, with a sentinel on both sides: expect.hash refused; expect.text, expect.rows, expect.row_set and expect.ordered_rows all PASSED. Four of the five comparisons accepted two failed queries as agreement. expect.num, expect.at_least and expect.rowcount refused already, by their type guards rather than by anything about sentinels.

    There is now one refusal, called by every comparison, so an assertion added later inherits it instead of being the next hole – which is how this survived: hash had a refusal and the four written after it did not. It matches the prefix at any depth, because a sentinel arrives as a CELL inside a row as often as it arrives as a whole side. row_set refuses BEFORE it maps its rows through repr, since repr(("QUERY_ERROR.1",)) does not start with the prefix: delegating an assertion does not delegate its refusals when the delegation transforms the data.

    query_error() produces a value unique per occurrence, for the paths that compare without the layer – a helper comparing by hand, which is what the two existing hand-rolled sentinels in test_hilbert_locality.py do. The refusal is the mechanism; the producer is the second line. One of those two sentinels is produced by coalesce(...) inside SQL and cannot use the Python producer, which is why the refusal has to be the mechanism rather than the other way round.

    Each refusal is proved load-bearing: removing it from one assertion at a time makes the arm name that assertion and no other, five times out of five. The mutations are only distinguishable with __pycache__ cleared between runs – the five deleted lines are byte-identical, so three of the five leave the file the same size and Python reuses the stale bytecode, which made two of the refusals look like they did not bite.

    THREE DEFECTS @jdatcmd FOUND IN THE FIRST VERSION, each reproduced before it was fixed.

    The HATCH WAS OPEN and the arm that said otherwise was tautological: it rewrote pgc_vacuity.QUERY_ERROR and THEN minted its sentinels with query_error(), which read that same global – producer and matcher moved together, so the refusal matched whatever the prefix had just been set to. The faithful hatch mints while armed and rewrites afterwards, which is what a corpus file does, and against the first version that COMPARED two sentinels instead of refusing them. End to end it reported 1 passed over two failed queries. The prefix is now bound in a DEFAULT ARGUMENT, evaluated once when the function is defined and never read from the module namespace again, so rewriting the global changes neither what is minted nor what is refused. ZZZ_NOT_A_PREFIX is in the arm’s spellings deliberately: 'Q' cannot disarm a prefix-reading matcher, because 'QUERY_ERROR.1'.startswith('Q') is true.

    ordering_observable WAS MADE WEAKER BY THE PRODUCER, and this is the one place the change regressed the layer. It takes (forward, reverse) rather than (got, want), so it sat outside the refusal and outside the derivation that finds comparisons. With the old shared constant two failed readings were IDENTICAL and it went red – loudly. With unique sentinels they differ, so it passed and greenlit every ordered assertion resting on the premise. The refusal is now the first thing it does, the derivation recognises (forward, reverse), and an arm pins both.

    And TESTS.md stated the split twice in one sentence while only the first half was gated: 26 refused and “the other 47” sums to 73 against the 72 the inventory names. selftest/350’s regex matches the half a change naturally updates. Both halves are now read by an arm in the corpus’s own docs guard, against the inventory’s count of the ids it names rather than a number typed twice.

    VACUITY_MODES.md said “the port has the sentinel constant but nothing produces it”, and that was wrong in both halves: two sites did produce sentinels by hand, and the thing actually missing was the refusal in four of the five comparisons. The correction is recorded in the document rather than quietly replacing the sentence, because a map that names the wrong gap is worse than one that admits it does not know.

  • The sweep that forbids piping a captured string into grep -q now joins a pipeline split across two lines, and the six suites that had split one are fixed (#486).

    The sweep read one physical line at a time. producer | on one line with grep -q PATTERN on the next was therefore invisible to it: the producer’s line holds no reader, and the reader’s line holds no producer. Six live sites were written that way – three in test/vector_agg_rescan_memory.sh, one each in test/unique_conc.sh, test/native_groupagg_batch.sh and bench/run_clickbench.sh. Every one of them answers a premise that decides whether a whole arm measures what it claims to, and the failure direction is the expensive one: the pipeline reports the thing it was looking for as ABSENT, so a plan that contains the vectorized aggregate reads as a planner regression.

    test/unique_conc.sh is the one to read. The comment directly above it explains this exact trap, and captures the output into a variable for that reason. The next line pipes that variable into grep -q anyway.

    The sweep now builds logical lines before it matches, and applies one pattern to both the physical and the joined stream. Three behaviours of bash were measured rather than assumed: a pending | skips blank and comment lines, a \ joins the next physical line with no skipping, and a comment never continues at all. Two premises the joiner rests on are asserted instead of coded around – no line opens a heredoc and also continues, and no \ continuation is followed by a blank or a comment – so if either stops being true the gate says so rather than reading past it.

    One false positive is accepted, and a fixture pins it: a double-quoted string continued across a line break, whose first line ends in a bare |, reads as a pipeline once the two lines are joined. It fails loud, where the blindness it replaces failed silent. A second latent defect went with it – the physical stream now passes -H, because grep -n omits the filename when it reads a single file and the heredoc exemption keys on file:line.

    A COMMENT NAMING A HEREDOC USED TO EXEMPT THE REST OF THE FILE. The exemption scanner matched the opener anywhere on a line and left heredoc mode only on a line equal to the tag, so a COMMENT that merely named the idiom switched the rule off for everything after it. @linuxhikerpm measured it: with a genuine two-line violation restored this part went red, and adding one comment line 24 lines above it – changing nothing else – took it back to 37 passed while the violation was still there byte for byte. This change is where that becomes load-bearing, because it deletes the filename exclusion and rests the argument on the exemption being DERIVED rather than listed.

    Two conditions now, each measured. An opener is recognised only on a NON-COMMENT line, and only when a later line EQUALS its tag – one with no terminator exempts nothing. The second condition is what stops a TRAILING comment doing the same thing, and it retires the old per-file reset: an unterminated candidate can no longer leak into the next file. Proved by restoring the violation, adding the comment, and staying red; and by writing cat <<'X' | plainly in this file’s own prose, which used to take its exempt-line count from 6 to 187 and now changes nothing.

    AND BOTH NEW PREMISE ARMS WERE NUMERATOR-ONLY. They reported zero whether or not their detector worked: replacing the heredoc-opener pattern with one that cannot match left the arm green, and so did making the continuation detector never arm. The denominators are printed and asserted now – 179 openers and 5,533 continuations – which is the inputs == sum(buckets) rule the rest of this directory applies. An earlier draft of this entry claimed “if either stops being true the gate says so”; that was the half which was not true.

    Planting any one of the six sites back in its old form takes the rule red and names the file and the line.

  • TESTS.md’s contents list no longer carries a link that goes nowhere, and the corpus gate now checks every one of them.

    The entry added for test_harness_deps.py stripped the underscores out of the file name – #14-testharnessdepspy-... against a heading GitHub renders as #14-test_harness_depspy-... – so the link silently resolved to nothing. The eleven entries above it keep the underscores, so the document already stated the convention. Neither existing arm could see it: both sweep for NAMES, and a broken anchor is still a string containing the name it points at. Selftest 350 now derives each heading’s anchor by GitHub’s rule and requires every in-document link to reach one. Measured over the three documents in that directory it reports nothing, and over the document as it shipped it reported exactly the one entry.

  • No sentence in the pytest harness states how many tests the corpus holds (#908).

    Eight places said “61 of 142” or “152 tests”: test_harness_deps.py twice, conftest.py, TESTS.md three times, the pytest-guards job’s comment in ci.yml, and selftest 350’s own comment. The corpus held 154 on the day they were written, so the job’s comment was already wrong, and a concurrent branch adds a thirteenth test file, which would have made every one of them wrong again. These are the hand-maintained derived values #908 spent a day removing from TESTS.md’s totals line, reintroduced as prose, where that line’s guard cannot see them: it matches only the bold fixed-form line. The numbers are gone. The job prints how many files it ran and pytest prints how many tests passed, the membership arm prints the partition, and an arm over the job and its comment block refuses a written count there.

  • ALTER TABLE ... RENAME COLUMN now carries the new name into pgcolumnar.projection_declaration, for the named relation and for every inheritance descendant, including a PARTITION OF child (#888).

    The materialized projection stores attnums, so it already followed a rename without any catalog change. The declaration deliberately stores NAMES, because a restore assigns new attnums – so leaving the old name behind broke pgcolumnar.rebuild_projections() after a dump and restore, even though the live projection had kept working right up to the backup. The failure was therefore invisible until the moment it mattered.

    The descendant half is held by an arm that was proved able to fail: changing the walk to use the named relation instead of each descendant takes test/projection_rename_restore.sh from 8 passed to 6 passed and 2 failed.

  • An in-place TRUNCATE now clears each projection’s storage as well as the base (#896).

    PostgreSQL truncates a relation created in the current transaction in place, because a rollback discards the relation anyway. Measured on 18.4: the relfilenode and the storage id are both unchanged across such a TRUNCATE, where a table from an earlier transaction gets fresh ones. So relation_set_new_filelocator never runs, and the teardown that retires a projection’s storage never happens.

    pgcolumnar_relation_nontransactional_truncate cleared only the base storage. The projection’s row groups survived, and the next write to the projection collided with them:

    ERROR:  duplicate key value violates unique constraint "row_group_pkey"
DETAIL:  Key (storage_id, group_number)=(10000000001, 2) already exists.

    The whole transaction rolled back, so TRUNCATE and reload in one transaction was unavailable on any table carrying a projection. It needed four things together: the table created in the same transaction, a projection, the TRUNCATE, and a write after it. Each was isolated by a control.

    The pgcolumnar.projection rows are kept, deliberately. This is not pgcolumnar_delete_storage_tree, which also deletes them. That is right for a rewrite, where the base storage id changes and the rows are re-recorded under the new one. Here the metapage keeps its storage id, so the rows still describe this relation correctly; only their content is being truncated away.

  • ALTER TABLE ... DROP COLUMN is now refused when a projection depends on the column, instead of leaving the table unreadable (#891).

    Dropping a column any projection stores left the table broken, in one of two ways. Dropping the sort-key column produced ERROR: type with OID 0 does not exist on the next INSERT. Dropping any other stored column let writes continue while pgcolumnar.read_projection raised cache lookup failed for type 0 and pgcolumnar.rebuild_projections() reported repairing nothing – the quieter and worse half, because it tells an operator there was nothing to do.

    The refusal raises 2BP01 (dependent_objects_still_exist) and names the projection, so the remedy is to drop the projection first. One loop covers sort keys too, because add_projection() requires every sort-key column to appear in the stored columns. DROP COLUMN IF EXISTS of a column that is not there is unaffected.

  • A rewrite no longer loses a declared projection (#876, #887).

    TRUNCATE, ALTER TABLE ... ALTER COLUMN ... TYPE and ALTER TABLE ... ADD COLUMN with a volatile default each mint a new base storage id. pgcolumnar.projection is keyed by that id, so afterwards pgcolumnar.read_projection raised 42704 for a projection that was still declared over an intact table. 1.0-alpha3 shipped only a HINT naming pgcolumnar.rebuild_projections(); the projections are now re-recorded automatically and the manual call is no longer part of the routine path.

    Five shapes lost the projection, not the two the issue named. Swept on 18.4 rather than reasoned about: TRUNCATE including its multi-table form and its CASCADE form, a type change on a covered or an uncovered column, ADD COLUMN with a volatile default, a partitioned child rewritten by a type change on its parent – where the statement names the parent, which is not itself a columnar relation – and REFRESH MATERIALIZED VIEW, which is neither an AlterTableStmt nor a TruncateStmt and so escaped a gate naming only those two. REFRESH ... CONCURRENTLY is not a rewrite: the storage id is unchanged across it, measured. ADD COLUMN with a constant default, DROP COLUMN, VACUUM, SET ACCESS METHOD to the same method, SET TABLESPACE and a no-op type change do not rewrite and were never affected. Core VACUUM FULL and CLUSTER are refused on a columnar table, which bounds the class.

    The repair runs after the statement, in ProcessUtility, not in the table-AM callback. pgcolumnar_relation_set_new_filelocator cannot do this job, which is measurable rather than arguable: with the callback logging its own relid, TRUNCATE reaches it as the user’s relation with the old fork attached and both projection rows in scope, but a rewriting ALTER TABLE reaches it as the transient relation make_new_heap builds – pg_temp_<oid>, no columnar fork – so the rewrite branch is not taken and neither the old storage id nor the projection list is ever in scope. A re-record placed there also records under the id the rewrite just retired, because PgColumnarStorageId(rel) still returns the old id after the new metapage is written.

    The projections are re-derived from the declaration, not copied forward. pgcolumnar.projection_declaration records column NAMES and survives a rewrite, and resolving those names against the relation as it is now is what makes ADD COLUMN correct: the base projection records every live column, so a copy of the old row would leave it naming a stale column set. materialize_projection is extracted from pgcolumnar.add_projection so both paths drive one implementation.

    A repair that cannot run degrades to a WARNING and never fails the statement that triggered it. A declaration can name a column the table no longer has; ALTER TABLE ... RENAME COLUMN used to leave one behind and no longer does, so the reachable case is a database created before that fix. Before this was handled, the repair raised column "a" does not exist inside an unrelated ALTER TABLE ... ALTER COLUMN id TYPE bigint and rolled that type change back – turning a silently lost projection into a blocked schema change. It now reports

    WARNING:  42703: could not restore projection "p" on "public.t" after rewrite
DETAIL:   Its declaration names a column the table no longer has.
HINT:     Call pgcolumnar.add_projection('public.t', 'p', ...) again,
          naming columns the table has. That replaces the declaration.

    The HINT names add_projection, not rebuild_projections, because the other two candidates were measured to fail. rebuild_projections() re-runs the same stale declaration and raises the same missing-column error; drop_projection() refuses with 42704, because the projection row is exactly what is absent. add_projection() with the same name replaces the declaration and materialises it. The relation name in both messages is schema-qualified: unqualified, a HINT for a table outside the reader’s search_path told them to run a statement that fails with relation "t" does not exist.

    Every other failure is contained too. The resolves-check covers one cause, and materialize_projection can raise for others, so it runs in an internal subtransaction. A failure is rolled back and reported as a WARNING carrying the original SQLSTATE, and the user’s statement continues.

    The repair decides under AccessShareLock and escalates only when there is work. It is reached for every AlterTableStmt on a relation with a declared projection, not only for one that rewrote it, so an unconditional ShareLock blocked concurrent writers on statements that rewrite nothing: ALTER COLUMN SET STATISTICS and SET (autovacuum_enabled) take only ShareUpdateExclusiveLock, and each opened the relation with ShareLock with nothing to repair.

    The rewritten relations are recorded, not re-derived from the statement. A TRUNCATE ... CASCADE rewrites tables it never names, so the table-AM callback records each relation whose storage it retires and ProcessUtility drains that list. The list survives a nested utility statement – a cascade whose trigger runs one – because it is cleared only for the outermost statement, and the drain removes only the relids it repaired rather than emptying it.

  • The 42704 hint no longer names a rewrite as the likely cause, since a rewrite now re-records. It names the two cases that remain: a declaration that no longer resolves, and the implicit base projection, which is not readable by name at all.

  • A conftest.py can no longer switch off the order-collapse guard by rebinding the module-level name it used to read (#924).

    The scan looked up _ORDER_KILLERS on each call. A conftest is imported before collection, so pgc_vacuity._ORDER_KILLERS = () turned the refusal off for every test in that directory, with no reason recorded. Two lines, less to type than the honest form. Measured: the same collapse test was uncollectable with no extra file, and reported 1 passed with only that rebind.

    The killer names are bound at definition time, in a default argument, the same way query_error already binds its prefix. There is no module-level name left to rebind. _RECORDERS is a registry the layer writes, not rule data, and is unchanged.

  • The pytest harness no longer reports results against a library another process installed (#956).

    build_once() skipped the build when its marker matched, and the marker recorded the pg_config, the major and the source fingerprint. That answers “did this layer last build this source”, and it was read as “does the prefix hold that build”. The two differ whenever anything else writes the shared prefix: the bash harness, a timing run, a manual install, another worktree. Measured twice in one day, a measurement run installed an older library and the corpus then reported ten failures in one file on one machine and nineteen on another, with the code under test entirely innocent.

    The installed library is now part of the marker, so a prefix someone else wrote is rebuilt rather than certified. A library that is absent counts as changed. Where the prefix cannot be observed at all, no marker is written, so the next call builds rather than matching another unobservable run. The arms that exercise the marker supply a pg_config that answers, so whether the skip happens no longer depends on whether the machine running the tests has one.

    The digest cannot be predicted from the source, because the build path is compiled in: one commit built in two directories produces two different libraries. So what is recorded is the digest installed at the moment the marker was written, which is a statement about that prefix over time.

    Blast radius worth knowing, since it is what made this hard to spot: a stale library fails exactly the tests of the feature it lacks, so it presents as one whole file failing while the rest of the suite passes. Scattered failures are usually the code; a clean file boundary is usually the environment.

  • pgc_ledger.py gate no longer certifies a census that contradicts its own ledger (#952).

    The gate printed ledger census: rows=N and never compared that number to the checks_never_observed_red the budget states, so it returned 0 on a twenty-row ledger claiming five. Reporting is not enforcing. The comparison existed one layer out, in a selftest arm, which runs on a pull request and therefore reports the disagreement after the merge that creates it rather than before.

    It creates it because the census is a measurement of the tree, so every merge invalidates it: two pull requests each re-derive it from the same base, the merged ledger takes both sets of rows, and the budget keeps whichever side won the conflict. Three in flight at once set 769, 762 and 800 against a base of 756, and no two composed. The new refusal is decidable from the two inputs alone, needing no prior and no --against, which is what lets it speak about a merge commit.

    It refuses a contradiction in either direction and does not make the census a ceiling. A ceiling refuses a rise, and bounding this number deadlocks: every added check enters as never, so landing one would mean raising a number the design says may only fall. A budget that states no census at all is reported rather than refused, since absence is not a contradiction; what holds the committed budget to naming both numbers is a separate arm in each harness.

  • The harness no longer reports that the installed library matches when it has not looked at the library (#959).

    lib.sh compared a recorded source fingerprint against the current one and then printed “source

    Measured on two trees whose sources differ by five files. One built and installed through the harness, the other installed its own library into the same prefix, and the first then ran a suite with PGC_SKIP_BUILD=1: the run printed the library’s fingerprint, asserted that the source matched the binary, and failed nine checks of a feature the installed library did not contain.

    The stamp now records the installed library’s digest beside the source fingerprint, and the claim requires both to match what is on disk. A library that changed under the stamp is refused the way a changed source already was, naming both digests and the prefix another build wrote. A stamp written before this change records no digest, so it reports the source claim it earned and says the library is unverified rather than implying it was checked.

    The digest cannot be predicted from the source, because the build path is compiled in: one commit built in two directories produces two different libraries. So what is recorded is the digest installed at the moment the stamp was written.

    All three places that write a stamp record it: the build function, the matrix runner and the development loop. Without that the matrix, which builds once per major and then sets PGC_SKIP_BUILD, would have reported every suite as unverified.

    The reader for the source field now reads its first line only. Stripping hex from the whole file was right while a stamp was one line, and wrong as soon as there were two: a source that cannot be fingerprinted writes an empty first line, and the whole-file read returned the library digest as the source, turning a documented unverified into a refusal that named a library digest as a source fingerprint. That was introduced by the second line rather than found lying in wait, so it is fixed here. Reported by Joshua D. Drake, who swept every hex extraction in the harness to establish it was the only one.

    The refusal on a changed source had never been exercised by anyone before this change, only read. It is now driven end to end, along with the three other states.

  • The ten suites that recorded nothing now record their checks (#965).

    Ten registered suites print their own PASS <name>: <value> lines and their own verdict and emit no machine-readable records at all. Counted across the ten, 293 checks were invisible to every mechanism built on that vocabulary: the ledger, the census, the count of checks never observed red, the red-observation record, and the duplicate-name detection. They pass, and nothing that reads records can see them.

    This converts the first of the ten. Its nine checks now emit a record each and a total, measured on PG18: nine records where there were none, and checks run: 9 where there was no total.

    The human output is unchanged, byte for byte. pgc_record takes the display whole, so each line still reads PASS count(*): 100000 with the measured value rather than a label. That value is the comparison, not a message, which is why the local helper records through pgc_record rather than delegating to lib.sh’s own check – the latter composes its own display and would drop the value.

    The suite’s own verdict line stays, and so does its exit logic. That line is load-bearing until a reconciliation replaces it: smoke.sh runs under set -euo pipefail, so a failing command aborts it, and the verdict is what distinguishes a suite that finished from one that stopped. Removing it in the same change would make the suite report less than it did before.

    This converts nine more of the ten, the same way, after the shape was reviewed on the first. Measured on PG18: 227 records where there were none, and every suite’s human output byte-for-byte unchanged.

    It also fixes a call that was failing silently. unique_conc.sh calls check_skip for the case where the citext extension is absent, and that function lives in lib.sh, which the suite did not source – so the baseline log carries line 392: check_skip: command not found, the suite continued under set -uo pipefail, and the case was reported nowhere at all. It now emits a SKIP record with its reason.

    Fifty-eight checks across four of the suites are still not recorded – fifty-eight on PG18 and fifty-seven on PG16, because one of them sits behind a version gate – since those suites have further check-like helpers of their own, eq_on_off in phase6 alone accounting for thirty-nine, each printing its own display. Those need a second pass rather than the same substitution, and the count in #965 should be read as the number of checks rather than the number of helpers.

    It does not add the suite to the ledger. Making a suite coverable and covering it are separate decisions, and the second one is blocked on a measurement: check NAMES differ between majors in at least one suite, the ledger has no major dimension, and a gate seeded from one major would refuse runs on another.

  • The last fifty-eight checks in those ten suites record too, so all ten are now complete (#965).

    The previous change converted each suite’s own check helper and recorded 236 of the 293. The rest went through four further helpers with four different displays, which is why they needed a second pass rather than the same substitution:

    suite helper checks
    phase6 eq_on_off 39
    phase4 expect_fail 5, assert_plan 2, assert_plan_seq 1, one written inline 9
    audit expect_error 5
    phase5 assert_plan 5

    Measured on PG18, each suite’s records now equal both its own human check lines and its checks run: total, and every human line is byte-for-byte what it was:

    suite    records before -> after   checks run:   human lines
audit                26 ->  31             31            31
phase4               29 ->  38             38            38
phase5               31 ->  36             36            36
phase6                4 ->  43             43            43

    phase6’s eq_on_off has three outcomes and two of them return early. Each one records, because a return that skips the record leaves the check counted nowhere, which is the state this conversion exists to end.

    Two displays span more than one line – assert_plan in both phase4 and phase5 prints the whole plan under a header when it fails. The dump is passed as part of the display rather than echoed after the record, so a failing run’s output is also byte-identical instead of having a record line wedged between the header and the plan.

    It also makes a version-gated arm visible to the ledger. audit.sh gates its partitioned-parent arm on server_version_num >= 170000, because PG16 and earlier refuse PARTITION BY ... USING pgcolumnar, and the gated branch printed a bare note and recorded nothing. So on PG16 the ledger received three fewer rows for audit with nothing saying why. It now records a SKIP with its reason, the way unique_conc.sh already does for its own version gate, and PG16’s human output gains that SKIP line in place of the note.

    One SKIP for the block, not one per gated check. Naming each of the four checks in a branch that never runs them would make the count the same on every major, and would also put four check names somewhere nothing exercises them, where they would drift. Comparing counts across majors needs a major dimension in the ledger, which belongs to #432.

    Every count here is a PG18 number. On PG16 the same suites give 28 records for audit rather than 31, because that gated arm holds three check calls and the expect_error above, and one of those records is now the SKIP standing in for all four. The per-major table is in #965, which is where the remainder should be read from rather than from either run alone.

  • The unprivileged reader in the fingerprint arms needs a harness it can reach, and a failed source must not be reported as a refused fingerprint (#907).

    The nightly coverage job went red for two nights and nothing said so: every suite job was green, the PR gate was green, and the only failure was one suite inside the job nobody reads. harness_selftest reported 249 passed, 1 failed.

    Three checks in 340-the-binary-must-be-built-from failed, with six test/lib.sh: Permission denied lines beside them. The arms there read a fixture tree as a second user, because chmod 000 is invisible to root, and they sourced lib.sh from the checkout. In GitHub Actions the checkout lives under /home/runner/work, which postgres and nobody cannot traverse, so the unprivileged shell could not load the harness at all.

    Two of the arms passed anyway, and that is the defect worth naming. The reader did . "$PGC_TESTDIR/lib.sh" || exit 1, so a denied source produced empty output – which is exactly what an arm asserting “an unreadable file yields no fingerprint” wants to see. Measured under a tree the reader cannot traverse:

    everything readable, lib.sh UNREACHABLE   -> []              the arms PASS on this
b.c unreadable,      lib.sh reachable     -> []              what they mean to test
everything readable, lib.sh reachable     -> cde49bff94a7

    So the premises were the only thing standing between the suite and a clean report on two arms that proved nothing.

    Three changes, each with its own reason. A readable copy of lib.sh, portlib.sh and pgc_fingerprint.py is staged beside the fixture, which the reader can always reach – the same three files, read from a different directory, which is an environment property rather than anything the part asserts. A failed source now answers harness-unreadable, a value that cannot be mistaken for a hash or for empty, so no arm can pass that way again. And the arms run only when the premises they rest on were met, skipping loudly otherwise, because a guard that cannot run is not a guard that held.

    Measured, three conditions, PG18:

    tree readable by the reader            unfixed 803 checks 0 FAIL   fixed 804 checks 0 FAIL
tree under a mode-750 parent           unfixed 3 FAIL, 6 denied,   fixed 0 FAIL, 0 denied,
                                               2 arms PASS vacuously      arms run
reader cannot fingerprint at all       --                          2 FAIL, 5 honest SKIP

    The last row is a probe rather than a condition anyone meets today: python3 was removed from the reader’s PATH to prove the skip branch can be reached, since an else that cannot run proves nothing either.

  • The per-file coverage table is computed from the tracefile, so it no longer ranks the best-covered files as the worst (#974).

    The nightly’s “least covered first” table printed

    columnar_parquet_codec.c  | 2.0%    100|3200%     2|    -      0

    for a file whose records say LF:100 LH:100 FNF:2 FNH:2 BRF:64 BRH:47. A file at 100% presented as 2.0% and sorted to the top of a list headed least covered, with a function rate above 100% on its face and an empty branch column where the tracefile carries 13,900 branches. Four of the files it named as least covered were between 94% and 100%. A coverage table is read to decide where to spend effort, and this one inverted the ranking.

    The table came from lcov --list. The run’s own lcov --summary, four lines earlier and on the same tracefile, was correct, and so was lcov --list run here on the run’s own uploaded tracefile. The two builds differ only in distro patch level – the runner installs 2.0-4ubuntu2, this was checked on 2.0-1 – and which patch does it has not been bisected.

    So the table is computed instead. A rate is a division of two integers the tracefile states outright, and LF: cannot be got wrong by a patch to --list. Cross-checked against lcov --list on all 39 files in that tracefile and independently against the raw counters: no disagreement in either comparison. lcov --summary keeps its job.

    The format now carries hit/found rather than only the total, because 93.7% 22765 cannot be checked by a reader and 93.7% 21320/22765 can – and a reader who can check the number is the only one who will notice when it is wrong again.

    An absent counter prints -, not 0.0%. A header with no branches at 0.0% would sort to the top and read as the least covered file in the tree, which is how a table misleads while every individual number in it is defensible.

    Twelve arms in 250-the-coverage-runner-must-refuse drive the generator over a synthetic tracefile rather than grepping it, because a static check that the runner calls the right script cannot tell whether the script is correct, and “the table is wrong” was the defect. Three of them failed when first written: two matched the comments in the runner that explain the replaced call, and the third asserted a cell by the wrong field number. They are counted over code with comments stripped now, under a premise that stripping comments did not strip the code.

    A fourth failed on a substring: 0\.0% matches inside 50.0%, which is the line rate of the fixture that has no branches. It is anchored, and a control with a genuinely zero-covered file proves the anchor did not defeat the assertion.

  • The matrix controller is asserted to record the installed library, because the failure if it stops is a green matrix-wide downgrade (#961).

    run_all_versions.sh builds once per major and runs every child suite with PGC_SKIP_BUILD=1, so each child checks the controller’s stamp to learn whether the binary it measures came from this tree. If the controller’s stamp write loses its third argument, every child reaches source-only and passes – because source-only is also the state of every stamp written before #959, so it cannot be a failure. The whole matrix degrades to UNVERIFIED with a green rollup on both majors, and nothing says so.

    @jdatcmd asked for this arm while approving #960, on the ground that an arm catching a green failure is worth more than most arms.

    The defect is a dropped argument at a call site, so an arm that calls pgc_write_source_stamp itself would prove nothing: the function would be correct and the caller wrong. The new part extracts the controller’s stamp block and runs it, so what executes is the real call site’s own text.

    It costs no build. The block reads the installed library and fingerprints a tree; it does not compile. So it runs against a copy of the tree with builddir and pgc set the way the controller sets them – 24 MB at 31 ms a copy, against minutes for a per-major build.

    Measured by mutating run_all_versions.sh itself, dropping the third argument in a way that still parses:

    the controller's stamp carries BOTH fields          got [1] want [2]
a child reaches verified, not source-only           got [source-only] want [verified]
every caller records the installed library's digest got [2] want [3]

    Three arms, on an edit a careless hand would make. The part also carries its own control: it removes the argument from the extracted block and asserts the same driver reaches source-only, so the two arms above cannot pass for a reason that is about the driver rather than the controller.

    The static sweep covers the other two call sites, pgc_setup and devloop.sh, where driving either would cost a build. It joins line continuations first, because all three calls are written across four lines and a per-line grep finds the function name on a line carrying no arguments at all.

    What it cannot see, which is worth stating: that the controller REACHES that line. The make install guard above it could start failing closed and this part would not notice. It asserts what the line does, not that control flow arrives there.

    One of the arms reported got [] when first written, because it read $PG_CONFIG and no selftest part sets that – the harness passes PGC_SELFTEST_PG_CONFIG. Under set -u that aborted the command substitution the driver runs in. The input is now asserted before use and the driver answers driver-could-not-run rather than nothing, because an empty result reports the same emptiness for “the controller is broken” and “this part misspelled a variable”.

  • A continuation check carries the discriminator its headline already names, so two checks stop sharing one ledger row (#982, first of eight).

    The ledger keys on (suite, part, name). Twenty-four checks across eight suites share a key with another check, so one going red would mark its namesake observed-red for a claim nothing attacked – the same failure pgc_record’s BASH_SOURCE part-derivation fixed across parts, one frame further in.

    The cause is a convention rather than carelessness. Suites name a follow-up check as a short continuation of a distinct headline, which reads well:

    PASS  a skipped timing check emits exactly one record
PASS  and its verdict is SKIP

    Two parallel blocks in 400-a-check-result-must-be-machine test check_timing and check_ratio_needs_quiet_machine. Their headlines say which; only the continuations were short enough to collide.

    The rule this proposes, for the remaining seven files: a continuation check carries the same discriminator its headline already interpolates. No new convention, no readability lost, and at loop sites the variable is already in scope:

    before   check_num "and does NOT report it as a missing file"
after    check_num "and the s3 URL is NOT reported as a missing file"

    Six renames here, measured on the resulting log rather than asserted, against a control run on clean main:

    clean main bf325b34   857 records   854 distinct keys   3 colliding   3 lost
this branch           857 records   857 distinct keys   0 colliding   0 lost

    The record count is unchanged, so this adds and removes no checks – it only renames.

    A rename creates orphan ledger rows, and this removes the three it creates. Each carried verdict never and no mutation, which is the criterion f80ca7d05 established for dropping a row without losing history; a row with a date or a mutation would have had to travel with the rename instead.

    And a collapsed key cannot be un-collapsed with its history intact. rename-scan pairs each old name with one of the two new ones arbitrarily, because one row cannot become two. It does not matter here – all three carried nothing – but if any of those six had ever gone red, the ledger would hold that red against a key covering both, and splitting it would leave nobody able to say which check earned it. That is an argument for fixing collisions before one of them reddens rather than after.

    The orphan check written for this also found two rows whose checks no longer exist anywhere in the tree, removed by #917 with their rows left behind. Those are not created by this change and are filed separately rather than tidied away here.

  • The controller-stamp extraction anchors on the write rather than on file position (#961 follow-up).

    Selftest 460 took the first one-tab if ( in run_all_versions.sh. There is exactly one today, so it was unambiguous, and the premises would have caught it if that stopped being true – the extracted block would hold zero or two stamp writes. It was the premises doing the work rather than the anchor.

    The anchor now finds the stamp write and walks back to the if ( enclosing it, then forward to the first terminator at or after it. A subshell added elsewhere at the same indent cannot move the range, because the range is defined by the line it is about.

    A subshell that NESTS around the write can still widen it, and that is a premise rather than a fix. The anchor matches if ( at one tab, so a write inside a deeper subshell leaves the opener pointing at the outer block – which holds exactly one one-tab if ( and exactly one stamp write, so every other premise passes on a block wider than the call site. Measured on a fixture with the write two tabs in: eight lines out, all premises green. A premise counting if ( at ANY indent distinguishes them – one in the real block, two in the nested shape – which is cheaper than teaching the anchor to track depth and fails closed, refusing a shape it does not understand rather than driving it.

    Proven in both directions, because either half alone says nothing. Identical on today’s input – the extracted block hashes 47b4f1a9193c before and after – and different on the input that motivated the change:

    a second one-tab subshell injected ABOVE the stamp block
  OLD anchor   4 lines, 0 stamp writes, so it extracted the WRONG block and the
               `exactly one stamp write` premise reads 0: loudly wrong
  NEW anchor   7 lines, 1 stamp write, unchanged

    md5-only would prove the change does nothing that matters; injection-only would prove it does something without showing what else moved.

    And it closes a boundary the new design could open rather than one the old one had. A backward walk has to decide what to do when it runs off the top of the file, and one of the three possible behaviours satisfies every guard in the part: emitting the write alone gives exactly one stamp write, so both premises pass on a block that is not the call site. This emits nothing instead, because open is never assigned and the guard exits before the print loop – a property that arrived from the guard’s shape rather than from foresight, now written into the code as load-bearing so the next reader does not default start to 1 as a tidy-up.

    One premise added, the extraction produced a block at all, because an awk whose condition never fires prints nothing and an empty block would otherwise read as a block with no stamp write in it – two different failures arriving at the same number.

    Two stamp writes in separate subshells is the case the count premise cannot see: the extracted block holds one and the premise passes. The static caller sweep catches it – injected, both the premise and the arm report got [4] want [3].

  • The hilbert premises name the table they digest, so five checks stop sharing three ledger keys (#982, third of eight).

    hilbert_cluster held the second-largest loss: a premise repeated before four separate arms, and two pairs of (d) checks repeated across two fixtures. Measured against a control run on clean main:

    clean main    181 records   176 distinct keys   3 colliding   5 lost
this branch   181 records   181 distinct keys   0 colliding   0 lost

    The record count is unchanged, so this renames and nothing else.

    A third form of the rule, and the files keep supplying them. #984 said a continuation carries the discriminator its headline interpolates; #989 added that where the headline names none either, it gains one. Here nothing interpolates anything – all eight sites are hand-written – so the discriminator comes from the check’s own value expression:

    premise: the plan being digested here is the columnar custom scan too
  "$(pgc_is_columnar_scan 'SELECT * FROM s3hi')"     -> ... digested for s3hi ...
  "$(pgc_is_columnar_scan 'SELECT * FROM s5hi')"     -> ... digested for s5hi ...
  "$(pgc_is_columnar_scan 'SELECT * FROM s6t')"      -> ... digested for s6t ...
  "$(pgc_is_columnar_scan 'SELECT * FROM av_hi')"    -> ... digested for av_hi ...

    That keeps the name and the assertion in agreement, which is worth more than brevity: a reader can check one against the other without leaving the line. The word doing the colliding was here, which named the site to someone reading top to bottom and named nothing at all to a key.

    The two (d) pairs take the table their own count(*) and physlayout name – moved s4d1's layout, no row was lost from s4d2 – for the same reason.

    No ledger change: hilbert_cluster is not one of the two suites the ledger covers.

  • The object-storage loops name the case each iteration tests, so six checks stop sharing three ledger keys (#982, second of eight).

    objstore_module lost the most records of any suite to key collapsing: two loops of three iterations each, where the check names did not carry the thing the iteration varies. Measured against a control run on clean main:

    clean main    30 records   24 distinct keys   3 colliding   6 lost
this branch   30 records   30 distinct keys   0 colliding   0 lost

    The record count is unchanged, so this adds and removes no checks.

    The two loops show both sub-shapes of the same defect. In the first, the headline already interpolated the scheme and only the continuation was short:

    check     "a s3 URL reports an object-storage error, not a missing file"
check_num "and does NOT report it as a missing file"      <- identical three times

    In the second, neither name carried the metacharacter, so three iterations produced one key for each of two checks – the headline collided as well.

    Both are fixed by the rule #984 proposed: the continuation carries the same discriminator its headline names, and where the headline does not name one either, it gains it. The discriminators come from the loop variable by parameter expansion (${url%%:*} and stripping the fixed prefix and suffix off the pattern), so they are already in scope:

    a remote glob (*) is handled remotely (an object-storage error, not a local one)
and the * glob is NOT reported as a local filesystem miss
and the https URL is NOT reported as a missing file

    The first loop’s headline now reads its scheme from a variable rather than a cut subshell, because both names need it. The resulting check name is byte-identical, so no ledger row moves on account of it.

    No ledger change at all: objstore_module is not one of the two suites the ledger covers, so its check names have no rows. That is also why the twenty-four collisions matter for #432 rather than for the census today – twenty-one of them are in suites that become covered only when the 240 are seeded.

  • The last five suites' continuation checks name the case they continue, closing the rename half of #982 (ten records, five files).

    Ten checks across five suites shared five ledger keys with another check. Measured by running every one of the five on clean main and on this branch, on the same box:

    suite                     control (clean main)        this branch
sorted_pathkeys           113 records 110 keys  3 lost   113  113  0
vector_agg_tlist_shape     68 records  65 keys  3 lost    68   68  0
alter_am_cleanup           45 records  43 keys  2 lost    45   45  0
eager_ordering_record      31 records  30 keys  1 lost    31   31  0
objstore_userinfo           7 records   6 keys  1 lost     7    7  0

    Every record count is unchanged, so this renames and nothing else. All five suites: rc=0, FAIL=0, on both trees.

    This is not a new convention. It is each file’s own convention, applied where it lapsed. Every one of the five already names the case at a neighbouring site – and DESC still answers correctly, and NULLS FIRST still answers correctly, and FILTER still answers correctly, and DISTINCT still answers correctly – and then falls back to a bare and it still answers correctly for the next four. The fix is to finish the pattern the author started:

    REFUSE: a non-prefix of the key is not an order the rows are in
  and it still answers correctly   ->  and a non-prefix still answers correctly
REFUSE: a column that is not in the key at all
  and it still answers correctly   ->  and a non-key column still answers correctly
REFUSE: FILTER inside an expression over aggregates
  and it still answers correctly   ->  and FILTER inside an expression still answers correctly

    Two sites took the discriminator from the value expression instead, because their headline names no table: control: and it moved the layout becomes ... moved the tailgate layout and ... moved the lexgate layout, matching the layout tailgate and layout lexgate the checks actually read.

    No ledger change: none of the five is one of the two suites test/check_ledger.tsv covers, and both ledger files are byte-identical to main.

    One coupling, found by review rather than by either PR’s own checks: #998 added a skip loop to sorted_pathkeys.sh that lists its arms by name, and one of those names is the arm this change renames. The two merge cleanly, so nothing would have presented a conflict – #998’s own guard would simply have gone red in main. The loop is updated here, and its guard reports no mismatch. Counting the old name is how you MISS this: an unanchored grep -F finds 3 occurrences because both renames EXTEND the name rather than replace it, so each renamed line still matches its own old form. Anchoring on the closing quote gives 1, which is the one that matters.

  • pgc_ledger.py orphan-scan reports a ledger row that no record in its own part matches, and --prune removes it only when no history would be lost (#983).

    The comparison already existed and already printed the answer. rename-scan pairs an appearance with a disappearance, so an UNPAIRED disappearance – a check deleted, or renamed in a run where nothing appeared – printed vanished=2 and returned 0. Two rows in the committed ledger named checks that no longer existed; the census counted both, and every run for days said so in a line nobody acted on. A guard that compels one list and ignores the second manufactures the confidence that the thing is handled.

    Driven on the real ledger, which is the only instance that matters:

    orphan: harness_selftest 330-... premise: all three runner functions were extracted
orphan: harness_selftest 330-... premise: and all three are callable
not checked: 44 row(s) in 1 part(s) this run does not contain
orphan scan: parts in the run=43, rows in those parts=861, orphans=2
             (0 carrying history), not checked=44
orphan prune: removed 2 row(s), the ledger now holds 925

    A row carrying history is never pruned, and one such row refuses the WHOLE prune. No run can recreate the catalogue of what has been seen red, and removing the safe rows while naming the unsafe ones would leave a partial job for whoever reads the output.

    Rows in parts the run does not contain are counted out loud as not checked, never as present. Otherwise a single-suite log would certify the whole ledger, which is the same defect one level up.

    --prune refuses a part that skipped, which was this change’s own worst bug. not checked protects a part the run does not contain. A part contained but skipped wholesale fell in the gap: one SKIP record put the part in the run’s parts, every other row of that suite became an orphan, and --prune deleted the suite while reporting not checked=0 and rc=0 – the most confident output the tool can produce. Measured on a three-row fixture for analyze_differential, whose PG17 run is a single SKIP; found by @pgcolumnar-9b in review. Nine suites skip wholesale on PG17 and suites_not_covered is 250, so seeding any one of them would have armed it.

    The rule is broader than that case deliberately: a SKIP anywhere in the part means some arm did not run, so the run cannot tell “this row’s check was deleted” from “this row’s check was skipped under a name that does not match it” – #994’s defect at suite granularity rather than branch granularity. One skipped timing check therefore blocks pruning that whole part, and that is the direction a deleting command should err in. The control holds the other half: the same rows are still pruned when the part’s record is a PASS, so this is not a tool that refuses to prune anything.

    The four categories – matched, orphan, unprunable, not checked – are asserted to account for every ledger row, because a classification that silently loses one is the failure this tool exists to report.

    It reports and is NOT wired into the gate, for a measured reason. Scanned against a real run, 340-the-binary-must-be-built-from.sh records ONE skip under a name neither of its two arms has (the unreadable-source refusal) whenever the box has no non-root user to read as. On such a box two live ledger rows have no matching record, so an absent record does not yet mean a removed check and a gate refusing on absence would redden a correct run. Arming it needs those branches to record a SKIP under the names they stand in for – the conversion #965 made for the eleven timeout paths – and an arm now pins that shape so the day it changes, the arm says so.

  • Nine git bundles are out of the tree, *.bundle is ignored, and 310-a-compiled-artifact-must-not-be.sh now covers transfer artifacts as well as compiled ones.

    I put them there. 76,194 bytes across nine files went into c697c8cd – a merged commit whose subject is a check name in sorted_pathkeys.sh. I create bundles in my clone to move a branch into the audit container, and I staged with git add -A. That is the same mechanism the existing comment in part 310 calls out for a stray .pyc: a tracked build artifact joins whichever commit is next.

    Nothing caught it. Five suites and the whole selftest ran green either side, because no suite has an opinion about files it does not read. What found it was a later rebase printing the filenames in a list I happened to read.

    Part 310’s scope note said Python only, and deferred the wider question:

    Whether every derived file in the tree deserves one rule is a larger judgement and is deliberately not decided here.

    This decides it for one more class, and only that class. A bundle is a transfer artifact, which is why it belongs beside the .pyc rather than beside the Parquet fixtures: it is derived from commits already in the history, it is named after whatever branch was in flight, nothing in the tree opens one, and the next person to make one will choose a different name – so it can never become a fixture anything depends on.

    Same two-part rule, same no-repo discipline as the Python arms, and a control: a source file merely named like a bundle (test/bundle_notes.sh) must not be ignored, or the suffix rule is broader than it claims.

    The blobs stay reachable in the repository’s history – removing a file from the tree does not unwrite it, and rewriting main is not something a stray artifact justifies. What this stops is the tree carrying them, and the next git add -A re-adding them.

  • s3:// and gs:// now refuse URL userinfo, as http(s):// has since #706. They were absorbing it into the bucket name and reporting a missing object (#995).

    http://u:p@127.0.0.1:1/x.parquet   22023  userinfo in "..." is not supported
s3://u:p@mybucket/x.parquet        58P01  "..." does not exist (HTTP 404)   <- before
s3://u:p@mybucket/x.parquet        22023  userinfo in "..." is not supported <- after

    The old message was true and useless: the object did not exist because u:p@mybucket had become a bucket name. s3://user:key@bucket/obj is a form other tools accept, so it is a mistake a user makes, and the answer sent them looking for a missing object.

    Measured on main with an endpoint configured, which is the only way to reach the authority parse at all:

    path style     HEAD /u%3Ap%40pgc-bucket/vh.parquet, Host untouched -> HTTP 404
virtual host   the bucket becomes the leftmost Host label -> could not resolve
               "u:p@pgc-bucket.s3.local"
write          export_parquet raised NO ERROR and PUT to
               /u%3Ap%40pgc-bucket/ui.parquet -- a bucket nobody named

    Not an SSRF, and that was checked rather than assumed. Nothing splits the authority at @, so the userinfo never becomes basic auth and never moves the host. The write row is why this is refused rather than documented: acceptance there sends bytes to a bucket the caller did not ask for.

    The guard sits before the endpoint is resolved. Placed after it, the refusal would be unreachable whenever no endpoint is configured, because the s3 branch demands one first – and the arms could then say nothing without an object-store fixture.

    Twelve arms added to test/objstore_userinfo.sh – five refusals and seven controls, taking the file from 7 checks to 19 – because the first probe of this gap proved nothing: with no credentials every s3 URL returned 28000, the clean one included, so a userinfo refusal was indistinguishable from an unreachable object store.

    The controls hold that a clean URL still reaches the endpoint demand and that the message names AWS_ENDPOINT_URL, that a malformed URL still gets the bucket/key refusal, and that an @ in the KEY is left alone.

    That positive matcher matters more than it looks: 28000 is raised by three different demands in os_resolve_s3 – a missing endpoint, a missing credential, and the authorization refusal – so a control that sees 28000 and no userinfo has pinned nothing about which one fired. The gs:// control makes the point concretely: its 28000 is the credential demand, because gs defaults its endpoint to the interop host and never reaches the endpoint demand at all. Same code, different cause, and only a matcher that names the variable can tell them apart.

  • The matrix runner’s accounting breakdown says which LINE it means, because the old wording produced a wrong planning number twice in one night.

    It printed:

    N via lib.sh's accounting; by their own mechanism: audit bench_guards concurrency ...

    and “by their own mechanism” was read – by two different readers, hours apart – as emits no RESULT records, which would make those twelve suites impossible to seed into the mutation ledger. A bound on how far suites_not_covered can fall was derived from that reading.

    It is false. Measured by running all twelve and counting:

    audit 31   smoke 9   phase2 42   phase3 32   phase4 38   phase5 36
phase6 43  concurrency 7   unique_conc 31   update_conc 25
bench_guards 0   docs_style 0

    Ten of the twelve emit records. The set is the suites whose log lacks lib.sh’s accounting: line and carries their own checks run: instead – a statement about the accounting LINE and nothing else. Only bench_guards and docs_style emit no records, for the reason pgc_log_shows_any_accounting’s comment already gives: those two never source lib.sh at all. Defining a private check is orthogonal – audit does it and records 31.

    So the bound is two suites, not twelve, with about 294 records sitting in the other ten.

    The label now names the line and disclaims the records in the same breath. The selftest arm that pins it follows the reword; widening that arm to pin the disclaimer separately needs either a new ledger row or a rename that would orphan an existing one, and main has no tool to remove an orphan until #983 lands – so the sequencing is written into the arm’s comment rather than quietly skipped.

  • A conftest.py can no longer stub an Expect method on the class so a false claim reports as a pass (#967).

    964 snapshots the module’s bindings. Expect.num = a stub that still counts

    is not a rebind of Expect – the name still points at the same class – so a test asserting 1 == 2 printed 1 passed and exited 0. That is strictly worse than switching off a meta-rule: the comparison never happens, the count still rises, and every guard downstream is satisfied by a test that concluded nothing.

    Public methods of Expect are now snapshotted by identity, the same way the module bindings are. The refusal names Expect.num and restores the method before raising, so an in-process pytester inner session cannot poison the rest of the file. Names that start with _ are excluded: stubbing _record still leaves the count at 0 and is refused by pytest_runtest_call, which is a different mechanism and the control this issue asked to keep.

    What this does not close: expect.num = stub on the instance, or a subclass yielded by an overridden expect fixture. Both still keep the count and drop the comparison. A snapshot of Expect.__dict__ cannot see either. #967 stays open for those two routes.

  • A collection-time vacuity refusal keeps its reason under pytest-xdist (#963).

    pytest_collection_modifyitems raises UsageError. Serial, that is rc 4 and the sentence on stderr. Under -n pytest still runs pytest_collection_finish in a finally, so the worker tells the controller it collected the tests and then exits. xdist’s worker_workerfinished asserts a worker that collected tests must not finish with them pending: a 35-line INTERNALERROR, rc 1, and the sentence is gone. Measured on the pinned runner (pytest 9.1.1, pytest-xdist 3.8.0).

    A worker now records the sentence on workeroutput and clears the items so no ids cross. The controller re-raises UsageError from pytest_testnodedown, which is the process serial already used. The in-test control (a body that concludes nothing) is unchanged in both modes.

  • A collection-time vacuity refusal is no longer also reported as a silent loss (#991).

    The layer refused a run and then contradicted itself:

    ERROR: the pgColumnar vacuity layer refuses this run: a bare skip is refused ...
VACUITY: 1 collected test(s) never reported an outcome, so the run lost them
         silently: tmp/test_offender.py::test_one_offending_arm

    The run did not lose it silently. It refused it loudly, one line above. So a reader who typed one bare @pytest.mark.skip got the correct diagnosis and then a second finding telling them a test vanished without saying so – and the natural response is to go looking for a lost test that was never lost.

    collected - reported is the right set difference and the wrong meaning: a silent loss is when nobody said anything, and here the layer itself is what stopped the run. The guard whose whole subject is a silent loss was firing on the one event that is its opposite.

    One assignment covers all five refusal sites, because #963 gave the layer a single chokepoint: _collection_usage_error records the refusal, and the reconciliation in _RunShape.pytest_sessionfinish skips the missing-outcome problem when it is set. Only that problem. The setup-skip problem still prints – a fixture removing every test that depends on it is not something a refusal accounts for, and the two are independent findings.

    Serial was the only path left. #963 clears items[:] in the worker, so the controller’s collected set is already empty under -n and the contradiction cannot arise there:

    main   serial   refusal + the silent-loss line
main   xdist    no refusal at all            (that was #963)
#963   serial   refusal + the silent-loss line   <- what this closes
#963   xdist    refusal, no line

    The second arm is the control and the first is worth nothing without it: an item collected and never reported, with no refusal anywhere, is still named and still reddens the run. Dropping a problem from a reconciliation is one edit away from dropping the guard. The fixture removes an item after the layer’s hook recorded it and without deselecting it, which is the shape of a crashed xdist worker – pytest_deselected is what tells a deliberate subset from a loss, and nothing calls it there.

    No shell check moved, so no ledger change: harness_selftest is 907 checks on both trees.

[1.0-alpha3] - 2026-09-02

Added

  • A pytest harness beside the bash suites, with a layer that refuses tests which assert nothing (#432).

    test/pytest/ connects through psycopg rather than parsing psql output, so a test can assert the TYPE as well as the value: psql -At returns text, and an int4 1 and a text '1' are the same string to a bash oracle.

    This ports ONE bash suite. test/ carries 4,429 anchored assertions across 256 suites, so this is 0.18% of them and is not coverage. The layer is the point: 71 tests in 6 files, of which 56 test the harness rather than the product.

    A pytest run fails open in several ways this project has already been bitten by: a test that asserts nothing passes, a filter that selects nothing exits 0, and a fixture that skips greens every test under it. pgc_vacuity.py is loaded for every run and refuses those shapes, along with an empty result compared with an empty result, a value compared against itself, a plan matched by substring rather than by typed key, an absence claim over an empty plan, cursor.rowcount of -1, and a broad except found by walking the AST rather than by line regex. xfail_strict is on, and --pgc-expect-tests N asserts the run’s own shape and refuses N = 0.

    Every refusal has a red test that runs pytest inside pytest and asserts on the INNER run’s outcome, which is what proves a guard refuses rather than assuming it. Each records the bare-pytest behaviour it exists to stop; every one of those measurements exited 0. Positive controls sit beside the guards, so a guard that starts rejecting good tests reddens there first.

    Asserting that an inner run failed is not asserting that a named guard fired. A census that neutered each guard alone found most of them deletable with test_layer.py still green, from two causes: several were never driven at all, and several are subsumed by a neighbouring guard, so the inner run fails either way and an outcome-only assertion cannot tell which fired. test_guards_pinned.py pins each refusal to its own MESSAGE through expect.refusal, which refuses to be called with no pattern – the same move as asserting on a SQLSTATE rather than on prose.

    plan_marker’s two arms are pinned, and the hole under them is closed. Both could be deleted independently with the suite green – and it is the worst place in the layer for that, because plan_marker is the port of pgc_is_columnar_scan and is used as the PREMISE that the vectorized aggregate engaged. A premise that cannot fail turns its test into one about an ordinary plan. Underneath both sat a third hole: an absence claim is satisfied by nothing being there at all, so plan_marker([], key, absent=True) passed against a plan that never arrived. That is now refused, for the present arm too. Each of the three neutered alone reddens exactly one test, and it is that test’s own – which is what proves they are distinguishable rather than subsumed.

    The corpus builds and installs before it measures anything. It did not at first: with #error appended to a source file and nothing rebuilt, the run reported 25 passed and exit 0 while the bash suite reported FATAL: the build failed and exit 1. The refusal now comes from pgc_build_and_install in test/lib.sh, driven from Python so there is one implementation rather than two that can drift, and it runs BEFORE the cluster starts – shared_preload_libraries maps the library at postmaster start, so a cluster started before the install keeps the old one mapped for its whole life.

    The third state is a state, not a comment. expect.cannot_run(REASON, detail) wrote a field nothing read, so a test declaring itself unrunnable reported 1 passed and exit 0 – a write-only flag, the shape selftest 320 already polices in the runner. It made the layer’s own escape hatch its largest hole, because a bare @pytest.mark.skip fails the run while the honest-looking alternative greened silently. A run holding one now exits 67, the same number as PGC_EXIT_INCOMPLETE in lib.sh, and prints the reason and detail in the same shape; a run holding a real failure as well still exits 1, because failure dominates. Verified serial and under -n 2, the declaration travelling to the xdist controller on the test report.

    test/pytest/TESTS.md is checked rather than trusted. It documents every test in the corpus, and it went stale inside a single rework: 29 of 54 tests were named nowhere in it while its header still claimed 25. A guard now requires every file and every test to be named there, and requires the totals it states to be the totals on disk – a document can name every test and still miscount them. It is written twice, as test/selftest/350-the-pytest-corpus-must-be.sh and test/pytest/test_docs_cover_the_corpus.py; only the first runs in the gate, and it reddened on its twin’s arrival before this entry was written.

    Not registered in test/run_all_versions.sh. That would add a psycopg build dependency to every CI leg for 0.18% of the assertions; test/pytest/README.md records what registering would cost and what has to be true first.

  • A nanosecond Arrow import says how many values lost precision, and still imports every row.

    pgcolumnar.import_arrow narrows nanoseconds to microseconds and always did. PostgreSQL timestamps and times are int64 microseconds; Arrow parameterises the unit in the type. Of Arrow’s four units, second, millisecond and microsecond all widen or match exactly, so only nanosecond can lose anything, and only for a value that is not already on a microsecond boundary.

    The narrowing was silent, and that is what changed. An import now ends with

    NOTICE:  columnar.import_arrow: 12 values lost sub-microsecond precision
DETAIL:  PostgreSQL timestamps and times hold microseconds; this Arrow file
         declares nanoseconds.
HINT:    Import the raw nanoseconds into a bigint column if the extra digits
         are significant.

    Counted per value, not per type. A pandas datetime64[ns] column built from second- or millisecond-resolution data is nanosecond-TYPED and entirely lossless to convert; it reports nothing. Only values that actually carried sub-microsecond digits are counted. The remainder is not extra work: arrow_floordiv already computes it to decide the rounding direction.

    The counter is reached from the time arm as well as the timestamp arm, and is threaded through the list and struct recursion, so a nested timestamp[] and a time64[ns] column are counted the same way.

    Nothing is refused. NOTICE, not WARNING or ERROR: a bulk load must not fail on the last row of a large file for a conversion the caller may have intended. Out-of-range values are still refused with 22008, unchanged.

    Truncation does more than reduce precision – it can make rows that were distinct in the file compare EQUAL here, so a UNIQUE violation or a lost ORDER BY tie-break now has its cause stated at the point it was introduced rather than surfacing later as a mystery.

  • The two visibility-map clears that no test held are now held (#877).

    Three paths retire a row group and give its live rows new row numbers, and each must clear the all-visible bits over the OLD numbers, or an index-only scan answers from the index for a TID whose group no longer exists. Only pgcolumnar.expire’s clear was covered, because only its absence had been reported as data loss. Deleting the other two – rewrite_one_group (src/columnar_vacuum.c:346, reached through pgcolumnar.compact_rewrite) and pgcolumnar_recluster_online (:765, through pgcolumnar.recluster) – left 319 checks across 14 suites green.

    test/vm_clear_on_renumber.sh is the two missing arms. Neuter the clear at :346 and one arm reddens; neuter the one at :765 and two redden; neither mutation reddens the other’s arms.

    The instrument is why this was not covered earlier. pg_class.relallvisible is a statistic VACUUM refreshes, and clearing a bit does not touch it, so an arm reading it reports the same number with the clear and without. pgcolumnar.vm_is_visible(rel, blk) reads the fork through visibilitymap_get_status, which is the call the index-only-scan executor makes.

    And a DELETE clears the bits of the blocks holding the deleted rows, so a group made compactable is already not-all-visible there and an arm placed on it cannot fail. These arms VACUUM first, delete only the front of the target group, and assert over the group’s later blocks, which are all-visible on the way in and reachable only by the rewrite.

    The two controls differ in strength and the suite says so. The rewrite’s control is a group in the same table, and widening that clear to the whole relation reddens it. Recluster renumbers every group, so that half has no in-table control; its control is a second relation, which catches a clear reaching beyond the relation it was called on but NOT an over-broad range – the same widening leaves it green. That limit is measured and recorded in the file rather than left for a reader to infer.

  • MERGE is documented as working, which it has been all along. It needs no index on the columnar target and takes every arm, including WHEN MATCHED ... DELETE, WHEN NOT MATCHED BY SOURCE, and RETURNING merge_action(). A columnar table can be the source as well as the target. Verified rather than assumed, on PostgreSQL 17.10, in four shapes.

    Nothing in the code changed. The gap was that a reader had no way to learn this: MERGE appeared in no user-facing document, and its absence from docs/limitations.md reads as easily as “unsupported” as “supported”.

    docs/features.md also records what a MERGE costs, because that is the part a reader acts on: its updates and deletes mark rows rather than rewriting row groups, exactly as a plain UPDATE or DELETE does, so space returns only after pgcolumnar.vacuum.

Fixed

  • A projection created mid-transaction receives the writes that follow it (#875).

    A write before pgcolumnar.add_projection() in the same transaction made every later write in that transaction skip the new projection. The rows landed in the base table and never reached the projection, with no error, and a covering projection scan then answered as though they had not been inserted.

    BEGIN;
  INSERT INTO t ...;                       -- any write will do
  SELECT pgcolumnar.add_projection('t', ...);
  INSERT INTO t ...;                       -- these rows were lost to it
COMMIT;

    Measured: 116 rows in the base table, 105 in the projection.

    PgColumnarProjectionFanoutRow builds the write state’s projection-writer list on first use and latches it, including when the list comes back empty – which is what it is before the projection exists. add_projection now drops that cache, after the back-fill, so the writes that follow rebuild it from the catalog.

    drop_projection had the same defect in the other direction. A writer cached before the drop kept taking rows, which landed in a projection storage whose catalog rows were already deleted and committed as an orphan: one orphan storage id when the drop happened mid-transaction, none when it had the transaction to itself. It drops the cache too.

  • An Arrow import reads the width, sign and scale the FILE declares (#881).

    imp_apply_field inspected only Date, Time and Timestamp. For every other tag the stride and the interpretation came from the TARGET column, so a file that declared something else was decoded as though it had not:

    uint64 2^63+5        into bigint          ->  -9223372036854775803
int64  1,2,3,4       into int             ->  0,0,1,2
decimal(10,2) 1.25   into numeric(20,4)   ->  0.0125
fixed_size_binary(32) into uuid           ->  the first 16 bytes

    All four imported without an error. They are refused now with 42804.

    The buffer-length check already caught the cases where the file’s carrier is NARROWER than the target; these are the ones where it is the same width or wider, so the buffer is long enough and nothing complained.

    Scope is within a family. A tag that does not match the column’s family at all – an int64 read into a timestamp as raw microseconds – is long-standing accepted behaviour with its own test, and is unchanged.

  • read_projection explains itself after a rewrite (#876).

    A rewrite mints a new storage id while pgcolumnar.projection keeps the old one, so a projection that is still declared reads as absent. The error said only projection "p" does not exist on "t", which is not true – the declaration is intact. It now carries a hint naming pgcolumnar.rebuild_projections(), which recovers it. The underlying re-recording is still open as #876.

  • DROP after ALTER TABLE ... SET ACCESS METHOD heap now takes the relid-keyed catalog rows with it, and the hook that does it stays out of the way in databases that have no extension.

    Two catalogs are keyed by relid rather than by storage id. pgcolumnar.options and pgcolumnar.projection_declaration survive the rewrite onto heap storage, which drops only the storage-id catalogs. The drop hook then looked at the access method, saw heap, and returned before touching them. config_dump emits a bare oid for a relid that no longer resolves, and rebuild_projections() aborts on an orphan declaration, taking every other table in the database with it. That is the blast radius #304 closed for a plain DROP of a still-columnar table, reached instead through access-method conversion.

    The hook runs in every database of the cluster, because the library is preloaded rather than loaded by CREATE EXTENSION. So it must reach the columnar catalogs only where they exist. It now gates on pgcolumnar.options resolving, not on the schema: DROP EXTENSION leaves the schema behind and takes its tables, so a schema test passes in exactly the case where the catalogs have gone.

    It also skips the extension’s own member relations rather than everything in the extension’s schema. A user table that merely lives in pgcolumnar is not a member, and it gets cleaned up like any other.

    This matters beyond DROP TABLE. Every table rewrite drops a transient relation through the same hook, so VACUUM FULL, CLUSTER, ALTER TABLE ... ALTER COLUMN TYPE and CREATE MATERIALIZED VIEW take the path too, and vacuumdb --full --all visits every database in the cluster.

  • pgcolumnar.expire() no longer drops live rows, and every path that renumbers live rows now clears the visibility map (#403).

    Three separate ways to lose data or read a row that is gone.

    A group holding a row with a NULL retention value could be retired, taking live rows with it. expire now keeps any group whose retention column has a LIVE NULL, because a NULL has no age and the group cannot be known to be wholly expired. Deciding that reads the retention column of a candidate group, on a table that has any deletes. A table with no deletes still decides from row-group metadata alone. Nothing is rewritten either way.

    Live is the operative word. The zone map’s null count is recorded when the group is written and never revised, so it still counts rows a later DELETE marked. Reading it as the live count keeps a group whose every live row is past retention and whose NULL rows have all been deleted, and keeps it permanently, because nothing rewrites a zone map on delete. That trades data loss for silent over-retention. expire now checks the live rows, and only for a group that has both recorded NULLs and deletes: with no delete vector the recorded count is still exact, so the metadata-only path is unchanged and expire still reads nothing.

    A negative ttl_interval put the cutoff in the future, so expire retired groups that were entirely inside their retention. set_options range-checked every other option it accepts and not this one. Zero and negative intervals now raise 22023.

    Retiring a group leaves its old row numbers in the visibility map, so an index-only scan answers from the index for a row group that is gone. None of the three paths cleared them. expire’s absence is the one that was reported; pgcolumnar.recluster() and the partial-group rewrite behind pgcolumnar.compact_rewrite() renumber live rows through the same retire and had the same gap, with no report against either.

    All three clear now. As of #878 all three are held by tests; before it, only expire’s clear was. The rule is that visibility-map bits go wherever row numbers are reassigned, not only where rows expire.

    docs/sql-reference.md gains the accepted range for ttl_interval and the NULL rule, which is stronger than the straddling behaviour the page already described: a straddling group is released once its newest row ages past the cutoff, and a group holding a NULL never is.

  • The matrix runner counts incomplete suites per major, not per run, and the INCOMPLETE dispatch is now testable end to end (#858).

    suites_incomplete=${suites_incomplete:-0} is a set -u guard, not an initialiser. It keeps whatever the previous major left, while suites_ran and suites_skipped on the lines above it are zeroed unconditionally, and verfail is reset per major too. On the five-major matrix the count accumulated: PG16 would report PG15’s incomplete suites in its own summary line and still print PASS, because the one number in a per-major report belonged to the whole run.

    Latent today. check_unrunnable has no production call site, so no real suite can reach the INCOMPLETE state in a matrix run yet.

    The tally is now a function, pgc_tally_suite, rather than four branches in the middle of the per-major loop. The regression #859 shipped was not in the classifier: it returned INCOMPLETE correctly and the caller threw the answer away into a write-only flag. Nothing could reach the caller, because a loop that needs a suite list and a populated build directory is not something a selftest can drive. Extracted, the whole chain is drivable, and test/selftest/330-the-incomplete-path-must-run-whole.sh drives it: a real suite exits 67, the runner’s own classifier reads the files that suite wrote, the runner’s own tally consumes the classifier’s verdict, the runner’s own collect loop runs over both fixtures, and the runner’s own major-verdict branch decides PASS or FAIL.

    The arm that earns the file reads the loop’s own text. Every behavioural arm stays green if the runner defines the tally and never calls it, which is the defect class the file exists to prevent.

  • The planner estimate counts live rows, and reads the delete count in one catalog scan rather than one per row group.

    row_group.row_count is physical occupancy. It still counts rows that a later DELETE marked in the delete vector, and the planner uses this callback instead of pg_class.reltuples, so every scan of a heavily deleted table was priced as if the deletes had not happened.

    The count now comes from one indexed scan summing delete_vector.deleted_count over the storage. The earlier shape walked each group’s bitmap a bit at a time, once per row group, on every plan of a columnar relation.

    Summing is exact rather than an approximation. delete_vector carries a unique index on (storage_id, group_number), so there is one row per group and no two summands can count the same row. The comment that justified the per-group fold said the opposite, and the schema forbids what it described.

  • UPDATE now fans the new row version out to every covering projection, so a projection scan stops answering as if the updated rows were gone.

    A projection stored the base row number, and UPDATE never gave it the new one. A projection joins back to the base table on the row number and takes visibility from the base delete vector, so DELETE needs no rewrite: the vector hides the old number from the projection too. UPDATE is delete-old plus insert-new, and only the delete half reached the projection. The projection kept the old number, the delete vector hid it, and the new number was nowhere. read_projection returned no rows, and a covering projection scan (projected columns with a sort-key qual) answered as if the updated rows had been removed.

    The base table was always correct. The loss was confined to reads the planner served from a projection, which makes it the worse shape: the same query returns different answers depending on whether the projection is chosen.

    tuple_update now calls the same fan-out that insert calls.

    docs/features.md, docs/administration.md and docs/how-to.md said only that inserts write projections. All three now say updates do too, and say why a delete does not.

  • Arrow import reads the temporal unit and carrier width the file declares, rather than assuming the ones our own exporter writes (#864, #865).

    A file could say what it meant and be read as something else. The importer built its decode plan from the target column type alone and never opened the Arrow Field table, so the unit went unread. A date64 column holding 2000-01-01 was decoded as a day count and stored as 4908285-05-04; a timestamp in seconds became 1970-01-01 00:15:46.6848, in milliseconds 1970-01-11 22:58:04.8, and in nanoseconds 31969-04-01. A time64 in nanoseconds holding noon stored 12000:00:00, a legal PostgreSQL time. None raised an error. time32 could not be imported at all, failing with “value buffer too small for the row count”.

    Only microsecond timestamps and times, and date32 dates, were read correctly, and those are exactly what export_arrow writes – so a round trip through our own exporter never showed the defect.

    The import now reads Date.unit, Time.unit, Time.bitWidth and Timestamp.unit from the file and scales to PostgreSQL’s units, treating an absent field as its FlatBuffers default. That last part matters: a writer omits any field equal to its default, and two of these defaults are not zero, so pyarrow emits date64 and time32[ms] with no unit field at all. Reading an absent field as zero is what produced the date64 result above.

    Scaling a coarse unit up can leave PostgreSQL’s range, so each conversion is overflow-checked and refused with 22008 rather than wrapped. Nanoseconds are narrowed to microseconds, which PostgreSQL cannot store beyond: narrowing keeps the instant, where reading nanoseconds as microseconds is wrong by a factor of

    1. Every narrowing floors, so an instant before the epoch reports the day and the microsecond it falls in rather than the one after.

    The unit is applied to nested fields too, so a timestamp[] or a composite with a temporal member is read by its own declared unit.

    Reading the file’s declared type also closed a third case, found while fixing these two and present since Arrow import shipped: a temporal file whose type the target column cannot hold was read anyway, taking the low four bytes of an eight-byte carrier. A time64 file imported into a date column stored 687342-02-27, and a timestamp file into the same column stored 2722128-09-17. Both are now refused with 42804, naming the Arrow type and the column type. A file whose type is not temporal is unaffected, so importing a plain int64 file into a timestamp column still works.

  • TABLESAMPLE on a columnar table is refused rather than silently ignored (#866).

    The AM’s sample callbacks have always raised 0A000, and docs/limitations.md has always said so. The planner hooks never let a query reach them: set_rel_pathlist_hook added a custom scan in place of the Sample Scan, and the vectorized aggregate paths did the same for an aggregate. The sample was then not applied at all. Measured on 10,000 rows with a heap mirror as the oracle: TABLESAMPLE BERNOULLI (10) returned 1030 rows from the heap table and all 10,000 from the columnar one, and sum(id) came back as the whole table’s sum rather than a sample’s. SYSTEM behaved the same way.

    A wrong answer is worse than a refusal, and the refusal is the documented behaviour. All three hooks that read the range-table entry now decline when rte->tablesample is set, so the query reaches the callbacks and raises. An unsampled query on the same table still takes the custom scan.

  • A custom scan no longer hides the children of an INHERITS hierarchy (#871).

    A legacy inheritance parent is an ordinary RELKIND_RELATION carrying rte->inh. set_rel_pathlist_hook still runs on that appendrel after core has built the Append over its children, and a custom scan added there reads only the parent’s own storage. Every child’s rows became invisible, with no error and no warning. Measured with a heap mirror as the oracle, parent 1 row and child 5000: SELECT count(*) FROM parent returned 1 where 5001 is right, from a plan containing no Append. The ungrouped vector aggregate path had the same hole; the grouped path already refused this shape.

    Both hooks now decline when rte->inh is set, leaving the hierarchy to the ordinary Append plan. The children keep the custom scan, because a child’s own range-table entry has inh false: the resulting plan is an Append of two PgColumnarScan nodes with pushdown intact, not a fallback to Seq Scan.

  • Both compaction thresholds are validated, in both entry points (#860). pgcolumnar.compact_rewrite accepted NaN for min_deleted_fraction, because NaN < 0.0 and NaN > 1.0 are each false. The call then matched no row group and reclaimed nothing while reporting success.

    pgcolumnar.maintenance_due is the gate the pgcolumnar.autovacuum daemon consults before it ever calls compact_rewrite, and it validated nothing at all. Measured on a table with 10000 of 20000 rows deleted: NaN and 2.0 both reported compact_rewrite_due = f, suppressing the work for good, and -1.0 reported t, which makes the daemon believe compaction is always due and rewrite every columnar table on every sweep. NULL behaved like NaN, because the daemon reads a NULL verdict as “not due”. Both thresholds now raise invalid_parameter_value (SQLSTATE 22023) for all four, matching compact_rewrite.

    0 and 1 remain legal on every threshold, and test/native_reclaim.sh now pins both endpoints as accepted. That is the direction a bounds fix usually breaks, and nothing could see it: the suite’s only call at 0 was a bare psql_run whose exit status nothing read, so a rejected 0 reached nothing but the server log. Measured with minFrac < 0.0 changed to minFrac <= 0.0: exactly one arm fails, the new one, and the other 32 pass. The same mutation applied to compact_due_fraction reddens the matching maintenance_due arm and nothing else.

    Every deny arm asserts the SQLSTATE, not “it failed”. Four calls that satisfy “it failed” without touching the guard are pinned as controls: a missing function and a wrong-arity call (42883), a null table name (22004) and 1/0 (22012).

  • A parallel export refuses a destination too long to hold the names it generates (#863).

    pgcolumnar.parallel_export_parquet checked the destination’s length not at all. Worker paths cross shared memory in MAXPGPATH buffers and have a generated name appended, and snprintf truncates rather than failing, so a long destination produced short names and the export completed as though nothing had happened.

    Two distinct failures, both silent. A destination of 1000 bytes or more truncated the part names themselves: the run wrote part-0000.parqu instead of part-0000.parquet, stamped _SUCCESS beside it, and reported success for an export pgcolumnar.read_parquet does not recognise. A destination of 994 to 999 bytes wrote correct part names but truncated the path that pexport_remove_outputs composes for its cleanup scan – "%s/%s" from the same directory and the sink’s in-flight part-NNNN.parquet.tmp.<pid>, which is 30 bytes past the directory with a seven-digit pid – so the scan unlinked a truncated path and left the temporary file behind.

    The destination is now probed against the longest name the file constructs, "/part-%04d.parquet.tmp.%d" at INT_MAX for both, which is 39 bytes past the directory against the final part name’s 24. Anything that does not fit is refused with 54000 before the destination directory is created, because a truncated run publishes a differently named object and still stamps _SUCCESS over it.

    The sink itself was never at risk and is unchanged: columnar_sink.c builds its temporary name with psprintf, which allocates.

  • A shebang and the execute bit go together, and every directory that documents a command is swept (#856). Two things were left over from #852.

    The rule flagged both states, so a sourced fragment could not be correct. It reddened a file that declared an interpreter without the bit, and it also reddened a file with no interpreter line at all. Its header documented the way out as “drop its shebang and say why”, and that way out did not exist: dropping the shebang moved the file from one red to the other. bench/cb_guards.sh is the file that proved it – sourced by bench/run_clickbench.sh and test/bench_guards.sh, its header saying “Sourced, not executed” since it was written, and unfixable under the old rule. CONTEXT.md had described the rule correctly all along, as failing “if one has either without the other”; it was the code that was stricter than the documented rule.

    The rule is now that biconditional, and a file with neither a shebang nor the bit passes.

    That removed the exemptions. test/selftest/ and test/fixtures/ were pruned by path because the old rule would have reddened them wholesale. Measured under the new one before the prune came out: selftest/ is 31 scripts and every one is already correct, fixtures/ is 14 .sh/.py of which 13 are the host tools this change gives the bit. Nothing is excluded now, so nothing is concealed.

    bench/ joins the population. docs/benchmarks.md names five bench/ scripts as bare commands. All five are executable today, so all five work: correct by habit with nothing checking it, which is what test/ was before

    852.

    A third check, anchored on the documents, closes the hole the biconditional opens. Delete a documented command’s shebang and its bit and the file is internally consistent and still broken for a reader, so neither of the first two rules can see it. The third requires every script a document names to be executable.

  • No compiled Python artifact is tracked, and the tree ignores the ones the interpreter writes (#854). test/__pycache__/ste_check.cpython-312.pyc was tracked. Its source, test/ste_check.py, was renamed to test/plain_language_check.py in e9de048; the .pyc outlived it by 146 commits and was still tracked at fe1f3a2, the base of this change. So the file was compiled code for a module that no longer existed and that CPython would never open: it reads a __pycache__ entry only when the matching source sits beside it.

    A tracked build artifact does not stay still. This one had already re-committed itself inside an unrelated logical-replication fix, where the diffstat reads Bin 6028 -> 6028 bytes and exactly two bytes differ – the PEP 552 source-timestamp word. The compiled code was identical either side.

    .gitignore gains __pycache__/ and *.pyc, which it had neither of, and test/selftest/ gains a part that fails when a compiled Python artifact is tracked or when either rule goes missing. Ignoring is not enough on its own and the suite proves it: restoring the tracked file with both rules in place still reds two checks, because .gitignore has no effect on a file git already tracks.

    test/devloop.sh stages its build directory with tar --exclude=.git, so the tree the suites run out of was not a checkout and the new checks reported five failures on a clean tree. It now writes a one-line gitfile into the build directory instead: 45 bytes rather than the 32 MB of copying the object database, and the loop stays as cheap as it was. Every check in the part also answers no-repo where there is no repository, rather than the answer git gives by default – ls-files prints nothing, which reads as a clean tree, and check-ignore says “not ignored”, which reports a present rule as missing.

  • Every test script is executable, so the commands this project documents run as written (#852). No released version is affected: nothing in the extension changed, and this is test tooling only.

    103 of the 262 top-level scripts in test/ were mode 100644, and 30 documented invocations named one of them. test/temporal.sh /path/to/pg_config and the 29 others died with Permission denied before running a single statement. The fix gives those 103 files the execute bit.

    Every green run in this project’s history is honest, and the reason is the point. ci.yml:485 and nightly.yml:188 both invoke bash test/run_all_versions.sh, and the runner starts each suite as bash "$builddir/test/${s}.sh" at lines 712 and 749. An interpreter named on the command line does not consult the execute bit, so a suite’s own mode never reached the matrix. The gap was between the documentation and a reader’s shell, and no green matrix could stand in it.

    Two situations, not one. Exactly one 100755 to 100644 transition exists in the whole history of test/: 56ae5f8eb, a perf commit that added a line to SUITES and stripped the bit on the way past. The other 102 files were born 100644, so one is a regression and the rest are a habit nothing contradicted.

    test/crlf_listener.py was the only script with no interpreter line, and the bit alone would have made it worse: execve returns ENOEXEC, the shell retries the file under /bin/sh, and the reader gets a syntax error instead of a clean refusal. It now declares #!/usr/bin/env python3, as its eleven siblings do.

    A new harness_selftest part pins both halves for every .sh and .py under test/. The rule is anchored on the file’s own first line rather than on a name list or on what a document happens to mention: a script that opens #!/usr/bin/env bash has said it is meant to be run, and a mode that forbids running it contradicts the file itself. Run against the unfixed tree the check reports 102 files without the bit and 1 without a shebang, which is the population the issue measured, so the check is neither over nor under matching. Stripping the bit from run_all_versions.sh alone, which is what 56ae5f8eb did, reddens it naming that one file.

    The sweep exempts exactly two directories. The parts in test/selftest/ and the scripts under test/fixtures/ are sourced or imported, never executed, so the bit would advertise a way to run them that does not work. That is this same defect pointing the other way. Every other directory is swept, which is what covers test/pbt/run.sh: it is a documented command (docs/testing.md:132) that lives one level down, it was already correct, and a sweep of the top level alone would have left the file most like this defect outside the guard.

  • docs/limitations.md states the constant-typing rule correctly. The previous wording said smallint and real “always need a cast”. That is false: an unadorned quoted literal is unknown and takes the column’s type, so smallint_col < '500' skips. The rule is not special to the temporal types either. Their literals are simply always quoted, which is why they skip as written.

    Two limits on that rule are stated with it, because the first revision of this entry got both wrong. A literal that names its own type is that type rather than unknown, so timestamp_col < DATE '2026-01-01' does not skip, which is what the same page’s conditions list has always said. And a digit string is an integer only while the value fits in one: integer_col < 5000000000 is an integer column against a bigint constant and does not skip either.

    Measured across the nine skippable types in three literal forms, with the planner’s own constant printed beside each skip count, and every row count compared with a heap oracle. Nine checks in test/parquet_export_stats.sh now pin the rule, and each can fail: three go red when the reader’s exact-constant-type refusal is deleted, four when the writer stops emitting statistics, and the two plan-shape checks go red when the patterns they look for are swapped.

  • Exported Parquet files carry row-group statistics, so a file pgColumnar wrote can be skipped (#850). Reported from outside the project, and the report was right: every condition docs/limitations.md documents for row-group skipping could hold and Row Groups Skipped would still be 0.

    write_column_chunk() emitted ColumnMetaData fields 1 through 7 and 9, and never field 12, so no file we wrote carried a minimum or a maximum for the reader to test a predicate against. The read side was never at fault. On the reporter’s own repro, a 1,000,000-row table exported to 16 row groups: 0 of 32 column chunks carried statistics, and the foreign scan decoded all 16 groups for id < 5000. It now carries 32 of 32 and skips 15 of 16, decoding one. The native chunk-group skip on the same data always worked, and is unchanged.

    What the exporter now writes, per column chunk: null_count always, including where it is zero, because parquet.thrift says a reader must not read an absent count as zero; min_value and max_value for the columns stored as INT32, INT64, FLOAT or DOUBLE, which is exactly the set the reader can skip on; and nan_count for the floating point columns. A text, bytea, uuid, boolean or byte-array numeric column gets the null count and no bounds: UTF8 sorts by unsigned bytes, which is not any PostgreSQL collation, and a bound in the wrong order is worse than no bound at all.

    The file footer now also carries column_orders, one TYPE_ORDER per leaf column. It is not decoration. parquet.thrift states that without it the meaning of min_value and max_value is undefined, and Arrow acts on that by discarding both, so statistics without column_orders would have left the file exactly as unskippable under pyarrow as it was before. With it, pyarrow reports is_stats_set=True on our files.

    Three rules from the specification are followed for floats, and each is pinned by a test: a NaN never becomes a bound, a column whose non-null values are all NaN gets no bounds at all, and a computed zero is written -0.0 as a minimum and +0.0 as a maximum. A value with no Parquet representation, such as an infinite date or timestamp, is folded to null before the accumulator sees it, so it counts as a null and cannot reach a bound.

    test/parquet_export_stats.sh covers this, reading the footer bytes with test/parquet_stats.py rather than asking a third-party library whether it feels like surfacing them. Eight mutations of the fix were each run against the suite and each turned a named check red.

    One of the eight loses rows rather than merely losing the skip: taking a date bound from the PostgreSQL epoch instead of the Unix one. That shift leaves a well-ordered interval which is simply wrong, so no guard can see it, and the single check standing between it and silent row loss is a predicate placed past the end of the data, d > DATE '2060-01-01'.

    Swapping the two bounds does not lose rows, which is worth knowing rather than glossing: it inverts the interval, and the reader already refuses to skip on an inverted one. docs/limitations.md documents that refusal, and it caught this mutation. The swap loses 54 checks and no rows.

    Files exported by 1.0-alpha2 or earlier carry no statistics, and nothing rewrites them in place. Export again to make one skippable.

  • Index entries for live rows are no longer destroyed (#838). A released version is affected: this is present in v1.0-alpha2.

    pgcolumnar_index_delete_tuples answered nbtree from the calling backend’s MVCC snapshot. table_index_delete_tuples asks a different question, whether an entry is dead to everyone, which heapam answers from a global visibility test. nbtree acts on the answer in _bt_delitems_delete_check by removing the items from the leaf page physically, inside a critical section, and nothing restores them. A row the current uncommitted transaction had deleted read as not live while remaining live to every other session, so entries for live rows were erased.

    Measured on 400 live rows given twelve committed rounds of churn on a column that is not indexed: an index scan reached 0 of the 400 after a ROLLBACK and 228 of 400 after a COMMIT, while SELECT count(*) at shipped defaults answered 228 through an index only scan and SELECT count(b) answered 400 through the columnar scan, on the same table in the same session. Two controls place the fault: churning the indexed column instead makes index_unchanged_by_update() false so no bottom up pass runs and nothing is lost, and a heap given the same hint loses nothing.

    Native storage carries no per row transaction id to compare against a global horizon, because the delete vector is a bitmap whose visibility comes from the MVCC catalog rows that hold it. It therefore cannot prove global deadness and now vouches for nothing rather than guessing. The cost is that version churn no longer reclaims leaf page space, which docs/limitations.md already described: stale entries are reclaimed by REINDEX, not removed opportunistically.

  • A scrollable cursor no longer answers a backward fetch with forward rows (#842). A released version is affected: this is present in v1.0-alpha2.

    pgcolumnar_scan_getnextslot accepted a ScanDirection and never read it, so every call advanced forward. The fetch succeeded, with SQLSTATE 00000; it simply returned rows that were not asked for. Measured on a 20 row fixture with the plan asserted in both arms: FETCH FORWARD 5 then FETCH BACKWARD 3 gave 6 7 8 where the heap gave 4 3 2, and FETCH LAST returned nothing where the heap returned row 20.

    Forward is honoured, NoMovement returns no row, and backward is refused with feature_not_supported rather than emulated. The reader walks row groups and decodes vectors forwards, so scanning the other way is a feature rather than a correction, and inventing an answer is the one thing that must not happen on this path. At shipped defaults the planner gives a columnar relation a custom scan, which serves these cursors correctly, so the refusal is reached only where that path has been turned off.

  • An encoded NUL no longer defeats the Iceberg traversal guard (#844). A released version is affected: this is present in v1.0-alpha2.

    ice_has_encoded_dotdot catches a traversal smuggled through percent encoding, because an http or https origin may decode a key before serving it. A %00 in front of the traversal defeated it: ice_percent_decode_once wrote the decoded NUL and kept going, and everything downstream is NUL terminated string work, so the next pass measured with strlen and the scan walked with strchr. Both stopped at that byte and never reached the ../ behind it. The literal .. guard beside it saw nothing either, because the segments were encoded.

    Measured over the S3 fixture with the delete path recorded as %00%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/hostname: the literal and the plainly encoded forms were both refused with 22023, and the form behind an encoded NUL returned 58P01, meaning it was not refused but fetched, and failed only at the escaped location. A NUL has no place in an object key or a path, so the encoded NUL is now refused on its own rather than scanned past.

  • pgcolumnar.vacuum_sorted’s comment no longer calls cluster() numeric-only (#827 follow-up). A released version is affected: the sentence is in v1.0-alpha2, in the full script and in the upgrade script, so \df+ prints it to a user today.

    It is wrong in both directions. cluster_type_supported() takes boolean, smallint, integer, bigint, real, double precision, date, timestamp and timestamptz, several of which are not numeric, and it does not take numeric at all. The extension already contradicted the sentence: a rejected column raises an errhint reading “Z-order clustering supports integer, date/time, boolean, and floating-point columns”. The comment now uses that wording, so there is one description of the rule rather than two that disagree.

    Measured with a positive control so the deny arms are not vacuous, on a 20,000 row table: vacuum_sorted accepts numeric and text; cluster() accepts int and rejects numeric with “column n of type numeric cannot be used as a clustering key”, and rejects text likewise.

    The 1.0-alpha2 to 1.0-alpha3 upgrade re-issues the comment, as set_options, expire, parallel_copy and sort_status already do in that script. Without that half, a fresh install and an upgraded one disagree and native_upgrade_converge fails, which is the suite doing its job: it hashes obj_description(p.oid,'pg_proc') for every function in the schema. Verified in that order: 8 of 8 on unmodified main, failing on both the 1.0-alpha and 1.0-alpha2 paths with only the full script corrected, and 8 of 8 again once the upgrade script carried it.

  • pgcolumnar.set_options no longer writes past three stack arrays during ALTER TABLE ... RENAME COLUMN (#834). No released version is affected: the defect was introduced and fixed inside this cycle.

    pgcolumnar.options grew ttl_column and ttl_interval for retention and the Anum_options_* constants were extended to 9, but Natts_options three lines below them was left at 7. It sizes the values, nulls and replace arrays handed to heap_modify_tuple, which iterates tupdesc->natts, so a rename on a table with a declared sort_by read two slots past the end of all three and aborted the backend, taking the cluster into crash recovery.

    On an ordinary build the overrun lands in adjacent stack slots and is silent, so the whole matrix passed. test/catalog_natts.sh now pins every Natts_* constant against the width the live server reports, which makes it a check on every major rather than a sanitizer only one.

  • date_trunc(unit, ts) = 'infinity' returns the matching row again (#836). No released version is affected: the defect was introduced and fixed inside this cycle.

    The equality preimage builds the bounded interval k >= lo AND k < hi. Interval arithmetic saturates at an infinity instead of raising, so hi = lo + step returned lo again and the interval was empty, excluding the row it was built to select. Measured on a 5000 row fixture holding exactly one +infinity and one -infinity row, at default settings: the heap returned 1 and the columnar table returned 0 for both, while a bare ts = 'infinity' returned 1 from both.

    Equality now declines when the interval degenerates, which costs a scan rather than an answer. The ordered operators keep their exemption and are unaffected, because each emits one key, so lo == hi costs them nothing. That is why the two infinity arms already in test/preimage_rewrite.sh stayed green while equality was wrong: the suite carried infinity arms, and carried equality arms, and never their intersection.

  • sum(bigint) and avg(bigint) no longer accumulate across a rescan (#840). No released version is affected: the defect was introduced and fixed inside this cycle.

    With pgcolumnar.enable_ungrouped_vector_agg on, a re-executed aggregate node added to the previous execution’s running total. There were two accumulator reset sites and they had drifted: pgcolumnar_agg_specs_reset cleared nine fields including spec->i128sum, and PgColumnarReScanAggScan open coded the same loop over eight and never cleared it. The int8 kinds accumulate there, so on a rescan that total was never zeroed. Measured over four LATERAL iterations with the node asserted chosen and asserted rescanned: 800022400060000, then 1600044799119997, then 2400067196179988, where every true answer was near the first.

    The second copy is deleted rather than corrected, so the next field added to PgColumnarAggSpec is safe by construction rather than by remembering two places.

  • The documentation is brought back into line with the code, and the version check now covers the file that had drifted furthest (#753 follow-up).

    README.md said the version marker was 1.0-alpha while VERSION said 1.0-alpha3, two versions behind. It drifted because test/docs_style.sh compared VERSION against CHANGELOG.md and docs/ and did not read README.md, so the only file that was wrong was the one file the check could not see. README.md is now in that comparison, and the badge is corrected and resized for the longer string.

    Corrected against the code: docs/how-to.md said cluster takes numeric columns only, where the code accepts boolean, integer, floating-point, date and timestamp keys and rejects numeric and text. docs/installation.md said DROP EXTENSION removes columnar tables, where a plain DROP EXTENSION fails while they exist and only CASCADE drops them. docs/administration.md said there is no cache of decompressed chunk groups, where the fetch cache holds four decoded row groups per backend per statement under a 32 MB cap.

    Brought up to date: docs/roadmap.md listed five shipped features as planned, all five of their issues closed, and repeated that we read local files only. docs/configuration.md did not document ttl_column or ttl_interval. docs/features.md omitted pgcolumnar.expire and the dedup argument of pgcolumnar.parallel_copy. docs/how-to.md gains a retention recipe, so the feature is reachable by task and not only by name.

Added

  • pgcolumnar.expire drops row groups whose rows are all older than a declared retention, without reading or rewriting them (#403 item 5a). Declare the retention with pgcolumnar.set_options(..., ttl_column => 'ts', ttl_interval => '90 days'); both halves are needed, and either alone means no retention.

    This is the tractable half of the paper’s “merge-time data transformation”. The rewrites already retire whole row groups: pgcolumnar.compact drops every group that is fully deleted. Retention is the same operation with a different predicate, and the zone map already holds what decides it, so the decision is a catalog read. Nothing is decoded and nothing is rewritten, and it holds only ShareUpdateExclusiveLock.

    It is called by name and never runs on its own. It deletes rows, and an operation a user runs for maintenance must not do that silently, so it is not wired into VACUUM, compact or autovacuum. A table with no declared retention raises an error rather than reporting that it did nothing.

    A group is kept whole or dropped whole. A group holding rows on both sides of the cutoff is kept, so retention is approximate at the group boundary and errs toward keeping data. Measured on 5,000 rows in 5 groups with 1,440 rows past a three-day retention: one group dropped, 1,000 rows removed, and all 3,560 rows still inside the retention kept, including the 440 expired rows sharing the straddling group.

    The retention column must be timestamp or timestamptz.

  • pgcolumnar.parallel_copy can refuse a load it has already taken, with dedup => true (#403 item 7). A load that commits, whose acknowledgement the client never receives, is retried and the rows go in twice: measured, the same file loaded twice gives 100,000 rows and then 200,000.

    With dedup, the SHA-256 of the file is recorded in the new pgcolumnar.load_fingerprint catalog after a successful load. A later load of the same contents into the same table stores nothing, returns 0, and raises a NOTICE rather than failing. A file whose contents changed is a different load and is stored, even at the same path.

    It is off by default, because discarding rows a caller asked to store is not ordinary INSERT behavior.

    The unit is the whole load rather than a “part”. parallel_copy is atomic through 2PC, proved by a malformed row at line 50,001 leaving 0 rows and 0 prepared transactions, so parts never commit independently and a part hash would deduplicate nothing that is not already all-or-nothing.

    The check runs after every worker has prepared and before anything commits, which is the only point at which a repeat can be refused without charging every load for it. A refused load therefore does its work and discards it. The alternative, hashing before dispatch, costs 42% of a 264 MiB load; as implemented the fingerprint has no measurable cost, because the coordinator computes it while the loaders are already reading the same file.

    Three limits, all stated in docs/sql-reference.md: the fingerprint is recorded after the data commits, so a crash between them leaves data that a retry will store again; two identical loads running at once both store, because each checks the record before either writes it; and a refused load still reads and parses the file.

Fixed

  • The zone-map survival estimate reads the row-group geometry a table was written with, so a plan’s cost no longer moves with the planning session’s pgcolumnar.stripe_row_limit (#817). This is the half #806 left open; it fixed the index-fetch penalty and deliberately declined the same substitution here.

    pgcolumnar_zonemap_survival asked pgcolumnar_effective_stripe_row_limit, which answers “what limit would a write in THIS session use”. For a table with no per-table option, that is the session GUC. A table written at 2,000 rows per group and planned from a session at the 150,000 default was therefore modelled as holding ceil(20000/150000) = 1 group, the one sampled group survived, and the discount vanished. Measured on one unchanged table, varying only the plan-time GUC:

    plan-time stripe_row_limit before after
    150000, the default 306.00 61.20
    2000, the written value 61.20 61.20

    A 5x swing on a table that did not change, and at the default the pruning predicate was priced identically to a full scan of the same table – pruning had left the cost model entirely rather than merely drifting.

    Why this is now safe, and was not before. #806 declined it because the substitution then moved native_zonemap_narrow from wide=30/narrow=30 to wide=570/narrow=66, zone-map reads that scale with table width. That was the width-blind whole-group probe in the estimator, which #821 replaced with a per-column probe. The same substitution now measures wide=40/narrow=40, exactly width-independent.

    Planning-time zone-map fetches go from 1 to 10 on that fixture. The 1 was never a saving: it was the estimator examining a single group because it believed a 20,000-row table held one. Ten is the estimator sampling the ten groups that exist, bounded by PGCOLUMNAR_PRUNE_SAMPLE_GROUPS and charged per predicate column rather than per table column.

    test/doc_parallel_premise.sh carried SET pgcolumnar.stripe_row_limit = 20000 as a workaround for exactly this. It is removed, which makes that suite a removal proof: its cost-ordering check now holds only because the estimator reads the written geometry.

  • pgcolumnar.row_group.group_number is documented as one-based, which is what it has always been (#817). The column comment said “0-based row group ordinal”. A group number is the stripe id reserved from the metapage when the group began buffering, and PgColumnarInitMetapage starts reservedStripeId at 1, so there is no group 0 on any storage. Verified rather than reasoned: on a 27-group table, row_group and zone_map both report min = 1, max = 27 over 27 distinct numbers.

    This is a source comment, not a catalog one – there is no COMMENT ON for the table, so nothing reads it at runtime and native_upgrade_converge, which compares col_description, cannot see it. It is corrected because a reader believed it: the planner’s zone-map sample walked [0, ngroups) and paid for it in the defect fixed immediately below.

  • The planner’s zone-map sample is one-based, so it no longer under-prices a scan for matching the newest rows (#817).

    pgcolumnar_zonemap_survival samples row groups and asks the reader’s own skip predicates how many survive. It walked g = i * ngroups / nsample for i in [0, nsample), so it sampled [0, ngroups). A row group number is the stripe id reserved from the metapage, and the metapage starts reservedStripeId at 1: group 0 exists on no table. The first probe was always spent on a number that could not exist, and group ngroups was never probed at all.

    The wasted probe was harmless – an absent group narrows the sample rather than biasing it. The missing one was not. When the group count fits in the sample target the loop is a census, and it was a census that omitted the newest group every time, so the same predicate was priced differently according to where in the table its groups sat. Measured on ten groups of 2,000 rows, one clause each, identical row estimates, the only difference being position:

    predicate groups matched before after
    c1 > 8000 5..10, the six newest 166.89 180.24
    c1 <= 12000 1..6, the six oldest 200.27 180.24
    c1 > 17000 9,10, the two newest 33.38 60.08
    c1 <= 4000 1,2, the two oldest 66.76 60.08

    Exactly half, in the narrow pair. The under-priced half is the recency predicate this engine is aimed at: on batch-loaded time-series, WHERE ts > now() - interval '1 hour' selects the groups the sample never looked at.

  • The planner’s zone-map sample reads only the predicate columns, as the executor already does (#817).

    It called PgColumnarReadZoneMapList, which keys on (storage_id, group_number) against the four-column zone_map_pkey, so it fetched every column’s and every vector’s row from the heap and then used one column’s. pgcolumnar_native_group_can_match has asked the per-column question with a three-key probe since #314; the estimator now asks it the same way, through the same PgColumnarReadZoneMapForColumn and the same session cache. Planning-time zone_map fetches on a 30-column table go from 540 to 10, and no longer scale with table width: 30 columns and 2 columns both read 10.

    The estimator holds a #744 read session of its own, so it resolves zone_map once for the whole sample rather than once per probe. Its DEBUG1 report is labelled zone map estimate: where a scan’s stays zone map read:, because the two interleave in one backend’s log and native_zonemap_session counts scan reports to prove that a scan around an aborted one opens for itself. A scan’s line is byte-identical to what #744 shipped.

    The two halves could not ship apart. Fixing only the one-based sample makes the whole-group probe reachable at the default stripe_row_limit, where the single wasted probe had been hiding it, and native_zonemap_narrow correctly reddens at wide 90 against narrow 34.

  • PgColumnarReadZoneMapForColumn no longer discards the index oid it just cached (#817). #744 resolves zone_map_pkey once per read session and stores it; an unconditional second pgcolumnar_index_oid("zone_map_pkey") stood immediately before systable_beginscan and overwrote both that value and the no-session branch’s own lookup. The session’s opens counter never noticed, because it counts relation opens, which really were saved. Shown by poisoning the cached value to InvalidOid: with the dead store present the poison is inert (40 index fetches, 0 sequential scans, identical to the unpoisoned control), and with it removed the poison reaches the scan and forces 20 sequential scans of zone_map. A dead store draws no compiler warning, which is how it survived.

  • The cost model reads the row-group geometry a table was written with, not the geometry a write in the planning session would produce (#806). pgcolumnar.storage.row_group_limit records what the writer used and nothing read it back; the index-fetch penalty called pgcolumnar_effective_stripe_row_limit(), which returns the per-table option or else the planning session’s pgcolumnar.stripe_row_limit.

    The consequence is larger than a mis-costed table. A session that sets that GUC – before a bulk load, say – repriced every columnar table it then planned against, including tables it never touched. Measured on one unchanged table of three row groups, varying only the GUC at plan time:

    plan-time stripe_row_limit before after
    150000 775.26 775.28
    20000 113.26 775.28
    5000 36.26 775.28

    A 21x swing on a table that did not change. R is not a minor term: it sets the group count, the pages per group, the per-group decode CPU, and the decodedWidth * R test against the 32 MB fetch-cache cap, so it can be wrong in either direction there.

    One call site changed, the index-fetch penalty, which is where the defect was measured. Two others deliberately did not:

    • columnar_write_state.c still asks pgcolumnar_effective_stripe_row_limit(), because a writer deciding the geometry it is about to lay down is exactly what that function answers;
    • pgcolumnar_zonemap_survival() also still asks it. Substituting the written geometry there moved native_zonemap_narrow from 30/30 zone-map reads on a wide and a narrow table to 570/66 – reads that scale with table width, which is the property that suite exists to hold. There is no measured defect at that site and there is a measured regression from changing it.

    An explicit per-table stripe_row_limit still wins over the written geometry. This is about the planning session’s GUC, which has nothing to do with the table; a per-table option is a durable statement by its owner, and native_index_fetch_stripe_cost asserts the cost model can see it.

    Two things this leaves, stated rather than hidden. Because the survival estimate still reads the session GUC, a plan-time limit far enough below the written one can still flip a plan through that path: at 5000 the table above costs 87.79 rather than 775.07, because it stops being an index scan. And the recorded limit is one number, the last writer’s, so a table whose groups were written under changing limits is still approximated – the exact quantity is the real group count, which would be a catalog scan proportional to the number of groups on every plan.

    New suite test/cost_written_geometry.sh. It asserts planner costs rather than timings, so PGC_SKIP_TIMING does not drop it.

Changed

  • Chunk-group skip predicates are now evaluated most-selective-first, so a group that is going to be pruned is no longer probed through every other predicate’s column on the way there (#403 item 4). The skip loop returns on the first predicate that excludes, and each predicate’s first use in a group fetches that column’s zone map from the catalog, so the order decides what a pruned group costs.

    The order used was the order the ScanKeys arrived in, which is attribute order. Measured on 100 row groups with one predicate excluding 99 groups and one excluding none: 200 zone-map probes, and writing the selective predicate first in the query did not change that, because query order is not ScanKey order. The order was arbitrary with respect to selectivity rather than merely suboptimal. After: 102 probes, with the same 99 groups pruned.

    The reordering is a transpose on each exclusion rather than a sort: O(1), no statistics the reader does not already have, and it converges in one step for one selective predicate behind an unselective one. The paper’s caveat, that ordering should apply only when a highly selective predicate is present, falls out of the mechanism: a predicate that never excludes never moves, so a query with nothing selective keeps the order it started with and pays nothing.

    This cannot change an answer. The predicates are a conjunction, and the loop returns on the first exclusion whichever one that is. EXPLAIN (ANALYZE) reports Columnar Zone Map Probes, which is the only line that moves.

  • The vectorized grouped aggregate now sizes its hash table from the group estimate the planner already made, instead of growing it from nothing (#403 item 6). The table is open-addressing and doubles at 70% load from capacity 0, so a query with 200,000 groups walked 1024, 2048, … , 524288 and rehashed every live entry at each step. Measured on 2,000,000 rows with 200,000 groups: 10 allocations and 366,280 entries rehashed before, 3 allocations and 46,591 after, with identical answers.

    The table still starts at 1024 and is sized only once the data has proven it must grow, and a grow jumps toward the estimate but never past 64 times what the table has proven it holds. Both bounds are expressed in the live count rather than in a memory budget, which is the point: an earlier version of this change bounded the allocation by work_mem and so allocated 131,072 entries for a query with 47 real groups, because estimate_num_groups cannot see through a function and GROUP BY date_trunc('day', ts) reaches the node with an estimate of every row. A query whose groups fit in the starting 1024 never allocates more than it would on unpatched code. Under-estimating costs nothing new.

    EXPLAIN (ANALYZE) reports Columnar Group Table Entries Rehashed, the work done, alongside the allocation count and the table’s peak capacity, which is the memory cost. Entries rehashed rather than resizes because the two disagree: the first instrument counted resizes and reported that sizing removed 7 of 10 while it removed about a quarter of the rehashing, the work being dominated by the last two steps.

    Fixed

  • test/build_all_versions.sh now reports how many majors it built and refuses a run that built none (#809). Its verdict read only the failed flag, which only a build that RUNS and FAILS sets, so a pg_config that is not executable was skipped without touching it. A run where every major was skipped reached the end with failed=0 and printed PASSED with exit 0, having invoked no compiler.

    That is not hypothetical: the default list is /usr/local/pg15, pg16, pg17, pgsql and pg19, and a machine whose builds live elsewhere skips all five. It was recorded as a green five-major preflight for a pull request and caught by reading the body above the verdict rather than by any check.

    The script now prints built N of M before its verdict, in the same shape test/run_all_versions.sh already uses for versions run: N of M configured, and fails on zero with the invocation that names the paths. A PARTIAL run is deliberately left passing: whether three of five present should fail or warn is a judgement about how people run this, and the count makes it visible either way. A new harness_selftest case pins all three behaviours, since nothing covered this script before.

Added

  • pgcolumnar.sort_status reports sorted_kind, so the reporter can finally express the distinction the catalog has recorded since #758 (#761). It holds lexicographic after pgcolumnar.vacuum_sorted, zorder after pgcolumnar.cluster and pgcolumnar.recluster, and NULL when the table was never ordered or was ordered before the column existed.

    sort_key names the columns and says nothing about the arrangement, and a Z-order over two or more columns is not a sort on any one of them. The documented way to read the kind was to select it from pgcolumnar.storage directly. That table carries no GRANT, so only a superuser could follow that advice, while sort_status is SECURITY DEFINER and gated on require_caller_select and is therefore available to a table’s own owner.

Changed

  • The extension’s default_version is now 1.0-alpha3, and pgcolumnar--1.0-alpha2--1.0-alpha3.sql ships with it. Adding an OUT parameter changes a function’s signature, so #761 cannot be a CREATE OR REPLACE; it is the change that opens this cycle. Every other [Unreleased] entry so far is shared-library only and needs no catalog change.

    test/native_upgrade_converge.sh now exercises both 1.0-alpha and 1.0-alpha2 as starting points, and asserts that each reaches a catalog identical to a fresh 1.0-alpha3 install in function definition, ACL and comment, relation kind, column type and comment, non-base types, and foreign data wrappers. 1.0-alpha reaches it by the chain through 1.0-alpha2.

Fixed

  • The index-fetch penalty now charges decode CPU by projected column WIDTH, not by column count (#803). This is the same flatness #768 fixed for the sequential scan, in the other cost site. pgcolumnar_index_fetch_penalty’s CPU term was cpu_operator_cost * R * nproj, a count, so a 68-byte text column was charged exactly what a 4-byte int4 column was.

    Measured on two tables of identical shape whose columns differ only in type, so the decoded prefix is four columns on both arms and only the bytes move. Read out of an instrumented build, the decode CPU term was 200.00 on both arms across an 8.6x difference in decoded width (16 B against 138 B). The same ~39,000-row index fetch took 4,385 ms on the narrow table and 88,352 ms on the wide one, a 20.15x difference against a modelled 1.49x. Cost per millisecond spanned 13.4x, inside the 13x-22x #768 measured for the scan path.

    The consequence was an inverted ordering rather than a uniform under-charge. A wider decoded prefix makes every row-group decode dearer, so a wide table should abandon per-row fetches sooner; priced by count it did the opposite. The model kept the index up to 120-161 rows on the wide table and only 40-60 on the narrow one, while the measured crossovers are 40-120 wide and 120-279 narrow. At 120 rows the wide table’s chosen index plan measured 239.6 ms against an 82.9 ms scan.

    pgcolumnar_scan_decode_shape now accumulates decode units over the prefix it already walks, using the same reference width as the scan’s pgcolumnar_projected_decode_units, and the penalty charges those units. Substituting pgcolumnar_projected_decode_units itself would have been wrong: it sums the columns a query references, while the fetch decodes the prefix up to the highest referenced column (#363), and the two differ on exactly the shape #363 exists for.

    An all-int prefix is priced exactly as before, because a 4-byte column is one unit: the narrow arm’s flip point is unchanged at 40-60 rows. The wide arm moves to 7-15. Both arms now stop earlier than the measured crossover, which is the direction the model already erred on the narrow arm before this change; the

    376 bound and the clustered-index and point-lookup guards in

    test/analyze_stats.sh are unaffected and still pass.

    test/index_fetch_penalty_width.sh pins the ordering from the plan rather than the clock, so PGC_SKIP_TIMING does not apply to it. On the unfixed build its seven premises pass and the ordering check fails with INVERTED (wide 138B fetches to 120 rows, narrow 16B only to 40).

  • pgcolumnar_fetch_group_slot’s header promised a NULL return that neither the function nor its callers implement, and following it segfaults (#795). The comment read “Returns NULL when nothing should be cached, in which case the caller decodes into its own scratch context exactly as before”. There is no such path. The function has two returns, return e on a hit and return victim on a miss, and victim cannot be NULL: the branch that runs when every slot is live picks a least-recently-used entry. Both callers dereference the result immediately, one through entry->firstRowNumber in the geometry check and one through MemoryContextSwitchTo(entry->cx), neither guarded.

    So the sentence did not describe an unimplemented option, it described a trap. Forcing the documented return on an assert build of PostgreSQL 18.4 produced signal 11: Segmentation fault in a backend. Nothing in main reaches it – the function never returns NULL today, and the same fixture on a clean build is correct – but the comment invites the change that does.

    The comment now states the guarantee the code makes, that a slot always comes back and a miss returns a reset entry for the caller to fill, and records that a “do not cache this one” policy needs a scratch-context path in the callers before it can be expressed as a NULL. An Assert(entry != NULL) at each call site holds that to be true rather than leaving it to the comment, which is what rotted. With the Assert in place the same injected NULL return reports TRAP: failed Assert("entry != NULL"), File: "src/columnar_reader.c", Line: 3888 instead of faulting, so the next person to try it gets the line rather than a core file.

  • The columnar scan’s decode cost is now charged by projected column WIDTH, not by column count (#768). #503 gave the scan a per-value decode term, which fixed “nine columns are priced like one”; it counted columns, so a 324-byte text column was charged exactly what a 4-byte int4 column was. Measured on 4,000,000 rows, four 324-byte text columns were priced at 163,422 while taking 5875 ms, against 206,524 and 335 ms for eight int columns: priced below, and 17.5x slower. Cost per millisecond across those projections spanned 22x before the change and 1.7x after it, with an int-only projection priced exactly as before.

  • The shared test cluster no longer sets pgcolumnar.unique_lock_buckets (#799). test/lib.sh wrote 100003 into the cluster nearly every suite runs against, where the shipped default is 128, so the whole tree ran 781x above shipped behaviour to serve one suite that already sets the value on the cluster it builds itself. The visible symptom was a 20,000-row insert into a columnar table with a unique index failing with out of shared memory and a hint to raise max_locks_per_transaction, which reads as a product defect and is not one. A new harness selftest keeps the shared cluster config free of pgcolumnar.* GUCs; PGC_EXTRA_CONF remains the per-suite mechanism.

  • The one/many fetch cost guard in test/native_fetch_cache.sh now rebuilds its fixture before every reading, and asserts the group geometry at each one (#801). The guard could not fail. The UPDATE it times does not rewrite the row group it reads: it marks the rows deleted and appends them as a new group, so fc_one goes from one group of 20,000 rows to that group plus one of 2,000, and from the second reading on the rows it targets sit in the small appended group. That is the geometry the fc_many control has, so the arms stop differing after one reading, and min3 returned one of the cheap post-rewrite readings every run. Measured against a build whose fetch cache retains nothing, so every fetch re-decodes: the guard read 0.98x and PASSED, where rebuilding reads 5.89x and 6.13x on two runs and fails as it should. The #353 guard in the same file fails correctly on that build, which is what shows the ablation was real. A per-fetch decode counter says it in rows rather than milliseconds: fc_one decodes 40,000,000 rows on the first reading and 4,000,000 on the next two, and 40,000,000 on all three once the fixture is rebuilt.

    The new premise records the group count at each reading. Both faults it covers were proved by mutation, each changing one thing: dropping the rebuild records 1 2 3, and an INSERT that populates nothing records 0 0 0. The timing check passed in both mutants, at 0.77x and 1.14x, so the premise is what separates either fault from a green suite.

  • The fetch cache cost guards in test/native_fetch_cache.sh now assert that they reach the per-row fetch path, and set pgcolumnar.enable_index_fetch_penalty = off so that they do (#797). They had not exercised the cache since the penalty landed: enable_seqscan = off and enable_bitmapscan = off do not disable the columnar custom scan, so the planner answered these queries with a group scan and the guards timed a different mechanism while staying green. The #353 guard was written 83 minutes before the penalty existed. With the path restored the two guards read 2.00x and 6.42x, against 0.98x and 1.21x before, and the 6.42x reproduces the 5.8x recorded in the suite’s own comment when the guard was written.

  • sum() and avg() over numeric no longer take the ungrouped vectorized path, where they were slower than the ordinary Agg (#785). This closes the last of the three families pgcolumnar.enable_ungrouped_vector_agg names. bigint was fixed by giving it a cheap accumulator (#786); numeric cannot have one, because its input already carries a scale.

    Measured on 8,000,000 rows, interleaved, minimum of seven, plan and answers asserted on both arms:

    query before after
    sum(numeric) 1.10x slower 1.04x faster
    sum(float8) 2.73x faster 2.74x faster
    sum(numeric), sum(float8) 1.00x, no gain 1.01x, unchanged

    The refusal is unconditional, and the mixed case is why. Classification is all or nothing, so refusing numeric refuses the whole node, and the obvious worry is that a mixed query loses the float win with it. It does not. Measured on one cluster with only the installed library changing, a mixed query was being actively harmed:

    query numeric admitted numeric refused
    sum(numeric) 391.0 ms 351.0 ms 1.11x faster
    sum(numeric), sum(float8) 412.4 ms 367.6 ms 1.12x faster
    sum(float8) (control) 78.8 ms 77.2 ms flat

    So the mixed case is an argument for the refusal rather than a cost of it.

    It is also not conditional on the data. The penalty holds whether or not chunk-group pruning happens: 1.14x with 25 of 27 groups removed and 1.56x with none.

    The refusal is placed where the scan-fold path is chosen rather than in pgcolumnar_classify_aggref, which is shared with the grouped path. The grouped path is a different implementation and has not been measured for these kinds, so this narrows only what was measured.

Changed

  • test/native_fetch_cache.sh’s three cost guards now assert through check_ratio instead of computing the comparison in shell and passing yes/no to check. They were hand-rolled while they used check_timing, which takes a scalar; once #792 made them ordinary checks the ratio helper became available, and it is strictly better.

    check_ratio refuses a zero on either side. The hand-rolled form guarded only the denominator, so a numerator timing at 0 ms, which means the measurement fell below timer resolution, passed silently as a ratio of zero. Demonstrated in both directions with the same injected reading:

  check_ratio:  FAIL  ... a side of the ratio is zero, so nothing was measured: a=[0] b=[26]
  hand-rolled:  PASS  ...

It also compares as a float rather than truncating integer division, and prints the ratio with both sides, so a verdict now reads (1.39x, bound 3x, from a=39 b=28) rather than a bare yes.

One boundary moves by a hair and the comment says so: the old form failed at a ratio of exactly 3.0 and check_ratio fails above it. Nothing measured on these shapes is near that.

Changed

  • test/native_fetch_cache.sh’s three cost guards now run in CI. They were written with check_timing, which PGC_SKIP_TIMING drops, and that suite is not in is_timing_suite – so the suite ran, reported PASSED, and skipped all three, including the two that guard named regressions (#353, #359). Three cost guards in no automated gate (#792).

    They were already the exempt shape by test/cancel_decode.sh’s argument: two readings taken back to back in the same run, which move together under load. But that argument has a second half – “the best of three readings, not the average … an average would let one descheduled run widen the ratio on its own” – and these took a single reading per side. Measured on six busy cores of an eight-core box before the change, the one/many ratio reached 2.39 against its bound of 3 on one run. Unguarding them as they stood would have traded a check that is skipped everywhere for one that is flaky somewhere.

    So each side is now the minimum of three, and then they are ordinary checks. Under the same load afterwards, four runs: one/many 0.84 to 0.93, wide/small 0.98 to 1.00, over/under 1.18 to 1.23, against bounds of 3, 5 and 12. The suite costs about 0.9 s more (4.4 s to 5.3 s, build excluded).

    A repeated measurement must be idempotent, and this one was not: upd_ms updated with v = v + 1, which is correct once and wrong three times. The suite’s own correctness arms caught it – they assert v = id + 1 – so the update now sets v = id + 1, which is the same work per row and true after any number of runs.

    check_timing’s skip message said “wall-clock ratio” on a helper that takes a scalar, so it misdescribed every skip it printed. Its one remaining caller, test/native_cancel.sh, is in is_timing_suite, so the driver names that suite as skipped rather than reporting a pass over a dropped subject.

  • sum() and avg() over bigint now take the batch fold, which closes the filtered case that #786 left behind (#755 question 3). Before this, a filtered sum(bigint) on the vectorized path was slower than not using it, while sum(float8) on the same fixture was more than twice as fast. That asymmetry was the whole evidence for the question, and it was the batch fold rather than the aggregate: the int8 kinds folded row at a time while the float kinds folded column at a time.

    Measured on one cluster with only the installed library changing, 8,000,000 rows, values chosen to defeat run-length and dictionary encoding so the fold is not answered from encoded runs, minimum of seven, answers identical:

    Before After
    sum(bigint) 133.5 ms 92.0 ms
    sum(bigint) WHERE ... 228.2 ms 99.4 ms
    avg(bigint) WHERE ... 226.2 ms 98.7 ms
    sum(float8) WHERE ... (control) 85.2 ms 86.6 ms

    The float control does not move, which is what says the change is specific to the int8 kinds.

    It is a two-line change because #786 did the work: the fold reads a column at a time and then accumulates per value, so a kind is foldable when its accumulate is cheap, and since #786 the int8 kinds accumulate into an int128 and allocate nothing. This does not make them parallel-eligible. pgcolumnar_parallel_agg_ok is a separate predicate with its own callers, and the int8 kinds stay out of it: an int128 running total is not a partial state a core Finalize can combine.

    sum/avg over numeric are unchanged and still slower on this path. They cannot use an integer accumulator, and that residue stays on #785.

Changed

  • check_ratio_timing is renamed to check_ratio_needs_quiet_machine, and the distinction it encodes is now written where it is decided (#787). The helper was named for what it measures, a ratio of timings, rather than for what it does: both ci.yml and nightly.yml set PGC_SKIP_TIMING, so a check written with it runs in no automated gate at all.

    Two kinds of timing assertion need different treatment and only one needs that. A ratio against an absolute or cross-run baseline can be distorted by a loaded machine. A ratio whose two arms are measured back to back in the same run and compared by minimum cannot, because both readings move together under load, and test/cancel_decode.sh had argued exactly that for its own ratio and run in CI on the strength of it. An author holding the safe shape reached for the name that matched their units and lost the check silently, which is how #786’s guard came to be skipped everywhere until it was moved to check_ratio.

    No behaviour changed. The two call sites, both in test/planner_choice_quality.sh, keep the guard they had; that suite is in is_timing_suite, so the driver already names it as skipped rather than reporting a pass over a dropped subject. What changed is that the reasoning now sits beside the helper rather than being re-derived in three suite headers – bloom_sizing for a size, native_fetch_bigcap for buffers, cancel_decode for a same-run ratio – which is the recurrence #253, #254 and #764 each fixed for one suite.

  • The documentation style gate is now based on ISO 24495-1:2023, *Plain language

    • Part 1: Governing principles and guidelines*, in place of ASD-STE100. The project’s writing rules cite two standards and no others: ISO 24495-1:2023 for language, and ISO 82079-1 for the structure of instructions.

    No check changed. The gate enforces the same four rules over the same files, and docs_style runs the same nine checks. What changed is the basis each rule is attributed to, and the honesty of that attribution:

    • the 25-word sentence limit and the idiom list come from the standard’s “understandable” principle;
    • the two dash rules are this project’s typographic house rules and are now labelled as such, because ISO 24495-1 does not ask for them.

    The 25-word figure is named as this project’s measurable proxy rather than a number quoted from the standard, which does not give one. test/ste_check.py is renamed to test/plain_language_check.py so the file does not assert a standard the project no longer cites.

    Conformity is still not claimed, and the reason is now the accurate one. Only one of the four governing principles has mechanically checkable content, and the standard’s own test for another is that a reader acts on the document successfully, which no checker performs.

Fixed

  • sum() and avg() over bigint are no longer slower on the vectorized path than off it (#785). pgcolumnar.enable_ungrouped_vector_agg describes itself as the fast path for “sum/avg over int8/float/numeric”, and for bigint it was a pessimisation: 2.03x slower for sum, 1.91x for avg.

    The path converted every value to numeric and called numeric_add per row, which is two allocations and a full numeric addition for each row. A profile of it was 13.0% make_result_opt_error, 9.3% add_abs, 8.1% init_var_from_num and 8.5% AllocSet alloc and free: the numeric machinery, not the aggregate. PostgreSQL’s own bigint accumulators are 128-bit and convert once, and this now does the same.

    Measured on 8,000,000 rows, interleaved, minimum of seven, with the plan and the answer asserted on both arms:

    Before After
    sum(bigint) 2.03x slower 1.65x faster
    avg(bigint) 1.91x slower 1.74x faster

    float4 and float8 were already large wins on this path (3.18x and 2.88x here), so the setting is now a gain for three of the four families it names. numeric remains about 1.14x slower and is unchanged by this: its input is already numeric with a scale, so it cannot use an integer accumulator. That residue is recorded on #785 rather than fixed here.

Changed

  • docs/configuration.md now documents when encode_effort changes anything at all, and what it costs on read (#768). The section described the setting as a trade between load speed and compression ratio, which reads as though the choice has no consequence after the load.

    The writer builds the FSST symbol table, then compares the result against the same data without it after the block codec has run, and keeps the table only when the saving clears pgcolumnar.fsst_min_gain_percent, default 5.

    That decision depends on the data and cannot be predicted. It is made for each column chunk, and two people who both write “text-shaped” test data get opposite answers. In four shapes measured for this entry the table was dropped every time, so full and fast produced byte-identical storage and equal read times. A second set of measurements found shapes where it survives and full is 31.8% and 9.6% smaller, at read times of 1.08x and 0.99x, because the extra decoding is paid back by reading fewer bytes.

    So the section gives the reader a way to measure their own column rather than a verdict. An earlier draft of this entry concluded that at default settings the option changes nothing but write time. That was true of the shapes it was measured on and false in general, and a reader who acted on it could have paid a third more storage.

    The read-cost table is scoped to compression = 'none', which is the only setting that keeps the table unconditionally, and is labelled as isolating the encoding rather than describing a default installation:

    Text shape full fast fast is
    128-char hex 267.6 ms 143.9 ms 1.86x faster
    URL-shaped text 131.5 ms 96.0 ms 1.37x faster

    Those replaced 2.69x and 3.84x, which were measured before the reader learned to decode a symbol with one machine word. The section names the reader version its figures came from.

Fixed

  • ORDER BY no longer returns unordered rows after a column rename, and neither ordering self-gate skips work it must do (#778). Both follow from one cause: nothing maintained the recorded sort key across a rename. pgcolumnar.storage records the applied sort key as column names, and both gates compare that stored list against the current attname: pgcolumnar.vacuum_sorted’s gate and the online pgcolumnar.recluster’s. Nothing maintained the mark across a rename.

    A three-statement swap made the stored name resolve to a different column:

  ALTER TABLE t RENAME COLUMN a TO tmp;
  ALTER TABLE t RENAME COLUMN b TO a;
  ALTER TABLE t RENAME COLUMN tmp TO b;
  SELECT pgcolumnar.vacuum_sorted('t','a');   -- reported success, did nothing

The gate reported “already in this order” about a column that was never sorted, and the table was left neither ordered nor reclaimed with no error raised. For vacuum_sorted that is the failure the gate exists to prevent, reached through another door. recluster returned 0, which a scheduler reads as “nothing to do”.

The mark is renamed rather than cleared. A rename does not move data, so the ordering is still true of whichever column now carries the name, and following the rename keeps a real optimisation instead of discarding it on every rename. It composes through the swap above: {a} to {tmp} to {b}, which is the column the rows are ordered by. A rename that touches no sort-key column does not rewrite the storage row at all.

The wrong-answer half is the more serious one. The sorted-pathkey code (#751) reads the same mark, so a stale mark made the planner claim an ordering that did not hold: after the swap above, SELECT a FROM t ORDER BY a planned with no Sort node and returned 1,990 descents out of 2,000 rows.

The rename cascades to inheritance children and partitions, which are separate relations with their own storage and their own marks, so the fix walks every columnar descendant. PostgreSQL refuses ALTER TABLE child RENAME COLUMN with “cannot rename inherited column”, so for a columnar partition the parent is the only route a user has, and a fix that looked only at the relation named in the statement would never have fired for it at all.

The declared key in pgcolumnar.options is maintained the same way, and it is not optional once the mark is. Before, both were consistently stale; renaming only the mark would make the two catalogs name different columns, so a bare pgcolumnar.vacuum_sorted(t) would silently rewrite on a different physical key than it did the day before, with no change to the call and no change to the declared intent. options.sort_by holds names deliberately, because options is the catalog carried through pg_dump where attnums renumber and names do not, so maintaining them across a rename is the correct repair rather than a workaround.

One behaviour improves as a result. The sorted-pathkey claim used to be refused after a rename, because the recorded name stopped resolving; it now survives, so ORDER BY on the renamed column plans no Sort. That claim is sound for the same reason the mark is: the rows really are still in that order.

Changed

  • The harness now refuses a pgcolumnar.set_options call whose value the function will reject. set_options raises on an out-of-range limit, so a suite that calls it and discards the output runs on default limits while the script reads as configured, and every later assertion is about a fixture that was never built. Found in review probes that passed stripe_row_limit => 500, which errors with “must be at least 1000”.

    The naive form of that guard would be wrong. Measured on this tree before writing it: 182 set_options calls and exactly three out-of-range literals, all in test/audit.sh, all deliberate, all wrapped in expect_error because rejecting them is what that suite tests. A guard flagging every out-of-range value is wrong on three of three. The discriminator is whether the call’s result is inspected or discarded, so calls passed to expect_error are skipped and exactly the silent class remains.

    Line continuations are joined before matching, because two of those three calls are backslash-continued and a line-based scan sees neither the value nor the expect_error. That direction fails safe; the same blindness fails open on a real multi-line offender. Proved: without the join, those three become false positives.

    The guard also asserts that it swept every file containing such a call, not merely that it swept something. Its globs are test/*.sh and bench/*.sh, which do not match test/selftest/*.sh or test/pbt/*.sh, so it was complete by accident of the tree’s current shape rather than by construction. Comparing the files that contain a call against the files actually read fails loudly the day one appears anywhere new, including a directory nobody predicted, which a wider glob cannot do.

Changed

  • docs/configuration.md now says why pgcolumnar.enable_group_vectorization is off by default, which #755 records as not visible from outside the code. Measured rather than reasoned: on 4,000,000 rows in 200,000 groups the setting is worth 781.0 ms to 403.7 ms with identical answers, so performance is not the reason. That is the minimum of seven interleaved pairs on a non-assert build; an earlier figure here ran all of one arm and then all of the other, which on a contended host attributes drift to whichever arm ran second. An assert build measures about 2.2 instead, because AllocSetCheck runs per context RESET and the ordinary Agg resets far more contexts than the vectorized fold, so the skew lands one-sidedly on the slower arm. src/columnar.h also called the cap a plan-time one, which contradicted the code and the new section. The reason is the failure mode. The grouped path builds a hash table that does not spill, pgcolumnar.groupagg_max_groups bounds it, and the cap is checked during execution against the real group count because the plan is fixed by then and cannot fall back to an ordinary Agg. On a table with 1,500,000 distinct keys the same query succeeds with the setting off and raises 54000 grouped vectorized aggregate exceeded pgcolumnar.groupagg_max_groups with it on. A default would carry that to tables nobody chose it for, where the failure arrives when the table grows rather than when anything changes.

  • The reader now reuses one row-group buffer instead of allocating and freeing one per row group (#768). It materializes the whole group into a single buffer, sized stripe_row_limit x row width, which at the default stripe_row_limit = 150000 is about 19.8 MB for a 128-character text column. Allocated per group that is far past ALLOC_CHUNK_LIMIT, so it is its own malloc block, glibc returns it to the kernel when the group context is reset, and the next group faults every page back in.

    Measured on the same cluster, the same table and the same query, with only the installed library changing (1,000,000 rows of 128-character text, stripe_row_limit = 150000, vectorized paths off so the scan really decodes):

    minor faults per query time
    before 57,846 143.0 ms
    after 4,838 85.8 ms

    The trade is resident memory: the buffer stays at the largest group’s size for the life of the read state rather than being returned between groups. Only pages a scan actually touched are ever resident either way, since a projected read still reads only the columns it wants, so the cost is bounded by what the scan already touched.

Added

  • The documented CDC recipe is now tested end to end, and the decision behind it is recorded rather than implied (#754). test/logical_decoding_source.sh already pinned the limitation: a columnar table emits no decodable change. Nothing pinned the workaround docs/user-guide.md offers in its place, which is to capture rows into a heap table with a row trigger and decode that.

    That recipe makes four factual claims, none of which was checked. Row triggers fire on a columnar table and see the same rows a heap table would. An UPDATE arrives as one UPDATE rather than as the storage’s internal delete and insert, which is the claim a CDC consumer would be broken by and the one most likely to be false. The capture is transactional. The capture table decodes. test/logical_decoding_cdc_recipe.sh EXTRACTS the recipe from the guide at run time and executes it, so the guide is the thing under test, and asserts all four, plus the guide’s warning that FOR ALL TABLES really does pick up the pgcolumnar schema, and its cost claim of one heap row per changed row.

    docs/limitations.md now states plainly that this is a decision and not an open item: emitting a decodable change for a columnar write needs a WAL record type carrying tuple structure for columnar data, which is a new WAL semantic and so out of scope by the same rule that keeps the extension installable on a stock server.

  • pgcolumnar.vacuum_sorted() now self-gates: when the relation is already exactly the requested lexicographic run with nothing appended and nothing to reclaim, it skips the rewrite instead of materializing every live row through a tuplesort again. pgcolumnar.recluster() has had the ordering half of this since #415 so a scheduler can call it speculatively; design/ISSUE_415_AUTOVACUUM.md promised the mirror here.

    The mirror is not a copy, and that is the whole of the change. vacuum_sorted has two jobs: it orders the live rows and it physically reclaims deleted-row space. The obvious gate – “is it already in this order”, which is complete for recluster – would answer yes on a relation that is in order and full of dead rows, and silently stop reclaiming. Ported unchanged into a tree carrying it, that gate leaves 10,000 deleted rows stored where the un-gated call reclaims all of them, with no error and no report.

    So the skip condition is “already in this order AND there is nothing to reclaim”: a lexicographic run (a zorder run over the same columns is not this order, because Z-order over two or more columns is not a sort on any one of them), over exactly these columns in this order, covering every row group, with no deleted row and no empty group. Anything else falls through to the full rewrite, so re-sorting by a new key, re-sorting a Z-ordered table, and reclaiming all still work.

  • The columnar scan now tells the planner the order a sorted rewrite left the rows in, so ORDER BY on that key plans no Sort. Every pathkeys field in the tree was NIL, so a table that pgcolumnar.vacuum_sorted had physically ordered still paid a full sort to be read in that order, and ORDER BY ... LIMIT n could not stop early.

    Measured on 4,000,000 rows in 27 row groups, vacuum_sorted('t','k','j'), sort_status reporting key {k,j}, 27 sorted groups, 0 appended, 0 inversions on k in scan order. Instructions retired by one pinned backend, same query, pgcolumnar.enable_sorted_pathkeys off against on, and both arms return byte-identical answers:

    query Sort (off) no Sort (on) ratio
    ORDER BY k, j LIMIT 10 5,545,800,871 14,133,342 392x
    ORDER BY k, j OFFSET 3999990 8,395,455,193 3,392,835,665 2.47x

    The second row is the whole relation consumed, so it is the sort itself: 2,099 instructions a row against 848, about 1,251 a row that the sort was costing. The first is the early stop the sort made impossible. Re-run with the arm order reversed the four figures reproduce within 0.8%.

    A claim the rows do not satisfy is a wrong answer, not a slow plan, since the Sort that would have fixed the order is gone. The claim is therefore refused unless all of: the last ordering rewrite was lexicographic and recorded itself as such (#758), so a Z-order run and an unknown one are refused rather than guessed at; every row group lies inside the recorded run, so one appended or updated row retracts it; no sort column is collatable; and the requested ordering is a prefix of the recorded key, ascending, NULLS LAST, which is what the rewrite’s own tuplesort applies.

    Collatable sort columns are refused because only the column names are recorded. ALTER TABLE t ALTER COLUMN k TYPE text COLLATE "en_US.utf8" on a column already text COLLATE "C" needs no transformation, so PostgreSQL updates pg_attribute and leaves every row where it is: same storage, mark intact, ordering silently a different one. A text sort key therefore gets no ordered path. Lifting that needs the collations recorded beside the names.

    The claim lives in a plan, and no catalog object the plan cache watches changes when rows are appended, so a group written outside the run now invalidates the relation’s cached plans. Without it a prepared ORDER BY k NULLS LAST LIMIT 5 answered from the ordered run alone after 600 rows were appended below it. It fires once per row group and only on a relation that has a mark.

    Only the serial scan carries the claim. The parallel partial path and the projection path keep NIL: workers finish in any order, and a projection is a separate layout with its own sort key.

    A query that could not use an ordering does not pay to find one out. Deciding whether to claim reads the whole row-group list, and a query with no ORDER BY and no mergejoinable clause would have had any claim discarded at the end anyway, so has_useful_pathkeys is asked first. Measured as buffers touched during planning, on 1,000,000 rows in 1,000 row groups, a relation marked lexicographic, SELECT count(*) FROM t WHERE j = 3: 142 with the feature on against 121 with it off before, and 121 against 121 after. The query that can use the ordering still reads, which is what stops that from being satisfied by a function that reads nothing.

    New GUC pgcolumnar.enable_sorted_pathkeys, on by default. New suite sorted_pathkeys, 108 checks. (#751)

  • A predicate on date_trunc(unit, ts) now drives chunk-group skipping for the ORDERED comparisons too, not only equality. #739 inverted = and declined every other operator, so date_trunc('day', ts) >= '2024-02-01' still read every chunk group, which is the shape a time-series filter usually takes. Monotonicity is what makes the ordered operators invertible, so they come from the same unit table for one scan key each rather than equality’s two. Measured on the existing 500,000-row time-clustered fixture, chunk groups removed by the filter, date_trunc form against the explicit range it is equivalent to:

    predicate before after the equivalent range
    date_trunc('day', ts) >= c 0 30 30
    date_trunc('day', ts) > c 0 31 31
    date_trunc('day', ts) < c 0 19 19
    date_trunc('day', ts) <= c 0 18 18

    Two cases carry their own risk and their own arms. With the constant on the LEFT the strategy has to be commuted, because c < f(ts) is f(ts) > c; for equality that was a no-op, which is why it never came up before. And an untruncated constant, which equality treats as unsatisfiable, is satisfiable here and moves the bound to the next bucket boundary instead of emptying the result. timestamptz stays declined for the reason #739 gives. (#403)

  • A predicate on date_trunc(unit, ts) = constant now drives chunk-group skipping. A zone map holds ts, so a predicate about a function of ts could exclude nothing and the scan read every group. It is rewritten to a range on ts, which prunes with the machinery that already exists. Measured on 500,000 rows clustered by time in 50 chunk groups: the equivalent explicit range read 2 groups, the date_trunc form read all 50, and it now reads 2. The derived keys are conservative, so the executor still applies the original clause. Only the timestamp form is rewritten: date_trunc on timestamptz truncates in the session time zone, so a key frozen at executor start could be wrong if the zone changed. Units outside a known list, and a constant that is not itself truncated, are declined rather than approximated. A new suite, preimage_rewrite, measures the pruning and pins each declined shape. (#403)

Added

  • The vectorized aggregate now accepts a target list that CONTAINS aggregates, not only one that IS them. It required every entry to be a bare aggregate, so an entry that merely contained one fell off the path entirely, and on an unfiltered columnar table that is the difference between answering from the zone maps and scanning the whole relation. Measured on 8,000,000 rows:

    query before after
    count(*)::text 634.1 ms 0.057 ms
    avg(a)+avg(b) 329.5 ms 0.504 ms
    max(a)-min(a) 245.4 ms 0.430 ms
    round(avg(a)::numeric, 2) 230.9 ms 0.488 ms

    Two to four orders of magnitude, and every shape that was losing is ordinary SQL: a cast on a count, a difference of two aggregates, a rounded average. count(*) itself was already 0.030 ms, so wrapping it in a cast was costing a full scan of the table.

    The node is unchanged and still emits one bare aggregate per output column. The aggregates are pulled out of the target list, the node produces those, and a projection above it computes the expressions, which is how core’s own Agg relates to an upper target. A target list that is already exactly the aggregates keeps its previous plan with no projection added, so nothing that worked before is planned differently.

    The parallel arm needed no change and never did: it uses core’s partial grouping target, which is bare aggregates however the final target is shaped. Refusing the shape in the serial gate was killing the parallel arm with it.

    Unsupported aggregates, aggregates over expressions, and grouped queries decline exactly as before. A filter still routes to the scan-fold path behind pgcolumnar.enable_ungrouped_vector_agg, which is off by default; with it on, a wrapped aggregate takes that path too. New suite vector_agg_tlist_shape. (#755)

  • A suite for the claim under docs/limitations.md’s “Replication and backup”: that a columnar table is not a logical decoding source. The whole of #754 rests on that sentence and nothing asserted it. The only other logical-replication coverage, logical_subscriber, tests the opposite direction, a heap publisher into a columnar subscriber.

    Measured through a test_decoding slot created before any row is written, on a heap table and a columnar table of identical shape holding the same 50 rows:

    decoded changes
    public.heap_t (control) 50
    public.col_t (columnar) 0
    pgcolumnar.* (internal) 8

    The heap control is the load-bearing arm, because zero decoded changes is also what a broken slot, a mis-built plugin or a query against the wrong slot looks like. The second documented behaviour is asserted too: the slot is not silent, it carries pgcolumnar’s own catalog writes, and that churn scales with row groups and chunks rather than rows. 100 times the rows gives 8 records against 10, not 800. New suite logical_decoding_source. (#754)

Fixed

  • docs/limitations.md’s account of parallel scans stated two things that are not true, and gave a cause the arithmetic does not support. It said a columnar scan “ships fewer, already-decoded rows” than a heap scan, and that “a heap scan on the same shape also flipped” at about half the default parallel_tuple_cost.

    Re-measured on 4,000,000 rows in a 14 column table with a heap twin holding the same rows. The two plans ship the same number of rows, 999,987, so the Gather charge is identical at 99,999; what differs is tuple width, three projected columns against fourteen. And the heap does not flip at half the default: columnar turns parallel at 0.060 and the heap at 0.030.

    The effect itself is real and larger than recorded. Over the eleven query shapes where the planner chose the serial scan, the parallel plan ran 1.8 to 2.5 times faster on the median, and in every shape the slowest parallel run beat the fastest serial run, by 1.45 times at the narrowest margin. “Narrow” turns out to mean narrow in columns. Two mechanisms produce it, both now stated: parallel_tuple_cost is charged per row and is blind to width, overstating a narrow row by about 1.9 times at a constant volume of data shipped; and the columnar scan cost is two fifths to seven tenths of what its real time implies, by an amount that is largest on the narrowest projection. (#753)

  • Both eager ordering rewrites discarded the ordering they had just applied. pgcolumnar.storage.sorted_by and sorted_kind exist to record which ordering a rewrite left behind, and the base schema has always specified them as 'zorder' (recluster/cluster) or 'lexicographic' (vacuum_sorted). Only the online recluster wrote them: both eager paths reached the catalog through one function that passed NIL and NULL, and the string lexicographic had never been written to the catalog at all.

    The two eager layouts were therefore identical in the catalog, and pgcolumnar.sort_status falls back to the declared options.sort_by when the recorded key is NULL, so both reported the same thing. Measured on 5,000 rows with sort_by => ARRAY['k'] declared on each:

    rewrite sorted_kind sort_status.sort_key inversions on k
    vacuum_sorted('t','k') was NULL, now lexicographic {k} 0
    cluster('t','k','j') was NULL, now zorder was {k}, now {k,j} 930

    Z-order over two or more columns is not a sort on any one of them, so the second row reported an order the rows were not in. A single-column key cannot show this, because single-column Z-order is lexicographic order.

    Two consequences go with it. sort_status now reports the applied key rather than the declared one on an eagerly clustered table, which is a visible output change for anyone reading that column. And #415’s recluster self-gate requires sorted_kind = 'zorder', so after pgcolumnar.cluster('t','k','j') it could never fire: an immediately following pgcolumnar.recluster('t','k','j') on an already-clustered relation paid a full rewrite of every group. It now returns 0 and rewrites nothing, which is the case the gate was written for.

    A storage ordered by an earlier build still has NULL in both columns and still falls back to the declared key, until its next ordering rewrite. A reader that must not be wrong about the physical order should require sorted_kind = 'lexicographic' from pgcolumnar.storage and treat NULL as unknown, rather than read sort_status. An unsorted rewrite continues to claim no ordering. New suite eager_ordering_record. (#758)

  • pgcolumnar.vm_selftest and pgcolumnar.vm_is_visible accepted any relation, including a plain heap table. vm_selftest does not only inspect the visibility map, it writes an all-visible bit into it, and a wrongly set bit is how an index-only scan skips the heap visibility check and returns rows it should not. Both functions now refuse a relation that does not use the pgcolumnar access method, with SQLSTATE 42809, which is the same code the other C entry points use for that condition. Both also check ownership before opening the relation rather than after, so a caller who does not own the table is turned away before it can request a lock. test/vm_privilege.sh is rewritten around the change: the ownership boundary is asserted on a columnar table, where it still applies, and a heap table asserts the stronger new guarantee that no caller reaches its visibility map at all. (#748)

  • Five entry points requested a relation lock before checking that the caller owns the table. pgcolumnar.add_projection takes a ShareLock, and drop_projection, recluster, compact_rewrite and compact take a ShareUpdateExclusiveLock; each opened the relation first and validated ownership after. An unprivileged caller could therefore queue in the lock manager on a table it has no rights to, blocking readers and writers until it was refused. Measured against a held AccessExclusiveLock: the call reached a four second lock timeout before the ownership error, where it now returns the ownership error immediately. Ownership is now checked before the lock, which is the ordering pgcolumnar.vacuum, vacuum_sorted and cluster already used. drop_projection also reported whether an arbitrary relation was columnar to a caller who does not own it; it now reports only that they are not the owner, matching the four entry points beside it. (#749)

  • The columnar scan resolved pgcolumnar.zone_map once per chunk group per predicate column instead of once per scan. Every group a predicate could exclude ran a relation open with a lock and two catalog name lookups, then closed again; the reuse cache that the write path uses for the same catalogs is gated on a flag only the write path sets, so the read path never reached it. The scan now holds the relation and the resolved index for its own duration. Measured at 640 chunk groups with one predicate column, the same binary with the session bypassed, backend instructions pinned to one PMU and normalised by completed queries: 29,694,459 per query before and 26,676,590 after, a saving of 4,715 per chunk group and 1.113x on the query. That is 15% of the cost #744 measured for locating the surviving groups at that size, so the systable index probe, which stays inside the loop, remains the larger part. Buffer counts are unchanged, which is the expected shape: a catcache or relcache lookup reads no buffers once warm, so the buffers a probe costs are the index scan. (#744)

  • pgcolumnar.set_options refused a non-columnar relation with SQLSTATE P0001 rather than 42809. plpgsql’s RAISE EXCEPTION defaults to P0001 unless an ERRCODE is given, and the guard shipped without one, so the identical message relation "..." is not a columnar table carried one SQLSTATE from this function and another from the C paths, which raise it with ERRCODE_WRONG_OBJECT_TYPE. A caller keying on SQLSTATE, which is what this project’s own privilege suites do deliberately, got different answers depending on which path refused it. The guard now sets wrong_object_type explicitly and audit.sh asserts the code. (#757)

  • A date_trunc predicate within one bucket of the end of the timestamp range raised ERROR: timestamp out of range on a columnar table instead of answering. The rewrite computes the next bucket boundary as lo + step, and that addition throws near the maximum, which fails the whole query rather than declining to prune. Reachable on main before this release through equality alone: date_trunc('year', ts) = timestamp '294276-01-01' errored while the heap returned the row. The rewrite now declines within one step of the end. Infinities are unaffected, because interval arithmetic on them saturates rather than overflowing. (#403)

  • The nightly coverage report now measures something. It had never captured any coverage: the job failed every night from the night it was added on 2026-07-31, always at lcov capture produced nothing. test/run_coverage.sh runs under sudo in CI, so the instrumented build is owned by root, while the harness runs the server as postgres whenever it is root. gcov writes each .gcda beside its object, as the process that ran the code, so the backend could not create one and there was nothing to capture. The counter directories are now redirected with GCOV_PREFIX to a directory under /tmp, which is writable and traversable whoever the server runs as, and returned beside their objects before the report is built. Making the object directories writable was tried first and is not sufficient: creating a file also needs execute on every ancestor, and in CI the tree sits under the runner’s home. Measured in a configuration with an ancestor the server user cannot traverse, which defeats the ownership fix: 0 counters before and 33 after, through lcov and genhtml to a report. The runner also refuses a run that captured no counters, and says where they should have been, instead of leaving lcov to report an empty tree as a tool failure. The per-suite logs are uploaded, so a suite that fails inside this job no longer has its detail discarded when the runner is torn down. (#740)

  • Five differential assertions named an ORDER BY they could not fail on. The oracle in test/lib.sh hashes string_agg(_row::text, chr(10) ORDER BY t), which sorts the rendered rows before comparing them: two results holding the same rows in opposite orders hash identically. That is the right comparison for the ~150 call sites that do not name an order, and no comparison at all for the five that did: two in differential.sh, one in native_format.sh, and two in sorted_projection.sh, whose whole subject is sorted output. Measured on the oracle expression itself: five rows forward and the same five reversed both hash to 2603e60e802d02d5370794d279cb522a, while a genuinely different row set hashes differently, so it was order-blind rather than broken. A second oracle, pgc_seq_hash, hashes ORDER BY row_number() OVER () and so keeps the query’s own output order; diff_query_ordered is its comparison helper, and the five sites now use it. It keeps the EMPTY and unique QUERY_ERROR.$seq sentinels, so empty-versus-empty and error-versus-error still cannot pass vacuously (#418). A new test/selftest part enforces the split in both directions: a diff_query whose query names an ORDER BY now fails the harness self-test, as does a diff_query_ordered whose query names none.

  • The extension-upgrade guard now runs in CI, and had never verified an upgrade before. test/extension_upgrade.sh catches a break that is invisible until a user upgrades: a renamed C link name leaves every existing install pointing at a symbol the new library no longer exports. No workflow set PGC_RUN_UPGRADE, so it ran nowhere, and the one runner that did reach it, the coverage report, discovered it through not_a_suite() and invoked it with no old source, then counted the resulting environment shortfall as a failed suite. A nightly job now builds the previous release on PostgreSQL 18, loads data into it and upgrades it in place, and both upgrade suites are excluded from the coverage runner together. Running it also exposed a defect in the suite itself: every connection used psql -h /tmp, while the socket directory is decided by how PostgreSQL was built, /tmp for a source build and /var/run/postgresql for the Debian and PGDG packages, so the suite had never been runnable against a packaged server and reported the connection failure as “old install did not store rows”. It now states the directory it connects to. (#741)

  • pgcolumnar.set_options accepted a relation that is not columnar, and silently recorded options for it. The row was not merely useless, because options are read by the columnar writer and a heap table has none. It also leaked: the object access hook that clears pgcolumnar.options fires only for columnar relations, so the row outlived the table. Measured on one cluster: set_options on a USING heap table stored a row, DROP TABLE left it behind, and regclass then rendered as the bare oid, which a later relation reusing that oid would inherit; the identical sequence on a columnar table cleaned up. set_options now raises relation "..." is not a columnar table, matching the wording the C paths already use, with a hint pointing at ALTER TABLE ... SET ACCESS METHOD. The test is on the access method and on relkind, which is what makes it match the cleanup rather than merely look strict: the drop hook returns before it examines the access method for anything that is not an ordinary table, so 'r' is exactly the set whose options row can ever be cleared. That matters from PG17, where a partitioned table may itself carry an access method: an access-method test alone accepted a partitioned parent, which has no storage, which the writer never writes, and whose row the hook would never clear. Measured on 17.6: accepted, one row recorded, still present after DROP TABLE keyed to the dropped oid, while an ordinary columnar table in the same run cleaned up. PG16 and earlier refuse PARTITION BY ... USING pgcolumnar outright (checked on 16.14). Partitions themselves are ordinary tables and are still accepted, which is where the options belong. A materialized view is refused for the same reason: CREATE MATERIALIZED VIEW ... USING pgcolumnar works and its rows read back, but options recorded for one are inert, measured against a live fixture that moves the same measurement from 3 to 21 on an ordinary table, and its row leaks on DROP exactly as the others do. That conversion keeps the relation’s oid (measured), so the one workflow that might have wanted the old order, options first and convert second, is served by setting them after the conversion. Nothing in the test corpus called set_options on a non-columnar relation. The guard is mirrored into the 1.0-alpha → 1.0-alpha2 upgrade script, so an upgraded catalog still matches a fresh install. (#432 follow-up)

  • The vectorized aggregates now bound their per-execution memory with one mechanism instead of two. The scan-key context added for #717 covered the scan keys; the scratch context added for #727 covered the whole scan and so covered the keys as well, on the ungrouped node. The grouped node now has the same scratch wrap, the two Begin-time eligibility checks build in a temporary context they delete, and the scan-key context is gone. No behaviour changes and no measurement moves; what changes is that one thing owns the lifetime, and the regression check for it can name what enforces it again. (#717, #727)

  • A columnar scan no longer grows query memory on every rescan. A rescan reuses the read state rather than closing and re-opening it, and each restart rebuilt the row-group list and its per-group metadata into the read state’s own context without reclaiming the previous one, so a LATERAL or parameterized scan accumulated one list per outer row. The list now has a context the rescan resets. Measured against the same query over a heap table with identical data, so the figure is what pgcolumnar adds rather than what re-executing any node costs: 198 bytes per rescan against a 33 byte heap floor before, 33 against 33 after. (#734)

  • A plain EXPLAIN of the ungrouped vectorized aggregate now reports the filters it pushes down. The counts were assigned after the node’s EXPLAIN-only return, so a plain plan printed zero for both Columnar Pushed-Down Filters and Columnar Vector Predicates however many there really were, while EXPLAIN ANALYZE of the same query reported them correctly. A hard zero reads as pushdown not happening rather than as a number that was never filled in, and it left the two vectorized nodes disagreeing about one label, since the grouped node has always counted before its own return. Both counts are catalog and plan work only. (#726)

  • The FSST decoder now checks for interrupts inside its decode loop. Every other value decoder already did. The loop is bounded by the encoded length of one vector, so this is a latency bound rather than the uncancellable hang the Thrift and Avro skip loops had, but a large vector could still hold a backend past a cancel or a statement_timeout. The check uses an iteration counter rather than a byte offset, because the stride is a mask and a code that advances the pointer by two can step over the exact multiple and never fire. decode_interrupts now covers this decoder, and asserts the check sits in the decode loop rather than in the bounded symbol-table parse above it. (#712)

  • The Parquet level-width helper no longer performs a signed left shift that is undefined for a maximum level at or above 230, and no longer has two copies. It computed 1 << b with b reaching 31; real schemas cannot get there, because levels accumulate one per nesting level from a bounded recursion, but it was undefined behaviour in a decoder that reads files it did not write. Making the shift unsigned is not the whole fix on its own: it makes the comparison unsigned too, so a negative maximum converts to a huge value and the loop runs to a 32-place shift that is undefined in turn, and does not terminate under a recovering sanitizer. The helper now returns early for a non-positive maximum and bounds the loop. It also moved to columnar_parquet_format.h, which the writer and the reader already share, because a writer and a reader that disagree on a level width produce a file that is silently wrong rather than one that fails to parse. Values are unchanged on every reachable input. A new suite, parquet_level_width, pins the parts no query can reach. (#710)

  • The Iceberg name-mapping parse reclaims its per-entry scratch instead of holding it for the whole document. Walking one entry allocates a JsonbValue per container lookup, and none of it is needed once that entry’s names are copied out; left in the caller’s context it accumulated across the mapping. Measured as memory attributable to pgcolumnar above the cost of core parsing the identical string: 100 bytes per entry before, 3 after, which is 78 MB down to 4 MB over an 800000-entry mapping. The larger part of such a document’s cost is core’s own jsonb parsing and is unchanged; an unprivileged session can reach more of it with a plain cast, so this is a bounded improvement to one loop and not a limit on what a mapping can cost. A new suite, iceberg_name_mapping_memory, measures the difference against core’s own parse rather than a total. (#731)

  • The Iceberg name-mapping parse sizes its first allocation by the name cap rather than by the element count the file declares. The cap (ICE_MAX_NAME_MAPPING, 100000) was applied inside the loop, after an allocation taken from the document, so a mapping declaring N entries asked for 12N bytes up front however few names it actually held. The surplus is never written, so it costs address space rather than resident memory, and the effect on peak RSS measured 28 kB out of 1196 MB; this is defense-in-depth rather than a memory saving, independent of the 64 MB metadata bound several layers away that was the only other thing limiting it. iceberg_name_mapping gains arms at and one over the cap, built as a single entry carrying many names, which is the shape where the opening capacity and the real ceiling are furthest apart. (#711)

  • The object-store write path (export sink, delete, list) now refuses a URL with userinfo (user@host) with the same error the read path always gave. Before, the write parse admitted the URL and failed closed downstream by accident: the allow-list cannot match a host with an embedded @, and a user:pass@ URL split at the wrong colon into a rejected port. A new suite, objstore_userinfo, pins all three entry points on the SQLSTATE and the guard’s own message. (#706)

  • The debug metadata mutators (pgcolumnar_debug_advance_reserved_offset, pgcolumnar_debug_set_metapage_version) are now owner-only, checked before the table is opened like the other maintenance verbs. They ship unbound, but a binding is one CREATE FUNCTION away, and one of them can overwrite a table’s metapage version and brick every later read. A new suite, debug_hook_privilege, binds them the way the test suites do and proves the gate as a non-owner role, by SQLSTATE and the owner message. (#707)

  • A merge join over index-fetched columnar rows no longer aborts with pfree is not supported by the bump memory allocator on PostgreSQL 17 and later. A columnar index scan returns a deferred slot, and when a sort materialises it the current context is PostgreSQL 17’s bump context, which forbids pfree. The deferred decode pfreed there twice: the needed-column Bitmapset (bms_add_member/bms_free) and, on a storage’s first fetch, the format-version check’s catalog scan (which frees a btree search stack). Both now run in a private pfree-supporting context; the decoded values are unaffected because they were already copied into the caller’s context, where palloc is legal. Covered by a new suite, native_fetch_sort_context (#720).

Added

  • IN (...) and = ANY(array) predicates now drive chunk-group skipping. The scan-key builder derives a conservative [min, max] range over the array’s non-NULL elements. A single-valued list becomes an equality key, which keeps the bloom-filter probe. A parameterized array on a generic plan is frozen at executor start, like scalar parameters. That includes a mixed list such as IN (1, $1). A correlated array is not frozen and stays correct. A new suite, native_saop_pushdown, proves the pruning, the conservativeness, and both freeze rules. (#704)

  • The cross-engine benchmark arms are now reproducible from the repository. bench/build_timescaledb.sh pins TimescaleDB 2.29.0 with the exact cmake options recovered from the original benchmark build, and bench/build_citus.sh pins Citus v14.1.0, each built against one explicit pg_config and refusing an assert build. bench/provision.sh check continues to report their presence; installing them is now a scripted, pinned step instead of a manual one (#702).

Changed

  • The vectorized aggregate no longer grows query memory on every rescan. The node re-executes for each outer row of a LATERAL or parameterized sub-scan, and what it allocated in the caller’s context each time, the per-row value and null arrays, the projected set on the metadata path, and what the flushes and the reader left behind, lived until the query ended. It now runs in a scratch context released when the scan returns. Measured against the same query over a heap table with identical data, so the figure is what pgcolumnar adds and not what re-executing any node costs: 376 bytes per rescan against a 33 byte heap floor before, 31 against 27 after. The ordinary columnar scan beneath it has a separate per-rescan cost that this does not change. (#727)

  • The vectorized aggregates no longer stack a fresh set of pushdown scan keys in query memory on every rescan. The scalar scan builds its keys once at Begin; these paths rebuild them at execution, and did so into a context that lives until the query ends, so a LATERAL or parameterized aggregate sub-scan grew with the rescan count. An IN (...) key made it material, because building one also detoasts the array constant and deconstructs it. Measured on a LATERAL aggregate with a 20000-element list: adding the list cost 179,222 bytes of query memory per rescan before, and nothing measurable after, against a no-list control that is unchanged. The keys now live in a per-node context that is reset at each build. A new suite, vector_agg_rescan_memory, measures the two arms and asserts the difference. (#717)

  • The grouped vectorized aggregate (pgcolumnar.enable_group_vectorization) now folds column at a time instead of row at a time. Where every group key is a plain column, every aggregate is one the fold accumulates (count, sum and avg over integer and float columns), and the whole WHERE is expressed exactly by scan keys, the node walks row groups and reads each column’s packed values directly. The row path paid, for every row scanned, PgColumnarReadNextRow, an expression context reset, staging the row into a virtual slot, ExecQual, and one expression evaluation per group key; the fold pays none of those, and the group hash and the probe equality are inline for integer, oid, date, time and timestamp keys. Float keys keep the type’s own hash and equality functions, because -0.0 and 0.0 are one group with two bit patterns. Measured on 4,000,000 rows with 8 groups, 5 repetitions, counting instructions retired by the backend: 15,850,896,522 before and 7,352,675,551 after without a filter (2.16x), and 13,899,374,785 before and 6,292,504,808 after with one (2.21x). EXPLAIN reports Columnar Batch Fold on the grouped node, as it already did on the ungrouped one, so a plan states whether the fold ran. Shapes the fold declines, among them a by-reference key such as text, an expression key, min and max, and a filter no scan key expresses exactly, run the row path and return the same results. A new suite, native_groupagg_batch, proves the results against a heap mirror and proves the work separately from them. (#708)

  • An index or bitmap fetch no longer reads the whole row-group list out of the catalog for every row. The list is memoized per command and snapshot, with a refresh on any miss, so a group flushed earlier in the same statement stays visible. Measured on 3000 index-fetched rows: 6001 catalog index scans before, 2 after. The memo resets on subtransaction abort and on row-group retirement, and a new suite, native_fetch_group_memo, proves both by resurrection: through an open cursor after ROLLBACK TO SAVEPOINT, and through stale index entries after a same-statement compaction. (#709)

Fixed

  • The ungrouped vectorized aggregate (pgcolumnar.enable_ungrouped_vector_agg) no longer returns wrong results for a filter whose operator has no btree strategy, such as <>. The batch fold’s only per-row filter is the scan-key loop, but eligibility only checked that each clause was a strict comparison, not that it became a scan key. A <> clause is strict yet builds no key, so on an all-by-value shape the fold engaged and counted the excluded rows (count(*) WHERE x <> 5 over-counted by one). Eligibility now requires every clause to be expressed exactly by scan keys (PgColumnarQualsExactlyKeyed); conservative prune-only keys (anchored LIKE, and the IN-list ranges added above for #704) are marked inexact and also disqualify the fold, which then falls back to the correct scalar path. Covered by new cases in ungrouped_vector_agg (#715).

  • A corrupt column_chunk.page_offset no longer crashes the backend on a whole-group (unprojected) read. The projected read already refused a chunk lying outside its row group; the whole-group decode path derived base = nativeBuffer + (page_offset - file_offset) without the same check, so a poisoned offset (or one below the group start, wrapping the subtraction) read out of bounds and took SIGSEGV. The containment check now sits on the shared decode path, so both reads raise ERRCODE_DATA_CORRUPTED and survive. The containment test on both the shared and projected paths is written free of unsigned overflow, so a page_offset of -1 (storable in the signed bigint column, read back as ~UINT64_MAX) is rejected here rather than wrapping past the page_offset + page_length sum into an out-of-bounds read. New suite native_page_offset_bound reproduces the crash and the fix, including the -1 wrap on both paths; it closes the page_offset gap in corruption.sh, which only ever corrupted page_length through a projected scan.

[1.0-alpha2] - 2026-08-18

Added

  • The extension packages a 1.0-alpha to 1.0-alpha2 upgrade script, generated from the catalog delta between the two versions (16 new functions, 3 changed, two foreign-data wrappers, two pgcolumnar.storage columns, and the PUBLIC execute revokes on the internal projection and visibility-map functions). A convergence test, native_upgrade_converge, installs 1.0-alpha, runs ALTER EXTENSION pgcolumnar UPDATE, and asserts the result is byte-identical to a fresh 1.0-alpha2 across every function definition and ACL, relation, column, type, and foreign-data wrapper. The full 1.0-dev to 1.0-alpha2 path, including that an existing columnar table still reads unchanged across the C-symbol rename, is covered by a two-library test, upgrade_from_dev_twolib. This test caught three defects in the generated script before it converged (psql command tags captured into the SQL, function definitions concatenated without a terminating semicolon, and the ACL revokes omitted because a definition-only diff cannot see an ACL-only change).
  • The Iceberg REST catalog FOREIGN SERVER now accepts a warehouse option alongside catalog_uri. It selects a warehouse on a multi-warehouse catalog and is sent as the ?warehouse= query parameter on the GET /v1/config request. This completes the per-catalog credential model (issue #656). Regression test: iceberg_rest_server.

Changed

  • The native encoding-descriptor wire layout is single-sourced. The descriptor is a column chunk’s writer-to-reader contract for its per-vector encoding; its byte layout was hand-packed in columnar_write_state.c and hand-parsed in three passes in columnar_reader.c, so a field change had to be made in five disjoint places or a reader stride would land mid-field. Because the version byte does not move on a field change, the version guard would pass and the mismatch would surface as DATA_CORRUPTED or wrong values, not a clean rejection. A new columnar_encdesc.h owns the layout (PgColumnarEncdescPut* / ReadEntry), and a StaticAssert ties the entry length to the field widths. The on-disk bytes are unchanged (verified byte-identical to the hand-packed form). Regression test: native_encdesc_golden (pins the layout; catches a field-order change that the self-consistent round-trip suites cannot).
  • The delete-vector visibility logic is single-sourced. The fold that ORs a row group’s delete bitmaps into one mask was duplicated in the sequential scan and the index-fetch liveness cache, and the per-row bit test was open-coded in five places across the sequential scan, the index-only scan, the per-row fetch, and the buffered read-your-writes path. A change to one (a new bitmap encoding, a bound) could silently diverge the others, invisible until one access path hit the changed row. The fold is now pgcolumnar_merge_delete_vectors and the bit test dv_row_deleted; every path routes through them. Behavior is unchanged. Regression test: native_delete_visibility_paths (asserts all paths agree on the deleted set; a mutation of the shared bit test fails every path).

Fixed

  • Group and per-vector skipping now read only the predicate columns' zone maps. pgcolumnar_native_group_can_match and the per-vector skip builder read the whole group’s zone maps (every attribute’s min/max) up front and used only the columns carrying predicates, so on a wide table a scan that consults one or two columns paid min/max buffer traffic proportional to the table width. Both now probe per column via zone_map_pkey’s column_index, exactly as the per-column bloom fetch already did; the per-vector spans (identical across columns) come from a predicate column, falling back to the whole-group read only when no predicate column has per-vector rows. On a 30-column table a one-predicate scan went from about 1200 zone-map row fetches to 30, matching the 2-column table. Regression test: native_zonemap_narrow.
  • Chunk-group skipping now applies to parameterized predicates. A qual like col >= $1 from a prepared statement, PL/pgSQL, or the extended protocol kept a Param operand, and the scan-key builder accepted only a Const, so on a generic plan the scan built no key and read every chunk group. Begin now freezes execution-stable operands (a PARAM_EXTERN, or a subexpression with no Var, no PARAM_EXEC, and no volatile function) into Consts before building the keys. A correlated PARAM_EXEC changes per rescan and is deliberately not frozen, so results stay correct; the executor still re-applies the original qual to every surviving row. On a 20-group test a >= $1 generic-plan scan went from reading all 20 groups to reading 1. Regression test: native_param_pushdown.
  • Reads of the delete_vector catalog now use its delete_vector_pkey index instead of a sequential scan. PgColumnarReadDeleteVectorList (called once per row group while a scan builds its liveness cache), the reclaim deleted-count sum, and PgColumnarStorageHasDeleteVector each passed InvalidOid to systable_beginscan, so every call sequentially scanned the whole delete_vector catalog filtered by scan key. On a table with deletes the cost was O(row_groups * delete_vector_rows). The sibling metadata tables already index their reads this way; delete_vector now matches. Regression test: native_delete_vector_index (asserts the reads take the index, seq_scan = 0).
  • The Thrift and Avro field-skip loops are now interruptible. A Parquet footer whose unknown field is a list<bool> with a file-declared count up to 0xFFFFFFFF (or a struct holding many such lists) drove billions of zero-byte skips through PgColumnarThriftSkip, which checked stack depth but not interrupts, so the backend spun uncancellably on a sub-2 KB file reached through parquet_schema() / read_parquet(). av_skip had the same gap on an array<null> manifest block, where its interrupt check ran once per block rather than per element. Both now call CHECK_FOR_INTERRUPTS on the per-value skip path, so statement_timeout and cancel apply. This is a denial-of-service hardening in the same class as the parallel_copy FIFO fix. Regression test: decode_skip_interrupts (functional cancel for Thrift; per-element placement shape for both).
  • pgcolumnar.read_manifest_list now reports a null manifest-list min_sequence_number as SQL NULL instead of 0, matching sequence_number. The value is not used by the delete-application rules, so this is a display fix. Regression test: iceberg_malformed (#686, #691).

  • pgcolumnar.parallel_copy no longer hangs the backend when its path names a FIFO. pcopy_open_regular_file opened the path O_RDONLY and then checked for a regular file, but a FIFO blocks inside that open(2) and the block survives a cancel or statement_timeout, a denial of service. It now opens with O_NONBLOCK, rejects a non-regular file, then clears the flag, which also closes the stat-before-open race. Regression test: parallel_copy (#686).

  • The Iceberg read path now refuses four classes of malformed or hostile table metadata that an adversarial re-audit (#644) found it mishandled. A non-regular file (for example a FIFO) named as a metadata, manifest, or data path is refused before it is opened, instead of blocking the backend in a cancel-resistant open(2) (an availability denial of service). A manifest-list sequence_number that is the Avro union’s null branch is refused rather than decoded as 0, which had understated a data file’s sequence number and could mis-apply an older delete and drop a live row. A position-delete row with a null or negative pos is refused rather than silently dropped, which had left a row the delete was meant to remove. A current-schema-id that names a schema absent from the schemas array is refused rather than silently resolved to the deprecated top-level schema, which had bound columns through a stale schema and misprojected rows. A v1 table with only a top-level schema still reads. The same non-regular-file guard also covers pgcolumnar.import_arrow, which opened its path with the identical unguarded FIFO open(2) denial of service. Regression tests: iceberg_malformed, iceberg_deletes, arrow_import.

  • Concurrent UPDATE or DELETE of the same columnar row now serializes on the row identity, so the losing writer gets a retryable serialization_failure instead of duplicating the row and losing an update (issue #5, the UPDATE facet). Two sessions updating one row without a covering unique index each kept their own new version. The fix takes a transaction-scoped advisory lock on the row, then re-reads the committed delete vector under a fresh snapshot before writing. New GUCs pgcolumnar.enable_row_update_lock (default on) and pgcolumnar.row_lock_buckets (default 1024). Regression test: update_conc.
  • The ungrouped batch fold’s per-row gather now steps over the columns a query references rather than all of a table’s columns. On a wide table a scan that reads a few columns walked every column per row (twice on a deferred group), which is loop overhead proportional to the table width. It now iterates compact key/payload lists; a 46-column single-aggregate fold measured about 23% faster with no change in results. Regression test: native_batch_fold_projection.
  • The grouped vector aggregate’s input-scan estimate is now shared with the columnar scan node’s, via one pgcolumnar_refined_scan_cost helper. Its no-serial-survivor fallback used the bare seqscan formula, which omitted the projected-width I/O, the per-column decode CPU, and the zone-map survival scaling the real scan applies, so it under-priced the node’s input on a wide, low-pruning scan. Regression test: native_groupagg_wide_cost.
  • The Iceberg foreign-data wrapper now pushes projection down: it decodes only the columns a query references (from the output list and the recheck quals), not every column of every surviving file. A narrow projection over a wide table is much cheaper (a 1-of-40-column scan measured about 5x faster). The reader’s existing needTop mask carries it; the wrapper computes the mask and the results are unchanged. Regression test: iceberg_fdw_projection.
  • The index-fetch cost penalty now sizes row groups by a relation’s effective stripe_row_limit (the per-table option when set, else the GUC), matching the writer and the zone-map survival estimate. It read only the GUC, so it mis-priced the row-group decode for a table that set the option, and could steer the planner toward or away from an index scan on that table. The effective-limit lookup is now one shared helper. Regression test: native_index_fetch_stripe_cost.
  • The Iceberg foreign-data wrapper now estimates a scan’s row count from the manifests (the sum of the live data files' record counts) instead of a constant
    1. The constant mis-sized every scan and corrupted join planning above a large Iceberg table. Regression test: iceberg_fdw_estimate.

Security

  • The native varlena decoder now bounds a value’s stored length against its buffer. A varying-length value in a column chunk (and a zone map’s min/max) carries its own length prefix, which PgColumnarDecodeValue, pgcolumnar_skip_value, and pgcolumnar_build_val_offsets read and memcpyd with no check that the value fit; a corrupt chunk or catalog row could then declare a value running past the buffer (an out-of-bounds read), and a prefix carrying the external-TOAST tag would be detoasted through a bogus pointer. The decode sites now pass the value stream’s end and read through a bounded reader that refuses an over-long length or an external tag with DATA_CORRUPTED. This is the trusted-storage boundary (bit rot, a hand-written stripe), not a file-author-reachable one, but a clean error beats a crash. Regression test: native_varlena_bound (a min/max whose header claims ~1 GB now errors instead of crashing the backend; a mutation removing the bound reddens it).
  • Closed a stat-before-open race in the local file read path. The non-regular-file guard that keeps a FIFO from blocking the backend in open(2) (a cancel-resistant denial of service) screened the path with stat() before a separate open(), so a local principal who can write the directory could swap the checked regular file for a FIFO in the window between them and re-introduce the block. The five transient-fd local openers (the Iceberg metadata and manifest reads, the Avro manifest read, import_arrow, and the Parquet source) now go through one helper that opens O_NONBLOCK, fstats the fd it holds, refuses a non-regular file, then clears the flag, so the file checked is the file opened. The Parquet source reads positionally with pg_pread. The parallel-copy partition coordinator (pcopy_partition_aligned_offsets), which needs a stdio FILE* for getline, applies the same O_NONBLOCK open plus fstat inline. Regression tests: local_open_race_free, parallel_copy.
  • Fixed a backend crash on a hostile Iceberg manifest with a null path. The reader decodes a manifest and manifest list against the schema embedded in the file, which the table author controls, so a manifest_path (or data or delete path) can be declared nullable and encoded null. ice_rebase then called strncmp on the null pointer and segfaulted the backend. A null recorded path is now refused as DATA_CORRUPTED, matching the read path’s other malformed-metadata refusals. Regression test: iceberg_malformed (#691).

  • Fixed an HTTP request-line injection in the object-store client. A URL path or host containing CR or LF was written verbatim into the request line, so a crafted path could split the request and smuggle a second line to an allow-listed endpoint. The request path and host are now rejected if they carry CR or LF, as the caller-supplied header lines already were. Regression test: objstore_crlf.

  • Fixed an uninitialized-memory read in the native DICT decode path. A chunk whose descriptor declared a value_raw_length larger than its codes decode to left the tail of the raw buffer uninitialized, and a varlena column then read a length prefix out of that garbage (silent wrong results, or an out-of-bounds read). decode_dict now requires the decoded length to equal the declared raw length, mirroring the FSST path. Regression test: native_dict_underfill.

  • Fixed an out-of-bounds read in the Parquet dictionary decode path. A file whose RLE_DICTIONARY data page carried an index with the high bit set (reachable at bit_width 32) passed a signed bounds check that sign-extended it to a negative int, and the dictionary was then read far out of bounds, crashing the backend from a single crafted file. The bounds check is now unsigned and the index is rejected. The index decode runs only when the page has coded values, so an all-null column with an empty dictionary page still reads. Regression tests: native_parquet_dict_oob and native_parquet_streaming.

Added

  • Read-only Apache Iceberg support, filesystem-backed, at a table’s current snapshot (#388). pgcolumnar.iceberg_scan(metadata_path) reads a table given a column definition list, resolving each output column to a schema field id so a data file written before a column rename still reads. It applies row-level deletes of all three kinds, each under its own sequence rule: a position delete drops the row ordinals it names from a data file whose data sequence number is at or below the delete’s (same commit or earlier), and an equality delete drops every data row matching a delete row on the delete’s equality_ids columns when the data file’s sequence number is strictly below the delete’s (never same-commit data). Format-version 3 deletion vectors (Puffin files holding a portable roaring bitmap of row ordinals) apply under the position-delete rule, scoped to their referenced data file, and supersede position delete files for that file per the specification; the blob checksum, the manifest/footer offsets, and the recorded cardinality are verified, and at most one vector may reference a data file. A null delete value matches only a null data value, and columns beyond equality_ids do not take part in the match. A partition-scoped equality delete is applied within its partition: its stored partition values are matched against each data file’s, so it removes rows only from data files in the same partition. Equality deletes with no supported handling are refused rather than ignored, so a table using them errors instead of returning rows it should have removed: delete columns of types outside int/long/string/boolean/date, delete columns dropped from the current schema, and a partition value the reader cannot compare exactly. Supporting introspection functions: iceberg_current_snapshot and iceberg_data_files (which refuses any delete), and the Avro building blocks read_avro_manifest and read_manifest_list. Only Parquet data files are read; recorded paths are rebased onto the table’s actual location and refused if they resolve outside it. The table may live in object storage: a metadata path of s3://, http://, or https:// reads the metadata, manifests, data files, and delete files from the endpoint through the object-store module, gated by the same objstore_allowed_endpoints allow-list and ambient credentials as the Parquet reader. A data file written outside Iceberg, which carries no field ids, is read through the table’s schema.name-mapping.default property, which binds its columns by name; a file with no field ids and no such property is refused rather than guessed. read_parquet also gained a field_ids form that projects columns by Parquet field id. See Iceberg.

  • Read-only Apache Iceberg REST catalog support (#388). A table is named by a catalog (catalog URI, namespace, table) rather than a metadata path. pgcolumnar.iceberg_rest_scan(catalog_uri, namespace, table_name) reads it at its current snapshot, taking a column definition list exactly like iceberg_scan: the catalog resolves the table to its metadata location, which is then read through the same path, so every projection and delete rule applies unchanged. pgcolumnar.iceberg_rest_table_location returns that resolved metadata location on its own, and pgcolumnar.iceberg_rest_namespaces and pgcolumnar.iceberg_rest_tables list a catalog. Requests go over HTTP or HTTPS. The catalog endpoint is subject to the same objstore_allowed_endpoints allow-list and link-local refusal as every other remote access, and is carried by the object-store module, so no second TLS stack enters the server process. A bearer token, when the catalog requires one, is read from the PGCOLUMNAR_ICEBERG_REST_TOKEN server environment variable, never a function argument, so it does not appear in the statement log. The first argument may instead name a foreign server of the new validator-only pgcolumnar_iceberg_catalog wrapper (#656). The server holds catalog_uri, and the current role’s user mapping holds the bearer token in pg_user_mapping, which is not world-readable, so one role’s token is private from another. A role with neither a mapping token nor superuser rights is refused, and the validator keeps secrets off the world-readable server options. A user mapping may instead carry OAuth2 client credentials (oauth_client_id, oauth_client_secret, and optionally oauth_scope and oauth_token_uri) (#656). The catalog then mints a bearer by the client-credentials grant; the secret travels in the request body, never a URL or a log line, and a half credential is refused before any request. When the catalog vends short-lived storage credentials in its loadTable reply (the flat config keys or the storage-credentials array, longest prefix selected), iceberg_rest_scan reads the table’s files with those credentials rather than the server environment (#656). Vended credentials do not bypass the endpoint allow-list. A table that vends none reads with the ambient environment as before. See Iceberg REST catalog.

  • An Apache Iceberg foreign-data wrapper, pgcolumnar_iceberg (#388). A foreign table over an Iceberg table gets the query’s predicate, which iceberg_scan cannot, and prunes: a predicate on an identity-partitioned column removes whole data files before they are opened, reading each file’s partition value from the manifest. A predicate on an integer or boolean column removes whole files whose stored minimum and maximum exclude it, so an unpartitioned column prunes too. Pruning is only an optimization, so a predicate the wrapper cannot decide never changes the rows returned, and every projection and delete rule matches iceberg_scan. The table option is metadata_path; EXPLAIN (ANALYZE) reports Files Pruned. An equality predicate on a bucket[N]-partitioned column prunes files whose stored bucket differs from the constant’s, computed with the Iceberg murmur3 hash. A predicate on a truncate[W]-partitioned integer column prunes files whose truncated value range excludes it, and a predicate on a day()-partitioned date column prunes by day. The year(), month(), day(), and hour() transforms prune too, on a timestamp or timestamp with time zone column (and year()/month() on a date): each bucket spans a range, so a file whose bucket equals the constant’s is read and its rows are rechecked, never dropped at the boundary, and a timestamptz value is compared as its UTC instant. Partition pruning covers identity, bucket[N], truncate[W] (integer), and the temporal transforms on date, timestamp, and timestamptz; metrics pruning covers integer and boolean columns; other column types read in full. See Iceberg FDW.

  • The Parquet read and export functions and the foreign-data wrapper read from and write to object storage (#393, #394). A path may be an s3://, http://, or https:// URL wherever it may be a local path. s3:// requests are signed with AWS Signature Version 4; https:// verifies the server certificate and is available when the object-store module is built with OpenSSL. Support lives in a separate module, pgcolumnar_objstore, loaded on the first remote use. Reads take exact object keys only. export_parquet and export_arrow write to s3://, as one request for a small object or a multipart upload for a large one, and the object becomes visible only when the upload completes. See Object storage.

  • Credentials for object storage come from the server process environment (AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION or AWS_DEFAULT_REGION) for the function API, and from the catalog for the foreign-data wrapper: endpoint and region on the server, and access_key_id, secret_access_key, session_token, and credentials_required only on a user mapping, so a secret is never in a world-readable option. Ambient environment credentials are used only for a superuser or a mapping a superuser marked credentials_required 'false'.

  • pgcolumnar.objstore_allowed_endpoints lists the endpoints the object-store module may connect to (#393). It is empty by default, which refuses every remote endpoint, so a role that can read or write server files cannot reach an arbitrary host through the extension. Link-local addresses, including the cloud instance-metadata address, are refused whether or not they are listed. The setting is superuser-only.

  • pgcolumnar.parquet_schema reports a field_id column (#388), the Parquet schema field id each leaf column carries, which formats such as Apache Iceberg use to select columns by id. It is NULL when the writer emitted none.

  • pgcolumnar.maintenance_due(rel, compact_due_fraction, recluster_due_fraction) reports whether an online maintenance verb is worth running on a table, from its statistics alone (#415). It takes no lock and rewrites nothing. It returns the deleted and appended fractions, whether compact_rewrite or recluster has crossed its threshold, and a recommendation. This is the policy the pgcolumnar.autovacuum daemon consults, and a monitoring role can call it directly. It is SECURITY DEFINER and checks that the caller may SELECT the table, so the owner can run it without superuser rights.

  • pgcolumnar.autovacuum, a maintenance daemon for the online upkeep that core autovacuum cannot reach (#415). pgColumnar’s compact_rewrite and recluster live in extension functions, not table access method callbacks, so core autovacuum never runs them. A table’s dead rows and clustering decay then accumulate until someone runs the verbs by hand. This daemon runs them for you.

    It is off by default. When on, a launcher wakes every pgcolumnar.autovacuum_naptime seconds (default 60) and starts one worker per database. Each worker asks pgcolumnar.maintenance_due() which tables crossed a threshold, then runs the recommended verb over SPI in its own transaction.

    Two properties make it safe unattended. It calls only the ShareUpdateExclusiveLock verbs, never vacuum, vacuum_sorted, or cluster, so it cannot block a reader or a writer. And it yields the way autovacuum does: the worker sets PROC_IS_AUTOVACUUM, so the lock manager cancels its maintenance the moment a backend queues for a stronger lock. New settings: pgcolumnar.autovacuum, autovacuum_naptime, autovacuum_compact_threshold (0.2), and autovacuum_recluster_threshold (0.05). See the administration guide for the operator’s view.

  • pgcolumnar.parallel_flush dispatches a stripe flush across background workers (#445). Default off. When on, a flush of two or more columns fans the per-column encode and compress work out to a worker pool. Any column a worker does not finish is completed serially in the backend, so the stored bytes match the serial path either way. It helps one large flush of many numeric columns by up to 14 percent. A wide text-heavy flush regresses, and so do frequent small flushes, because it copies the buffered bytes through shared memory. So it is a per-session opt-in for a wide numeric bulk load, not a default.

  • pgcolumnar.fsst_verdict_reuse caches a column’s FSST keep/drop verdict for a bounded number of row groups (#472). Default 16; 0 asks every time, which is the behaviour before this setting existed.

    Deciding whether an FSST symbol table pays for itself costs a whole-corpus encode plus a compression pass, and the answer cannot be sampled: on a training prefix FSST can look 24 percent worse while over the whole column it is 23 percent better. So it was asked once per column per row group, and for a column whose data does not change character that re-derived the same answer for the whole load. Measured at 2,000,000 rows in 20 row groups: 2482 ms of a 5319 ms md5 load and 843 ms of a 2081 ms email-shaped load, with the verdict identical all 20 times.

    A text load is about 2.5 times faster as a result, measured in-suite at 1623 ms against 648 ms. Stored bytes are unchanged for a column whose verdict is stable, which is asserted rather than assumed: the suite compares the encoding descriptor, block codec and page length of every chunk. A column that changes character mid-load is noticed within the bound.

    Reuse is per statement. Nothing is persisted and no on-disk structure changes.

  • EXPLAIN (ANALYZE) now reports Columnar Usable Skip Predicates beside Columnar Pushed-Down Filters (#479). The existing line counts the filters the scan was handed and is unchanged; the new one counts how many of those the reader can actually skip chunk groups with. A filter whose types have no ordering function for the pair is dropped by the reader and excludes nothing, and until now the plan reported it as pushed down with no way to see the difference. That is how #477 went unseen for a year, and how test/zonemap_cost.sh validated a cost discount against a fixture that pruned zero groups.

    All three nodes that print the original line report the new one: the scalar custom scan and both vectorized aggregate nodes. The new line needs ANALYZE, since it describes what the scan built at execution.

  • pgcolumnar.analyze() now collects most_common_vals and most_common_freqs, and excludes those values from histogram_bounds (#414). Frequencies are exact counts over the total row count rather than sample estimates. PostgreSQL 18 and later, which is where pg_restore_attribute_stats exists; earlier majors raise and should use ANALYZE.

    The selection rule is PostgreSQL’s own. analyze_mcv_list() keeps the entire list when the whole table was read instead of applying its significance filter, because that filter exists to judge sample frequencies. Reading the column makes the values eligible on count alone, matching what core would store given the same information.

    Excluding most-common values from the histogram is required rather than cosmetic: keeping them counts those values twice in selectivity, once from the most-common list and again inside the bucket that holds them.

  • test/analyze_differential.sh, which compares the statistics pgcolumnar.analyze() writes against the shape PostgreSQL’s own ANALYZE produces across five column types. pg_restore_attribute_stats takes VARIADIC "any" and responds to a mistyped argument with a warning rather than an error, so a statistic can be dropped while the call reports success. Values cannot be the comparison, since exact and sampled statistics differ by design, so the suite compares the operator, collation and presence of each statistic kind, and verifies every stored value against an independent count.

Fixed

  • The Iceberg reader no longer crashes on a malformed manifest (#644). A crafted manifest that recorded no data-file path made iceberg_scan and iceberg_data_files dereference a null pointer. A manifest whose Avro record schema gave a fields element as a JSON array made the schema decoder read past an object container. Both now raise a clean error. These reproduce only from hand-crafted manifests, since no writer emits them, and they are covered by the new iceberg_malformed suite.

  • A failed export_parquet or export_arrow no longer leaves a partial file at the destination (#394). An export writes to a temporary name and renames to the final name only when it is complete, so a reader never sees a half-written file. Every write is checked, so a full disk during an export is reported rather than detected only at close.

  • EXPLAIN (ANALYZE) on a vectorized aggregate reports whether the batch fold actually ran, not whether it was predicted eligible (#602). A query that fell back to the row path, such as an aggregate over a column added after some row groups, no longer reads Columnar Batch Fold: yes. Plain EXPLAIN, which has no execution to report, still shows the prediction.

  • pgcolumnar.sort_status works for a non-superuser who owns the table (#608). The function reads pgColumnar’s internal catalogs, which carry no GRANT. As an invoker-rights function it therefore failed for any caller who was not a superuser. It is now SECURITY DEFINER and checks that the caller may SELECT the table. The owner can read the sort status of their own table, and no caller gains access to a table they could not already read.

  • pgcolumnar.analyze() counts null_frac over live rows (#485). It came from the zone maps, which record what was written, so a deleted row kept counting toward the denominator until the table was rewritten. VACUUM did not correct it. On 1,000 rows holding 100 nulls, deleting the 301 rows of one value left null_frac at 0.100000 against a true 0.143062.

    The size of the error is not the whole of it. null_frac came from the zone maps while the most-common frequencies came from a live count, so one pg_stats row carried two statistics normalised against different populations: a null_frac implying 1,200 rows beside a frequency implying 900, with 900 actually present. null_frac + sum(most_common_freqs) + rest = 1 stopped holding, and eqsel subtracts both when pricing everything else.

    The null count now comes from the read the function already performs, so this costs no extra pass. It does give up the “null_frac is a metadata read” property claimed for #414 slice 1, which cost nothing in practice because the function always goes on to read the column for n_distinct. A metadata-only fast path would need a live-row count, which is that same read. Whether pgcolumnar.zone_map’s counts should account for the delete vector, which would also affect pruning, is a wider question and is not addressed.

  • A column declared over a domain now prunes chunk groups (#483). The scan key was built and then dropped: a domain column carries the domain’s type in pg_attribute while the constant beside it carries the base type, so the comparison looked cross-type, and an operator family has no comparison function registered for a domain. Measured on identical values in one table over 20 row groups, int and bigint each removed 19 groups and a domain over either removed none, while all three reported the filter as pushed down.

    Answers were never wrong, because the executor re-applies the qual. The cost was reading the whole table on ordinary SQL. Both sides of the comparison are now resolved to their base types, so a domain compared against a value of a different domain over the same base type is also recognised. Ordering and hashing are unchanged: the comparison and hash functions were already taken from the column type’s resolved entry, which is what the writer used to build the zone maps and bloom filters.

  • A bigint column compared against an unadorned integer literal now prunes chunk groups (#477). The scan key was dropped because the column type’s default comparison function cannot take an int4 argument, so predicates of the form bigint_column > 16000 read every chunk group while bigint_column > 16000::bigint pruned normally. The comparison function is now resolved for both types from the column’s btree operator family, which supplies exactly this for the built-in numeric types. Where a family provides no such function the key is still skipped, as before.

    EXPLAIN did not show the difference. Columnar Pushed-Down Filters counts scan keys given to the reader rather than predicates able to exclude a group, so it reported the filter as pushed down while nothing was skipped.

    The bloom filter probe remains disabled for cross-type equality. The filter stores hashes of column-type values, so hashing a differently typed constant would probe a slot that was never written and could skip a group holding matching rows.

  • pgcolumnar.analyze() now honours the per-column statistics target set by ALTER TABLE ... ALTER COLUMN ... SET STATISTICS (#414). It read the global default_statistics_target for every column, so a column given its own target was sized by the global setting instead. A target of zero means the column is not to be analysed at all, and is now respected rather than overridden.

    Requesting only zero-target columns no longer raises. The function reported that it had collected statistics for no columns, with a hint about missing row groups, which pointed at storage for what was a deliberate setting.

  • Renamed the custom scan node from ColumnarScan to PgColumnarScan, and the custom path from ColumnarAgg to PgColumnarAgg (#428). ColumnarScan is also registered by Citus columnar and by TimescaleDB 2.29. PostgreSQL’s registry is one hash table keyed on that name, so two extensions cannot both hold it. Neither failure needed a pgColumnar table; our presence in shared_preload_libraries was enough.

    With Citus columnar the server refused to start at all, in either load order:

    FATAL:  extensible node type "ColumnarScan" already exists

    With TimescaleDB there was no startup error and serial queries returned correct results. TimescaleDB checks the registry first and silently skips registering when the name is taken, so a parallel worker then resolved TimescaleDB’s node through pgColumnar’s callbacks, and any parallel query over a columnstore hypertable failed with could not read blocks 0..0 in file .... That is the more dangerous of the two, because nothing announces it.

    This changes EXPLAIN output. Plans that read Custom Scan (ColumnarScan) now read Custom Scan (PgColumnarScan). Anything parsing plan text for the old name must be updated. The Columnar ... property lines, such as Columnar Projected Columns, are a different namespace and are unchanged. ColumnarAgg never appeared in EXPLAIN: it names a CustomPathMethods, and the planned node carries the scan’s methods (columnar_vector.c:717), so that half of the rename is hygiene rather than a visible change.

Changed

  • pgcolumnar.recluster no longer rewrites a table that is already clustered by the requested key (#415). The function records the clustering key and kind it establishes. A later call with the same key returns 0 and touches nothing when the existing sorted run still covers every row group. Before this, it re-sorted on every call, so a scheduled recluster rewrote the whole table each time, which is why the maintenance daemon could not have run it safely. pgcolumnar.sort_status now reports this recorded key as sort_key, and falls back to the declared sort_by when there is no recorded key.

  • A columnar scan whose filter cannot be pushed down now skips decoding the projected columns of a 1024-row vector that holds no matching row (#452). The scan decodes the filter columns first, rules out the vectors with no match, and decodes the rest only for the vectors that survive. A SELECT * under a leading-wildcard LIKE that matches few rows then approaches the cost of count(*). It no longer decodes every column of every row scanned. A count over one column gains nothing, because it has no projected column to skip.

  • The writer detoasts each value once per row (#445). It was detoasted once for the encoder, once for the bloom filter, and once for each of the two zone-map comparisons. For a toasted column each of those was a separate decompression. A load of a large compressed text column is about 11 percent faster, and the stored bytes are unchanged.

  • pgcolumnar.analyze() places histogram_bounds at PostgreSQL’s own positions (#414). The bounds were evenly spaced quantiles; core places bound i at values[floor(i * (nvals - 1) / (num_hist - 1))] among the rows left after the most-common values are removed, and percentile_disc resolves a fraction to a different index whenever the two disagree.

    This changes the emitted array. The length and both endpoints are the same, so the exactness of the minimum and maximum is unaffected, but an interior bound can move by one position. Both forms are valid equi-depth histograms; core’s is the one the planner’s selectivity estimators were tuned against. Anyone comparing pg_stats across this upgrade should expect interior bounds to differ and that is intended, not a regression.

  • The unsupported-rewrite error names REPACK on PostgreSQL 19 (#399). REPACK replaces CLUSTER and VACUUM FULL in 19 and dispatches through the same copy-for-cluster path, which pgColumnar does not implement, so a 19 user who typed REPACK was told that CLUSTER / VACUUM FULL was unsupported: two commands they had not typed, and on 19 the superseded ones. The message now names the command and hints at pgcolumnar.vacuum(), which does the work. This covers REPACK (CONCURRENTLY) too: given a table with an identity index, where PostgreSQL will run it, heap succeeds and a columnar table is refused.

  • CREATE TABLE ... USING pgcolumnar AS SELECT no longer fails when the source plan is parallel (#387). The storage-row creation path re-checked for an existing row against GetLatestSnapshot(), which raises “cannot update SecondarySnapshot during a parallel operation” inside parallel mode, and CTAS runs its whole executor in parallel mode whenever the source plan is parallel. That is the default for any source large enough to be worth loading, so bulk creating a columnar table from existing data failed on every supported major. The lock and the fresh snapshot are now skipped when the relation was created by the current transaction, because no other session can see it and the first-writer race they defend against cannot happen. A committed table first-written by two sessions at once is unaffected and still serializes.

  • The extension’s exported C symbols are namespaced under pgcolumnar (#382). Two extensions that both call themselves columnar could define the same symbol. columnar_handler and columnar_relation_storageid collided with Citus columnar. Four settings variables such as columnar_stripe_row_limit also shared names with the same settings there. That case binds one library’s setting to the other’s storage.

  • default_version moves from 1.0-dev to 1.0-alpha, so SELECT extversion FROM pg_extension now agrees with VERSION.

  • CREATE INDEX decodes only the columns the index needs (#413). The index build received an IndexInfo carrying the key columns and the expression and predicate trees, and discarded it, so a one-column index on a wide table read every column. Both readers are now projected: the one a serial build opens for itself, and the shared scan a parallel build arrives with, which comes through the table-access-method scan interface and has nowhere to carry a projection. The parallel branch is not a corner case. With every parallel setting left at its default, a 1.5 million row table of incompressible text, 459 MB on disk, is built in parallel, so that is the branch a table of consequential size takes. On 300,000 rows of 20 columns on PostgreSQL 18, a one-column index drops from 568 ms to 73 ms with workers allowed, against 563 ms for the same index on a heap table.

  • Logical replication into a columnar table says what is wrong and how to proceed (#435). Applying an UPDATE or a DELETE takes a row lock, which columnar storage does not support, so the apply worker raised “columnar: row locking is not supported yet”. That names something the user was not doing. PostgreSQL takes that lock for every applied UPDATE and DELETE and has no lock-free path, and it does not advance the replication origin when a transaction fails, so the subscription retries the same transaction for as long as it runs and no later change is applied. The error now names logical replication, says the retry is unbounded, and points at CREATE PUBLICATION ... WITH (publish = 'insert'), which does work. See Limitations.

  • The grouped vectorized aggregate’s parallel arm is no longer declined on a truncating time key (#369). estimate_num_groups cannot see through a function, so for date_trunc('minute', ts) it falls back to the timestamp column’s distinct count, which measured 19,996,000 against 720 actual. That number is charged twice on the parallel arm, once by the Gather for tuples it believes it must ship and again by the Finalize, and not at all on the serial node, which is priced per input row. The serial node therefore won by construction on the shapes where the parallel arm is fastest. The estimate is now bounded by the number of buckets the scanned time range can span, and only when the planner had nothing to estimate from. A plain column, an expression index and a user’s CREATE STATISTICS ON (expr) all count as informed and are left alone. Measured on 20 million rows: a one-aggregate windowed query goes from 2,017 ms to 497 ms and a ten-aggregate one from 4,687 ms to 1,146 ms, while a plain-column key keeps a bit-identical estimate and its existing plan. Both settings involved are still off by default.

  • The ungrouped vectorized aggregate no longer errors on a varlena filter column (#423). SELECT count(*) FROM t WHERE s LIKE '%x%' raised “unsupported byval length: -1” with pgcolumnar.enable_ungrouped_vector_agg on. The batch fold gathers each projected column with pointer arithmetic on attlen, which is -1 for a varlena, so the offset and the fetch were both wrong. The eligibility check walked the scan keys and asked whether each type was comparable, while the gather walks the projected set and needs each type passed by value. A text column filtered with LIKE is projected and is not a scan key, so it arrived unchecked. uuid and name failed the same way for a different reason: both are fixed width, 16 and 64 bytes, but passed by reference, and the gather hardcodes by-value. Such a shape now falls back to the row path, which is what the ALTER TABLE ADD COLUMN case already did. This was ClickBench q21.

Upgrading

Run ALTER EXTENSION pgcolumnar UPDATE; in every database that has the extension, after installing this build.

The rename moves the C symbol names that each installed function recorded when it was created. Replace the shared library without this step and those records point at symbols the new library does not export. The extension then stops working until the catalog is updated. Reading an existing columnar table fails with could not find function "columnar_handler".

Nothing happens to your data, and no conversion runs. The upgrade replaces catalog entries only, and keeps each function’s identity, so the access method binding and every dependency survive. The SQL you write does not change.

See Upgrade for the commands, including how to list the databases that need it.

[1.0-alpha] - 2026-08-04

First tagged release. Everything below shipped in it.

Known limitations

  • The grouped vectorized aggregate’s parallel arm is declined on shapes with an expression grouping key, because the core Finalize is priced off a group estimate that can be 25x to 42x wrong (#369). Both settings involved are off by default.
  • The by-row-number fetch cache is bounded by `4 x (cap + retained position indexes
    • groupBuffer)rather than4 x cap`. On a table of many wide varlena columns one entry measured 62 MB against a 32 MB cap (#364). Releasing the position indexes with the decoded stream holds the bound but costs 47% in time, so this design keeps the speed and records the trade.
  • The index-fetch penalty is bounded by a multiple of one full scan rather than modelled against the consumer, so a plan that stops early inherits more of it than it should (#376). The bound keeps the penalty steering correctly on every shape measured; the model is post-alpha work.
  • Point lookups remain slower than heap, and the cost of a fetch grows with table width, because an index fetch decodes the attribute prefix up to the highest column the query reads. See docs/limitations.md.

Added

  • pgcolumnar.parallel_copy(target, path [, workers]) loads a COPY text file into a columnar table with several background workers at once, and returns the row count (#300). Each worker runs core COPY over a byte range of the file, so parse and write behavior match COPY FROM. The load is atomic through two-phase commit: every worker prepares its transaction, and a coordinator commits them together only when all succeeded, so any failure rolls the whole load back. The target is a single columnar table, where any record-aligned split is correct, or a RANGE-partitioned table with columnar partitions, where the file must be sorted ascending by the partition key and the key type must be numeric or a date/time type. The columnar encode step is CPU bound, so the load scales with worker count up to the physical core count. It landed in two parts, partition-parallel (#323) and single-table (#324). See docs/user-guide.md and docs/benchmarks.md.

  • Column projection reads only the columns a query references, and is on by default (#339, pgcolumnar.enable_column_projection). A columnar scan previously decoded every column of every row group it touched regardless of the query’s target list, which discards the main advantage of the storage format on wide tables. Measured on a 100M-row 21-column fixture, a single-column aggregate improved 6.9x. The gain is smaller on grouped queries, which also read their grouping keys: 1.24x, 1.13x and 3.13x on three TSBS-shaped grouped aggregates. Turning the setting off restores the previous behavior.

  • Vectorized aggregates, all off by default and opt-in while they are proven:

    • pgcolumnar.enable_ungrouped_vector_agg folds a plain SELECT agg(col) FROM t over the decoded column buffer instead of one Datum tuple per row (#337).
    • pgcolumnar.enable_parallel_vector_agg makes that fold parallel-aware (#343), extended to integer sum and average partials (#346), and to grouped aggregates (#366). Each worker claims distinct row groups through a shared counter and emits per-worker transition state that a core Finalize combines.
    • pgcolumnar.enable_group_vectorization answers GROUP BY from a vectorized grouped node (#321). pgcolumnar.groupagg_max_groups caps its hash table and errors with guidance rather than growing without bound.

    These remain off by default because plan selection for them is not settled: a grouped query with an expression grouping key such as date_trunc() can decline the parallel path on a group-count estimate that is 25x to 42x wrong (#369).

  • pgcolumnar.enable_index_fetch_penalty, on by default (#355), prices the per-row heap fetch of an index or bitmap path on a columnar table. A columnar fetch decodes the row group the row lives in, while core prices it as a page or two, so an unclustered ordering column made an index scan look cheap and then run for minutes decoding the table many times over. The penalty counts the distinct row groups the fetches force, interpolating on the square of the leading-key correlation. Turning it off restores the previous planner behavior.

Changed

  • The fetch cache holds the columns that fit rather than dropping a whole entry when it exceeds its size cap (#359). An entry one byte over the 32 MB cap was not retained at all, so every fetch re-read the row group and re-decoded every column it touched. On a 100M-row fixture that was 2,833 ms at four aggregated columns and 134,147 ms at five, flat on either side of the step. Each column now decodes into its own context and the one that crosses the cap is released after its value is read, so exceeding the cap costs the overflow fraction rather than everything. An earlier fix moved the decode scratch out of the cached entry, shrinking entries about 3x (#353). A group whose raw bytes alone exceed the cap is still dropped whole.

  • The index-fetch penalty is applied before the columnar path is offered to the planner, not after (#362). add_path frees a path it judges dominated, so a columnar path offered while the index paths still carried un-penalized costs was discarded, and raising those costs afterwards changed what EXPLAIN printed with nothing left to switch to. The planner chose an index scan it priced at 13,954,742 over a columnar path it priced at 589,348, running 224 seconds where the columnar path runs 4.7. Two related defects were fixed with it: the parallel columnar path was conditional on a sequential scan surviving add_path, so it did not exist on exactly the selective queries where it was needed, and the projection path read the base path’s cost after add_path may have freed it.

  • The grouped vectorized aggregate path is charged for the folding it does (#349), cpu_operator_cost per input row per aggregate. It previously priced itself just above the scan it performs, which made it unpriceable against: every competing plan paid a per-row aggregation cost and this one paid none, so it won by construction, including against a parallel plan several times faster. That cost about 1.9x on a full-scan GROUP BY with few groups.

  • The vectorized batch fold pushes scan keys, so it no longer forfeits zone-map row-group pruning (#349). The fold opened its reader with no predicates, so no group skipping occurred: on a clustered fixture with a selective predicate it read 200 of 200 row groups where the ordinary path read 2.

  • Server-file functions now gate on the pg_read_server_files and pg_write_server_files roles instead of superuser() (#330), matching core COPY ... FROM/TO 'file' so a DBA can delegate server-file access without handing over superuser. This is a deliberate loosening. The read functions (import_parquet, read_parquet, parquet_schema, the pgcolumnar_parquet foreign-table scan, import_arrow) parse files this project wrote, so they are now reachable from a role short of superuser; give an untrusted Parquet or Arrow file the care in docs/administration.md while the Arrow parser fuzzing (#214) is incomplete. The write functions (export_parquet, export_arrow, parallel_export_parquet) gate on pg_write_server_files. file_split_offsets and parallel_copy already used the read role. test/server_file_privilege.sh now covers the full set and fails if a new file function lacks a check.

  • The C standard flag for PostgreSQL 19 is probed rather than hardcoded (#294). This project sets -std=gnu23 for PostgreSQL 19, whose headers use C23 constructs. GCC 13 accepts only the older gnu2x spelling of the same language and rejects gnu23 outright, so building against PostgreSQL 19 with GCC 13 failed on a flag the user never set. The Makefile now asks the compiler which spelling it takes. Every source file compiles under GCC 13 with gnu2x against PostgreSQL 19 headers.

  • pgcolumnar.recluster records its ordered extent, so pgcolumnar.sort_status no longer reports a reclustered table as entirely unsorted (#311). It runs under a lock that permits concurrent inserts, and the mark is a boundary, so it can only be set where no other session’s group is numbered below it. The rewrite records the stripe ids it reserves and marks the contiguous run from its first; a concurrent reservation leaves a gap in that sequence whenever it commits, which the visible catalog cannot show. With no concurrent writer the whole relation is recorded. With one, the run stops where it was interrupted and the rest is reported as decay, never the reverse.

  • A row group’s bloom filter is read for the columns a query filters on, not for every column (#314). A predicate probes one column, so a group that is examined needs the filters of the columns carrying predicates and no others. bloom_pkey is (storage_id, group_number, column_index), so naming the column makes the fetch an exact index lookup rather than a range scan whose unwanted rows are discarded. Measured on one group of 200,000 rows over 12 columns with one equality predicate: 715 buffers to 323, against a floor of 251 with the bloom read deleted outright. With #310 the same probe query falls from 9577 buffers to 1547.

  • A row group’s bloom filters are read only when a predicate reaches them, not before every skip decision (#310). A bloom filter is consulted only for an equality predicate whose zone map did not already rule the group out, so a group the zone map skips needs none of them. The reader loaded them for every candidate group, and the cost scaled with the column count and the group size, because a filter holds one bitmap per column sized by the group’s distinct values.

    The scale of that is easy to understand: on a 100 million row TSBS-cpu table a single filter is 256 kB, and the whole bloom catalog is 3.5 GB, larger than the data it describes. A selective scan copied it per query through 256 kB allocations, and profiling put about 55 percent of the query’s CPU in anonymous-page faults under the group-skip check.

    On that table a clustered hostname query falls from 4610 ms to 106 ms, a factor of 43. On a smaller shape, 20 groups of 200,000 rows over 12 columns, the cost is 466 buffers per skipped group out of 504, and the query falls from 9577 buffers to 1946. Results do not change; the filter was always a pruning step.

Added

  • pgcolumnar.sort_status(rel) reports how much of a sorted table is still in sorted order (#301). vacuum_sorted and cluster order a table once; rows inserted afterwards append in insertion order, and until now nothing measured how large that unsorted tail had become. An ordering rewrite now records the row group its run ends at, in a new pgcolumnar.storage.sorted_through column, and the function reports sorted and appended groups and rows alongside the declared sort_by key. A boundary rather than a count, so retiring a group inside the run does not move the mark onto a later replacement. The mark lives on the storage row, so any rewrite resets it with no invalidation step. The online recluster does not set it and therefore reports more decay than a table has (#311).

  • Declarative sort_by clustering key (#288). pgcolumnar.set_options(t, sort_by => ARRAY['col', ...]) records a physical sort key; pgcolumnar.vacuum_sorted(t) with no columns re-applies it, like PostgreSQL CLUSTER remembering an index. The sorted rewrite works on any btree-orderable column, text included (the Z-order cluster() is numeric-only), so a segment key such as hostname tightens its zone maps and lets equality/range filters on it skip chunk groups. Stored as column names, so it survives pg_dump/restore. Not auto-maintained; re-run after inserts. Virtual generated columns are rejected as a sort key.

  • Column-oriented table access method (USING pgcolumnar) with per-column compression, chunk-group minimum and maximum skipping, per-chunk bloom filters, and a vectorized aggregate path.

  • Native on-disk format PGCN v1: row groups, per-column chunks, an adaptive per-vector encoding cascade, zone maps for skipping, and per-chunk bloom filters. Delete, update, index scan, index-only scan, and projections all work on native tables. The earlier 1.0-dev format line has been removed; the v1.0-dev git tag preserves it.
  • Compression codecs none, pglz, lz4, and zstd. lz4 and zstd are compiled in when their system libraries are present.
  • count(*) answered from catalog metadata without scanning.
  • Parallel scan.
  • Read stream prefetch in the scan on PostgreSQL 17 and later (pgcolumnar.enable_read_stream).
  • Full index-only scan through a columnar visibility-map fork, with lazy VACUUM setting all-visible bits and clear-on-write, on by default (pgcolumnar.enable_index_only_scan).
  • Multiple projections (C-Store model): a pgcolumnar.projection catalog, write fan-out, planner projection scan, back-fill, and vacuum coordination (pgcolumnar.add_projection, pgcolumnar.drop_projection, pgcolumnar.enable_projection_scan).
  • Sorted storage with pgcolumnar.vacuum_sorted.
  • Arrow IPC and Parquet export (pgcolumnar.export_arrow, pgcolumnar.export_parquet), self-contained with no libarrow or libparquet dependency. Coverage: scalar types (int2/4/8, float4/8, bool, text/varchar, bytea, date, time, timestamp, timestamptz, uuid, numeric, json), one-dimensional arrays, and composite types, with nulls at every level.
  • Arrow IPC and Parquet import (pgcolumnar.import_arrow, pgcolumnar.import_parquet). The Parquet reader parses Thrift metadata, decompresses uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, and decodes PLAIN and dictionary encodings from data-page versions 1 and 2. Both readers reconstruct one-dimensional arrays and composite types: Arrow from its List and Struct buffers, Parquet from the Dremel repetition and definition levels.
  • Reading external Parquet in place. pgcolumnar.read_parquet(path) returns a file’s rows without importing, pgcolumnar.parquet_schema(path) reports its columns and inferred types, and the pgcolumnar_parquet foreign-data wrapper exposes a file as a foreign table. A path may be a single file, a directory of *.parquet files, or a glob pattern, read as one relation in sorted order. The foreign scan skips row groups excluded by the query’s predicate (min/max statistics) and decodes only the referenced columns; EXPLAIN ANALYZE reports the row groups and columns read and skipped and the number of files.
  • Value encodings are chosen from a strided sample rather than by applying every candidate to every vector. Measured on a 6,000,000-row load: 20.9 s to 15.7 s, with byte-identical output. pgcolumnar.encoding_sample_rows controls the sample size and 0 restores the previous exhaustive selection.
  • Partition values are percent-decoded, so a directory named region=a%3Db reads as a=b, and __HIVE_DEFAULT_PARTITION__ reads as NULL rather than as that literal string, matching what Hive and Spark write.
  • Hive-style partitioning on the pgcolumnar_parquet foreign-data wrapper. A foreign table declaring partition_columns reads col=value directory names as column values, and a predicate on a partition column drops whole files before they are opened, so a pruned file costs no I/O. EXPLAIN ANALYZE reports Files Pruned. The columns are declared rather than inferred, and a file missing a declared component raises rather than yielding nulls.
  • A directory path now reads *.parquet files at any depth below it, where it previously read only the files directly inside. Entries whose name begins with _ or . are skipped, so a Spark or Hive output directory does not read its own _temporary staging tree. A directory reached through a symbolic link is not descended, since a link to an ancestor would make the walk endless; a symbolic link to a file is still followed. Nesting deeper than 32 levels raises rather than reading part of the tree.
  • External Parquet files are read on demand instead of loaded whole. The reader holds a file’s footer for the scan and pulls one page at a time, so peak memory for raw file data is one page rather than one file. A file of 1GB or more could not be read at all before this, because the whole-file allocation exceeded MaxAllocSize; that ceiling is gone. A row group excluded by predicate pushdown is now never read from disk, and pgcolumnar.parquet_schema reads only the footer.
  • A Parquet DECIMAL is also read when it is stored as an INT32 or INT64 holding the unscaled integer, which is how writers store small precisions; pgcolumnar.parquet_schema advises numeric(p,s) for those columns.
  • Parquet read type coverage extended to uuid and numeric (from fixed and variable DECIMAL, precision up to 38), fixed-length binary, and millisecond, microsecond, and nanosecond time units.
  • pgcolumnar.fsst_min_gain_percent, a cost margin for the FSST string encoding decision. FSST is kept only when it reduces the compressed chunk by at least this percentage, default 5. Building FSST codes for every vector is one of the larger costs of a text or varlena load, and a sub-margin reduction does not repay it.
  • The on-disk format version is enforced when data is read, not only stamped when it is written. Both the physical metapage version and the native data format version are checked, on every path that decodes columnar data, so a file this build cannot read is refused rather than misread.
  • User and administrator documentation under docs/: installation, user guide, administration, configuration reference, SQL reference, and limitations.
  • Benchmark harness (bench/run_bench.sh) covering storage size, query latency, vectorization, compression, sorted projection, index-only scan, projection scan, export, import, nested round-trip, and cross-engine reads of the Parquet output with DuckDB and pyarrow.
  • Project logo under logo/.

Fixed

  • Bounded importer memory. pgcolumnar.import_arrow and pgcolumnar.import_parquet built each row’s arrays and composites in one memory context and did not free them, using memory proportional to the row count. They now reset a per-row scratch context (and, for Parquet, a per-row-group context for decoded leaf streams), so peak memory stays bounded on large files.
  • Hardened the Parquet reader against crafted files. File-declared page sizes, DECIMAL scale, and per-row-group column-chunk counts are range-checked, so a malformed footer yields a clean decode error rather than a stack overflow, an out-of-bounds read, or a wrong value. Float and double row-group skipping accounts for NaN and for inverted min/max intervals, and narrowing a wide Parquet value into a smaller PostgreSQL type raises instead of wrapping.
  • Concurrent inserts of the same unique-index key now serialize correctly with a transaction-scoped advisory lock (pgcolumnar.enable_unique_insert_lock).
  • Lost delete marks under concurrent same-chunk-group deletes.
  • Relation-reference leak in parallel CREATE INDEX.

Removed

  • The decompressed-chunk cache, and the pgcolumnar.enable_column_cache and pgcolumnar.column_cache_size settings with it. Its only entry point had lost its caller when the earlier on-disk format was removed, so the cache had done nothing since. Two settings and four passages of documentation described a feature that did not run. A postgresql.conf that sets either parameter must drop the line. The implementation is in the git history if the performance case is made again against the current reader.

Changed

  • FSST string encoding is now kept only when it reduces the compressed chunk by at least 5 percent, rather than on any reduction at all. On shapes where FSST barely wins, such as high-entropy text, this costs about 2 percent stored size and reduces load time by roughly a third. Where FSST wins by more than the margin the encoding and the stored bytes are unchanged. Set pgcolumnar.fsst_min_gain_percent to 0 for the previous behaviour.
  • Renamed the per-table option functions to pgcolumnar.set_options and pgcolumnar.reset_options. The previous names were carried over from an earlier compatibility goal that no longer applies. No aliases are kept, since the project is pre-release.

Compatibility

  • Builds from one source tree on PostgreSQL 15 through 19. Every test suite runs on all five majors.
  • The Arrow and Parquet import and export functions require superuser and run on little-endian hosts.
  • Cross-major pg_upgrade is covered by an opt-in gate (PGC_RUN_UPGRADE=1 test/run_all_versions.sh), in both copy and link transfer modes.
  • All recorded test results come from x86_64. The suites have not been run on aarch64 or on a big-endian platform.