Running the pytest harness

This is the issue #432 pilot. It runs beside test/*.sh, and replaces nothing.

  • TESTS.md in this directory documents every test and every assertion helper.
  • VACUITY_MODES.md is the inventory of ways a pytest harness can report a false pass: 79 modes produced by the enumeration, 72 of them named in that file, 73 demonstrated by a run, and 28 refused by this layer today. VACUITY_MODES.md section 1a gives the counting rule and reconciles the run’s totals against what is actually written down.
  • design/ISSUE_432_PYTEST_HARNESS.md holds the design and the measurements behind each guard.

Prerequisites

The interpreter is marked EXTERNALLY-MANAGED, so install into a virtual environment rather than into system Python:

apt-get install -y python3.14-venv        # ensurepip is not in the base image
python3 -m venv /root/pyenv
/root/pyenv/bin/pip install -r test/pytest/requirements-test.txt

Running it

cd test/pytest
PYTHONPATH=. /root/pyenv/bin/pytest                      # serial
PYTHONPATH=. /root/pyenv/bin/pytest -n 4                 # four workers
PYTHONPATH=. /root/pyenv/bin/pytest --pgc-expect-tests 24 # assert the run's shape
PGC_PG_CONFIG=/usr/local/pg19a/bin/pg_config PYTHONPATH=. /root/pyenv/bin/pytest

Each worker builds its own throwaway cluster on a port derived from its worker id, and drops it at session end. Nothing survives a run.

Checking a port against its bash original

/root/pyenv/bin/python test/pytest/compare_to_bash.py \
    test/native_projection.sh test/pytest/test_native_projection.py

It compares the two by assertion NAME and exits non-zero if the bash suite asserts a property the port does not. A port keeps this working by passing each assertion the same name string the bash check uses.

Both halves are in the gate (#1016)

test/run_all_versions.sh does not run these tests and must not: the two harnesses stay independent, and the shell runner invoking pytest is the cross-harness call the project forbids. Registering the run in SUITES was the plan recorded in section 1a of the design document, and it was the wrong mechanism for that reason. A second CI job is the right one.

ci.yml runs two:

  • pytest-guards runs the files NO_CLUSTER names, in a venv where psycopg is deliberately ABSENT. That absence is what proves those files need no database.
  • pytest-cluster runs the complement, with every pin from requirements-test.txt and a PGDG PostgreSQL 18 with its headers.

Both pass --pgc-expect-tests from expected_tests.txt, so a run that collects fewer tests than it should fails instead of reporting a green that means nothing. Adding a test moves a number in that file, and the diff sits next to the test that moved it.

Warnings

The vacuity layer is loaded through pytest.ini and cannot be turned off by a test file. A test that concludes nothing fails, a bare skip fails the run, and a comparison that could not have failed is refused. If a guard blocks something legitimate, the escape hatches take a reason rather than a flag, and every one of them is listed in the design document. Adding a new escape hatch needs a red test that proves the guard still fires without it.