pg_mentat → embedded Mentat: advances port inventory

Comparison of ~/ws/pg_mentat (PostgreSQL extension, pgrx, v1.5.7) against ~/ws/mentat (embedded SQLite fork). Goal: identify which advances in the Postgres codebase can be brought into the embedded SQLite Mentat, excluding anything that only exists because it is a Postgres extension.

Verified against source on the dates in git. All classifications cite files.

TL;DR

  • The Datalog engine, pull, transact, rules, time-travel, excision in pg_mentat live in pg_mentat/src/functions/*.rs and generate SQL strings run through pgrx Spi. The algorithms are mostly storage-agnostic Rust; the storage touch-points are localized (e.g. query.rs is 7,556 LOC with only ~42 Spi/pgrx references).
  • The gating dependency for every query-side port is the diverged edn crate: pg_mentat’s ParsedQuery and PullAttributeSpec are supersets of the embedded ones. Port edn grammar first, then features.
  • core / core-traits are essentially identical between the two → low friction.

Classification legend

  • PORTABLE — A lacks it; logic is storage-agnostic Rust or SQL-string generation retargetable to SQLite. No Postgres-only feature required.
  • ADAPT — Portable in principle, but the impl leans on a Postgres feature (triggers, sequences, WITH RECURSIVE specifics, multi-table cascade, schema-per-store) needing a SQLite equivalent or rework.
  • PG-ONLY — Exists only because it is a PG extension. Excluded from the port.
  • ALREADY-IN-A — Embedded Mentat already has it (may need validation).

Verified findings (spot-checked against source)

Claim Verified
A has SAVEPOINT primitives but no :with/speculative op ✅ ~/ws/mentat/transaction/src/lib.rs:323 has savepoint/rollback_savepoint/release_savepoint; no speculative txn built on them
B implements :with via savepoint ✅ pg_mentat/src/functions/transact.rs:288 mentat_with → execute_speculative_transaction (SAVEPOINT)
A has no :as-of/:since query integration ✅ grep of query-algebrizer + timelines.rs empty
A ParsedQuery lacks in_bindings, offset, distinct, rules ✅ ~/ws/mentat/edn/src/query.rs:946 vs ~/ws/pg_mentat/edn/src/query.rs:989

Advances ranked by port effort

Advance Class LOC in B Effort Source file (B)
:as-of / :since / history queries PORTABLE 655 Low functions/time_travel.rs
Speculative transactions (:with) PORTABLE ~500 Low functions/transact.rs:263+
Pull: reverse refs, nested/recursive, defaults, rename, limit PORTABLE ~600 net Low–Med functions/pull.rs (2,964)
EDN: offset, in_bindings, distinct, rules PORTABLE ~100 Low (high cascade) edn/src/query.rs
Recursive rules + cycle detection PORTABLE 512 Med functions/recursive_queries.rs
Virtual tables (auto UNION-ALL views) PORTABLE 1,341 Med functions/virtual_tables.rs
Prepared-statement cache PORTABLE ~200 Low in functions/query.rs
GDPR excision ADAPT 180 Med functions/excision.rs
Materialized views ADAPT 613 Med functions/materialized_views.rs
Multi-store namespacing ADAPT 484 Med functions/store_management.rs
Partition banding + collision repair ADAPT 512 Med (schema/bootstrap)
Append-only log + current-state projections ADAPT schema-level High lib.rs + bootstrap DDL
Reactive subscriptions (LISTEN/NOTIFY) PG-ONLY 481 — functions/subscriptions.rs
Hot-standby read path PG-ONLY — — —
Full-text BM25; pgvector / pg_trgm / PostGIS / pg_infer / rum / fuzzystrmatch soft integrations PG-ONLY 2,000+ — *_tests.rs, where-fns

Full Datalog (or/or-join/not/not-join), named rules

Both codebases parse these (edn/src/query.rs WhereClause has OrJoin, NotJoin, RuleExpr; RuleInvocation/Rule/RuleClause present in both). A has query-algebrizer/src/clauses/or.rs and not.rs. Classification: LIKELY-ALREADY-IN-A, integration/coverage needs validation before assuming parity. B’s rules: Vec<Rule> on ParsedQuery is the recursive-rule hook A lacks.

Coupling detail (PORTABLE / ADAPT items)

  • :as-of / :since — ~95% pure Rust. Temporal predicate is WHERE tx <= N; basis-t tracking is pure Rust. Boundary: query translation.
  • :with — ~98% storage-agnostic. rusqlite already supports SAVEPOINT (A already exposes it). Boundary: transaction wrapper.
  • Pull richer features — reverse refs = standard SQL join + a new PullAttributeSpec variant; nested/recursive = a recursive Rust descent with a depth guard (same execution boundary as A’s existing pull). Gate: edn PullAttributeSpec extension.
  • Recursive rules — cycle detection is pure Rust; CTE generation emits WITH RECURSIVE (SQLite ≥ 3.8.3 supports it). Replace Spi exec with rusqlite.
  • Excision — referential checks + multi-table DELETE ... WHERE e IN (...). SQL is portable; swap Spi for rusqlite. Adaptation: A’s single denormalized datom table vs B’s nine typed tables changes the DELETE fan-out.
  • Materialized views — CREATE VIEW is portable; auto-refresh triggers are Postgres-native — rusqlite has no trigger API, so refresh becomes manual/explicit.
  • Append-only log — B separates immutable log from current_<type> projections; A conflates them. High-effort schema restructure with a migration path.

edn / core / core-traits divergence (porting friction)

  • edn — friction HIGH. ParsedQuery in B adds in_bindings: Vec<Binding>, offset: Offset, distinct: bool, rules: Vec<Rule> (~/ws/pg_mentat/edn/src/query.rs:989) that A’s lacks (~/ws/mentat/edn/src/query.rs:946). PullAttributeSpec in B is a superset (reverse refs, defaults, rename, recursion limit). Adding these cascades into the algebrizer’s validation. Port edn grammar first.
  • core — friction LOW. TypedValue, ValueType, Entid, KnownEntid are the same; ~733 LOC each.
  • core-traits — friction MEDIUM. Trait boundaries stable, but B may carry extra MentatError variants for new features (excision, subscriptions, matviews). Diff error enums before porting.

Suggested staging (not a commitment — for scoping only)

  1. edn grammar superset (offset, in_bindings, distinct, rules, richer PullAttributeSpec) — unblocks everything else.
  2. Low-risk query features: :as-of/:since, :with (A already has the savepoint primitive), richer pull.
  3. Recursive rules + cycle detection (needs rules from step 1).
  4. ADAPT items (excision, matviews, multi-store) as separately-scoped work.
  5. Never: subscriptions, hot-standby, PG soft integrations, BM25.

Progress log

Step 1 — edn grammar superset — DONE

Ported the query AST + PEG grammar superset into A’s edn and threaded the new fields through the algebrizer. Additive, minimal-diff; all workspace tests green.

  • edn/src/query.rs: added Offset enum; RuleInvocation/RuleClause/Rule structs; changed WhereClause::RuleExpr unit variant → RuleExpr(RuleInvocation); added ParsedQuery fields in_bindings, offset, distinct, rules; added QueryPart variants + from_parts parsing with scalar-binding→in_vars back-compat.
  • edn/src/lib.rs (PEG grammar): added offset(), rule_invocation(), rule_head(), rule_clause(), rule_definitions(); wired rule_invocation() into where_clause(); added :offset, :distinct, :rules query parts.
  • query-algebrizer/src/{types,lib}.rs: added offset/distinct/rules to FindQuery and threaded them through from_parsed_query + simple (data no longer silently dropped; SQL generation for them is later per-feature work).
  • edn/tests/query_tests.rs: 7 new parser tests (offset fixed/var/zero, distinct, rule invocation, recursive rule definitions, reserved-clause guard).

Deliberate scoping decisions (deviations from pg_mentat, on purpose): - Kept Limit::None (did NOT rename to B’s Limit::Unlimited): the rename would ripple across algebrizer/projector for zero functional gain. - Kept :in parsing as InVars (B changed it to binding-form InBindings): avoids regressing A’s existing :in tuple/coll/rel handling. InBindings + in_bindings are ported as dormant infra (marked with a ponytail: note), wired when a feature needs binding-form :in. - Used Datomic-standard :rules for rule definitions, not B’s overloaded :with (A’s :with already means with-variables — overloading is ambiguous). - Deferred the Pattern.added 5th-element field (history queries [?e ?a ?v ?tx ?added]): would touch ~33 Pattern call sites + the algebrizer’s evolved-pattern machinery for a feature not yet wired. Add when porting history queries.

Next: low-risk query features — :as-of/:since and :with (A already has the SAVEPOINT primitive at transaction/src/lib.rs:323). Then recursive rules (now that rules is parsed and carried through to FindQuery).

Step 2 — speculative transactions (:with / d/with) — DONE

Ported pg_mentat’s mentat.with semantics to embedded Mentat. B implements it by running the normal transact code path inside a PG subtransaction (PL/pgSQL RAISE-to-rollback) and capturing the report; A does the direct SQLite equivalent with a SAVEPOINT (which A already exposed).

  • transaction/src/lib.rs: added InProgress::transact_speculative and transact_entities_speculative. Snapshots partition_map + schema, opens a mentat_speculative SAVEPOINT, runs the real transact_entities, then ROLLBACK TO + RELEASE the savepoint and restores the in-memory snapshots — so the InProgress is left as before, and no datoms, entids or schema changes persist. Same code path as a real transact → identical tempid resolution / constraint checking.
  • src/conn.rs: added Conn::transact_speculative convenience wrapper (opens an IMMEDIATE transaction, runs the speculative transact, drops without committing).
  • Test conn::tests::test_transact_speculative: proves the report reflects the change, the datom does NOT persist, the entid is NOT consumed, and a subsequent real transact reuses the reported entid.

No Postgres-only machinery needed; fully storage-agnostic on SQLite.

Next: :as-of / :since temporal queries (WHERE tx <= N; basis-t is pure Rust). Then recursive rules (uses FindQuery.rules from step 1).

Sequencing correction (after reading A’s architecture more closely)

The first-pass effort estimates for :as-of/:since and recursive rules were too optimistic — they assumed B’s storage/query shape. A differs:

  • :as-of / :since — NOT low-effort in A. A’s datoms table holds current state only; full history lives in timelined_transactions / the transactions view (db/src/db.rs:171 vs :197). So :as-of N is not a WHERE tx <= N filter on the current query path (that path can’t see retracted/superseded values) — it needs point-in-time state reconstruction by replaying the log up to N. That touches the storage/query core → reclassify ADVANCE-NEEDS-ADAPTATION (High), tied to the append-only-log restructure. A already has a specialized tx-log query API (query-algebrizer/src/clauses/tx_log_api.rs) but not general point-in-time Datalog.
  • Recursive rules — NOT low-effort in A. B’s real rule expansion is woven into its 7,556-LOC SQL-string generator (functions/query.rs ~2149–2586), emitting WITH RECURSIVE CTEs inline. A instead algebrizes to a SelectQuery IR; expanding RuleExpr means new logic in the CC (ConjoiningClauses) machinery. Reclassify Medium–High. functions/recursive_queries.rs is a standalone mentat.recursive(raw_sql) convenience — PG-only, not the Datalog path.

Revised next step: richer Pull — genuinely low-risk. A’s pull is a self-contained 272-LOC crate (query-pull/src/lib.rs) with a clean Puller/PullAttributeSpec abstraction whose enum already has commented-out placeholders for the exact features B adds (LimitedAttribute, DefaultedAttribute). It operates on attribute lists + entity ids via lookup_values_for_attribute, independent of the algebrizer/storage-history complexity.

Step 3 — reverse-reference pull (:ns/_attr) — DONE

Ported reverse pull from pg_mentat, staying on SQLite. B does this in its SQL-layer pull.rs with its own spec parser; A does it through the edn AST + the Puller, querying the VAET index directly.

  • edn/src/query.rs: added reverse: bool to NamedPullAttribute (forward ident is stored; reverse marks the walk direction); updated From/Display.
  • edn/src/lib.rs (grammar): pull_attribute() now accepts a backward namespaced keyword (:person/_friend), storing k.to_reversed() as the forward ident with reverse: true.
  • query-pull/src/lib.rs: Puller gained reverse_attributes; prepare routes reverse specs (default output key = reversed keyword, or :as alias); pull runs SELECT DISTINCT e FROM datoms WHERE a=? AND v=? AND value_type_tag=0 AND index_vaet IS NOT 0 per pulled entity and emits the referrers as a Binding::Vec of Refs. (Refs store as plain ints, tag 0.)
  • query-pull/Cargo.toml: added db_traits path dep (already in tree) so rusqlite errors map to PullError via DbError.
  • Tests: tests/pull.rs::test_reverse_pull (end-to-end via (pull ?e [:person/_friend]): two referrers found, empty map when none) and edn/tests/query_tests.rs::can_parse_reverse_pull_attribute.

:as rename was already supported. Nested/recursive pulls, defaults and limits remain deferred (the crate’s own TODOs; larger cache-walk changes).

Next: :as-of/:since (High, needs point-in-time reconstruction from the transactions log) and recursive rules (Medium–High, algebrizer CC changes).

Step 4 — named (non-recursive) rules — DONE

Rule invocations are now expanded, not unimplemented!(). Implemented as an AST→AST pre-pass (expand_rules in query-algebrizer/src/lib.rs) rather than inside the CC machinery, so the query IR is untouched.

  • query-algebrizer/src/lib.rs: expand_rules inlines single-clause, non-recursive rule invocations — substitutes invocation args for head params, gensyms body-local vars to avoid capture, recurses into or/not bodies and into rules that call other (non-recursive) rules (depth-capped at 32).
  • query-algebrizer/src/clauses/mod.rs: the RuleExpr arm is now a clean UnknownRule error instead of a panic (rules are expanded away before this).
  • query-algebrizer-traits/errors.rs: added UnknownRule, RuleArgumentMismatch, RecursiveRuleUnsupported, MultiClauseRuleUnsupported.
  • Tests (tests/query.rs): test_named_rule_expansion (a grandparent rule of two parent patterns resolves correctly) and test_recursive_rule_errors_cleanly (recursive/multi-clause rule errors, no panic).

Deliberate scope (bounded, ponytail): single-clause + variable-args only. Multi-clause rules (OR alternatives) and recursive rules need WITH RECURSIVE in the SelectQuery IR — rejected with a specific error, deferred as future work. Constant rule args also deferred.

Step 5 — :as-of / :since — DEFERRED (High, storage-model work)

Confirmed not tractable as a small port: A’s datoms table is current-state only; point-in-time queries need reconstruction from the transactions log (db/src/db.rs). This is the append-only-log / current-projection restructure flagged as High. Left for a dedicated effort; the tx-log query API (clauses/tx_log_api.rs) already covers basic log reads.

Mentat port status summary

Advance Status
edn grammar superset (:offset/:distinct/:rules, rule invocations) DONE
Speculative :with transactions DONE
Reverse-ref pull (:ns/_attr) DONE
:as pull rename already present
Named non-recursive rules DONE
Recursive rules (WITH RECURSIVE) deferred — needs SQL IR work
:as-of / :since, history deferred — needs log-replay/storage restructure
Nested/recursive pull, defaults, limits deferred — cache-walk changes
Excision, materialized views, multi-store deferred — ADAPT
Subscriptions, hot-standby, PG soft integrations, BM25 out of scope (PG-only)