Design: user-defined transaction functions (Codeberg #325)

Status: implemented in 1.14.0 (PostgreSQL, script builds) — crates/pg/pg_mentat/src/functions/txfn.rs, tests in src/txfn_tests.rs. The decisions below are as built; where the first draft differed, it says so.

The gap

Datomic lets you install a database function and invoke it inside a transaction:

;; install
[{:db/ident :inc
  :db/fn #db/fn {:lang "clojure"
                 :params [db e a]
                 :code "(let [v (:v (d/entity db e))] [[:db/add e a (inc v)]])"}}]
;; use
[[:inc 42 :counter/n]]

The function receives the in-progress database value and returns more transaction data, which is spliced into the same transaction and processed recursively. This is how Datomic teams enforce atomic invariants (unique counters, conditional writes, derived facts) without a read-modify-write race.

pg_mentat today ships only two built-in tx-fns, :db.fn/cas and :db.fn/retractEntity (transact.rs, recognize_tx_fn). There is no way to install your own.

What we already have to build on

  • A sandboxed mino (Clojure-dialect) interpreter: edn_eval / functions/script.rs::build_interpreter() uses mino_rs::Interpreter::sandboxed() — no filesystem, no net, with step / heap / depth GUC limits and an interrupt hook. This is the project’s standing, security-reviewed way to run user code in the backend.
  • A ScriptBackend the interpreter already exposes to scripts: q, q_with_inputs, pull, entity, read, datoms, resolve_eid, basis_tx (crates/script/src/lib.rs). A tx-fn needs exactly these to read the in-progress db.
  • The transactor’s expansion loop already splices generated datoms: pending_datoms plus the built-in execute_*_fn functions show the shape a user fn would feed into.

So a user tx-fn is: run a sandboxed mino function, passing it a db value and its args; it returns tx-data (EDN); splice that back into the same transaction and keep processing. The machinery exists; this is wiring, not new infra.

Decisions

D1. Language: mino, not Clojure/JVM

pg_mentat has no JVM. The sandboxed mino interpreter is the only in-backend code runner we have and is already security-reviewed. Decision: tx-fns are mino functions, stored as the function’s source. (Datomic’s :lang "clojure" has no equivalent; we accept :lang "mino" or omit it.)

Trade-off: not source-compatible with Datomic :db/fn. A migrating team rewrites fn bodies, but the data model, d/entity-style reads, and the return-tx-data contract match, so the rewrite is mechanical.

D2. Where the fn source lives

Store it as a new value type on the ident entity, mirroring Datomic’s :db/fn. Add a bootstrap attribute:

:db/fn   :db.type/string   :db.cardinality/one

The string holds the mino source of a fn form: (fn [db e a] ...). Resolved at invocation by entid/ident, same as any attribute. No new storage machinery — it is a plain string datom.

Trade-off vs a dedicated bytecode/opaque type: a string is transparent (shows in pull, pg_dump), re-parsed each call (cheap: mino parse is fast and we cache the compiled fn per-transaction, see D5). Opaque storage would hide the code and complicate backup. String wins.

D3. Invocation syntax

A vector whose head is a namespaced keyword that resolves to an entity carrying :db/fn:

[[:my/inc 42 :counter/n]]     ; :my/inc is an ident with a :db/fn value

The transactor’s own list forms (:db/add, :db/retract, :db/cas, :db/retractEntity, anything in :db.fn/*) are never looked up. A keyword head with no :db/fn is left untouched, so the transactor reports its usual error.

D4. The fn contract

(fn [db & args] -> tx-data): - db is the db-value map the script layer already uses (DbRef: {:mentat.store/db true :conn N :basis-tx T ...}), at the basis before this transaction – as in Datomic. (The first draft proposed the in-progress basis; that would expose half-applied state, and pg_mentat applies a transaction only after expansion anyway.) For an atomic read-modify-write, return :db.fn/cas in the tx-data, as Datomic users do. - returns EDN tx-data (a vector, or a list, of entities), parsed and spliced in place of the call; it may itself invoke other tx-fns (recursion, bounded by the step/depth limits + mentat.txfn_max_depth, D6).

D5. Execution model

A pre-pass over the transaction vector, before the transactor’s two passes: each invocation is replaced (recursively) by the tx-data it returns, so the transactor itself is unchanged. (The first draft proposed a work-list inside the transactor’s entity loop; that loop iterates a fixed vector, and a pre-pass avoids restructuring it and can’t disturb tempid / upsert ordering.) One sandboxed interpreter per edn_t call, built lazily on the first invocation; the fn source is re-read per call. The interpreter’s ScriptBackend reads through SPI as the caller.

Trade-off: building the interpreter has a fixed cost (~ms). A transaction with no tx-fns pays nothing (lazy-build on first use). A transaction with many tx-fn calls amortizes one interpreter.

D6. Safety (hard requirements, non-negotiable)

  • Runs under Interpreter::sandboxed() — no slurp/spit/rm/net.
  • Bounded by the existing mentat.script_max_steps / _heap_bytes / _depth GUCs (PGC_SUSET), plus a new mentat.txfn_max_depth (default 16) for tx-fn recursion.
  • Never SECURITY DEFINER; reads go through the caller’s SPI with the caller’s privileges, exactly like edn_eval (standing project rule).
  • The whole transaction is already atomic; a tx-fn that errors aborts the tx.
  • Installing a :db/fn is just transacting a string, so it needs only write access to the schema — same trust boundary as defining any attribute. (A future hardening GUC could gate :db/fn installation to a role; noted, not built — YAGNI until asked.)

What shipped (1.14.0)

  1. Bootstrap :db/fn string attribute, entid 20 (fresh installs and the 1.13.0--1.14.0 upgrade).
  2. txfn::expand_user_tx_fns, called from execute_transaction_inner right after parsing, in script builds.
  3. GUC mentat.txfn_max_depth (default 16), plus the existing step/heap/depth limits.
  4. Tests (txfn_tests.rs): an :inc counter that reads with q; a fn that returns a lookup-ref :db.fn/cas and one that calls it (and the cas guard still aborts a stale call); clear errors for a throwing fn and a non-vector result; bounded self-recursion; a sandbox escape attempt (slurp) refused; a keyword head with no :db/fn left to the transactor; and the documented :acct/deposit example.

PostgreSQL only. The embedded SQLite and DuckDB backends have the same ScriptBackend seam, so the same pre-pass can run there; a follow-up.

Why this is safe to ship incrementally

It reuses the exact sandbox and limits already reviewed for edn_eval, adds no new host capability, and degrades to today’s behavior when no :db/fn is installed. The feature is additive and gated by the presence of a :db/fn datom.