Architecture

Mentat is one repository that builds two storage backends behind a single Datalog/EDN front-end. This chapter covers the front-end/backend split, the crate map, and which features live on which backend, then the internals of the PostgreSQL backend (storage layout, query compilation, transactions, pull).

Front-end / backend split

A query, transaction, or schema definition is EDN text. Both backends parse it with the same three shared crates:

  • edn — the EDN reader and the Datalog grammar. It produces a ParsedQuery (find spec, where clauses, :in bindings, :with, rules) and parsed transaction entities, independent of any backend. It caps nesting depth before parsing so deeply nested input is rejected with an error rather than overflowing the stack.
  • core-traits — the fundamental value model: ValueType, TypedValue, ValueTypeSet, and the type/value mappings.
  • core (mentat_core) — conversions between EDN and the value model, reusable keyword constants, schema traits, and shared utilities.

Because a query is parsed by one grammar, it means the same thing on either backend. The backends diverge only below the parsed representation, where each turns it into storage-specific operations:

  • The embedded backend algebrizes the parsed query into SQL over SQLite’s datoms table (query-algebrizer → query-projector → query-sql) and transacts through its own SQLite transactor (db, transaction).
  • The PostgreSQL backend compiles the same parsed query into SQL over its narrow per-type tables and executes it through PostgreSQL’s SPI (crates/pg/pg_mentat).

Crate map

crates/
├── edn/                 shared: EDN reader + Datalog grammar
├── core-traits/         shared: ValueType, TypedValue, ValueTypeSet
├── core/                shared (mentat_core): EDN↔value conversions, schema traits
├── mino/                the mino-rs interpreter (mentat.store/* prims) + tests/corpus/
├── sqlite/              the embedded backend
│   ├── mentat/          the `mentat` crate: Store, Conn, InProgress, scripting glue
│   ├── db/ db-traits/   SQLite storage: bootstrap, transactor, attribute cache
│   ├── transaction/     in-progress transaction wrappers
│   ├── sql/ sql-traits/  SQL text generation for SQLite
│   ├── query-algebrizer{,-traits}/   parsed query → algebraic query
│   ├── query-projector{,-traits}/    algebraic query → output shaping
│   ├── query-pull{,-traits}/         pull execution
│   ├── query-sql/       algebraic query → SQL
│   ├── ffi/             C ABI over the embedded store (mentat_ffi)
│   └── cli/             embedded CLI (mentat_cli)
└── pg/
    ├── pg_mentat/       the pgrx extension (SQL functions, narrow tables, SPI)
    └── mentatd/         HTTP/WebSocket server over tokio-postgres

A plain cargo build at the repo root builds only the SQLite side and the shared front-end — no pg_config, libclang, or PostgreSQL required. The extension builds with cargo pgrx from crates/pg/pg_mentat; mentatd builds with cargo build -p mentatd.

Features by backend

Feature Embedded (mentat) PostgreSQL (pg_mentat) SQLite ext (mentat_sqlite_ext) DuckDB (mentat_duckdb)
SQL functions edn_t / edn_q / edn_pull / edn_eval Rust API yes yes yes
Datalog q, pull, schema, transact yes yes yes yes
:db.fn/cas, :db/retractEntity yes yes yes yes
?added / history patterns in :where yes yes yes yes
Historical q (as-of / since) yes yes yes yes
:in scalar / collection / tuple / relation, mixed yes yes yes yes
mino scripting (edn_eval) mino feature script feature on by default on by default
C ABI (ffi) yes — — —
Join Datalog results against native SQL tables — via JSONB via json_each yes (table function)
Reactive subscriptions (LISTEN/NOTIFY) — yes — —
BM25 full-text, pgvector, pg_trgm, PostGIS, rum — yes — —
Materialized / virtual views, multi-store, excision — yes — —
mentatd HTTP/WebSocket server — yes — —

The SQLite and DuckDB extensions embed the SQLite-backed store, so they support whatever the embedded engine supports; each call takes the store’s file path first. On the embedded Rust API, historical q is Store::q_once_as_of(tx, …) / q_once_since(tx, …), and mixed :in inputs are built with QueryInputs::merge. The SQL functions all take the same options JSON: {"inputs": [...]}, {"asOf": tx}, {"since": tx}.

The DuckDB edn_q returns every column as VARCHAR for now; typed columns and a DuckDB-native storage backend are planned (see duckdb-extension-plan.md). The SQLite extension’s edn_q returns JSON text in pg_mentat’s {"columns": [...], "results": [...]} shape.

PostgreSQL backend internals

pg_mentat implements the Datomic data model within PostgreSQL using pgrx (Rust PostgreSQL extension framework). The rest of this chapter describes its storage layout, query compilation pipeline, transaction processing, and pull architecture.

Storage Model

Narrow Per-Type Tables

pg_mentat stores datoms (facts) in nine type-specific narrow tables. Each table stores values of exactly one PostgreSQL native type, enabling the query optimizer to use type-appropriate comparison operators and indexes without casts.

Table Value Column Type Datomic Type
mentat.datoms_ref_new BIGINT :db.type/ref
mentat.datoms_boolean_new BOOLEAN :db.type/boolean
mentat.datoms_long_new BIGINT :db.type/long
mentat.datoms_double_new DOUBLE PRECISION :db.type/double
mentat.datoms_text_new TEXT :db.type/string
mentat.datoms_keyword_new TEXT :db.type/keyword
mentat.datoms_instant_new TIMESTAMPTZ :db.type/instant
mentat.datoms_uuid_new UUID :db.type/uuid
mentat.datoms_bytes_new BYTEA :db.type/bytes

Each table has columns: e (entity), a (attribute entid), v (typed value), tx (transaction ID), added (boolean: assert or retract).

Compatibility View

A mentat.datoms view is provided as a compatibility shim over the nine narrow tables with INSTEAD OF triggers. This allows legacy code to INSERT INTO mentat.datoms or DELETE FROM mentat.datoms – the triggers route to the appropriate narrow table. New code should use the narrow tables directly.

Covering Indexes

Each narrow table has four covering indexes, mirroring Datomic’s index model:

Index Column Order Use Case
EAVT (e, a, v, tx) Entity lookup – “what do I know about entity E?”
AEVT (a, e, v, tx) Attribute scan – “all entities with attribute A”
AVET (a, v, e, tx) Value lookup – “who has name = ‘Alice’?”
VAET (v, a, e, tx) Reverse ref – “who points to entity V?” (ref tables only)

This is 4 indexes per table x 9 tables = 36 indexes total. The VAET index is only useful on datoms_ref_new but is maintained on all tables for uniformity.

Supporting Tables

Table Purpose
mentat.schema Attribute definitions (entid, ident, value_type, cardinality, etc.)
mentat.idents Keyword-to-entid bidirectional mapping
mentat.partitions Entity ID allocation ranges per partition
mentat.transactions Transaction log (tx ID, timestamp)
mentat.cache_generation Cross-backend schema cache invalidation counter

Query Compilation Pipeline

A Datalog query flows through four stages:

EDN text --> Parse --> Compile --> SQL text --> SPI Execute --> JSON result

Stage 1: EDN Parse

The edn crate (workspace member) parses the EDN query string into a ParsedQuery struct containing: - find – find specification (relation, collection, tuple, scalar) - where_clauses – pattern matches, predicates, function calls, not/or clauses - in_bindings – input parameter bindings (scalar, collection, tuple, relation) - with – additional grouping variables - rules – named rule definitions

Stage 2: Datalog Compile

The query compiler (pg_mentat/src/functions/query.rs) transforms the parsed query into SQL:

  1. Schema resolution – resolve keyword attributes (:person/name) to entid integers via the schema cache
  2. Variable binding – assign SQL aliases to each pattern variable (?e -> t0.e)
  3. Table selection – route each pattern to the correct narrow table based on the attribute’s value type
  4. Join generation – unify shared variables across patterns via JOIN ... ON t0.e = t1.e
  5. Predicate translation – convert Datalog predicates to SQL WHERE conditions with type guards
  6. NOT/OR compilation – not becomes NOT EXISTS (subquery), or becomes UNION
  7. Aggregate wrapping – wrap the base query in GROUP BY with aggregate functions
  8. Find spec projection – shape the output (array of arrays, single value, etc.)

All string constants use parameterized bindings ($1, $2, …) via the SqlBuilder to prevent SQL injection.

Stage 3: SQL Execution

The generated SQL is executed through PostgreSQL’s SPI (Server Programming Interface). Results are collected into a serde_json::Value and returned as JSONB.

Stage 4: Result Shaping

The JSON result is shaped according to the find specification: - Relation [:find ?x ?y] – array of arrays: [[1, "Alice"], [2, "Bob"]] - Collection [:find [?x ...]] – flat array: [1, 2, 3] - Tuple [:find [?x ?y]] – single array: [1, "Alice"] - Scalar [:find ?x .] – single value: 1

Transaction Pipeline

Transaction processing (pg_mentat/src/functions/transact.rs) follows these steps:

EDN tx data --> Parse --> Schema detect --> Tempid allocate -->
  Type route --> Upsert resolve --> Insert/Retract --> Commit

Steps in Detail

  1. Parse – EDN transaction data is parsed into assertion/retraction operations (:db/add, :db/retract, map forms, :db/retractEntity, :db.fn/cas)

  2. Schema detection – operations targeting schema attributes (:db/ident, :db/valueType, etc.) are identified; these will update the schema table and invalidate caches

  3. Transaction ID allocation – a new transaction ID is obtained from the mentat.transactions sequence

  4. Tempid resolution – string tempids ("tempid-1") are mapped to freshly allocated entity IDs from the partition sequence

  5. Lookup ref resolution – lookup refs like [:person/email "alice@example.com"] are resolved to entity IDs

  6. Upsert handling – for attributes with :db/unique :db.unique/identity, existing entities are found and reused rather than creating duplicates

  7. Type routing – each datom is inserted into the appropriate narrow table based on the attribute’s value type

  8. CAS validation – :db.fn/cas operations verify the current value matches the expected value before asserting the new value

  9. Commit – the transaction record is written to mentat.transactions; if schema changed, the cache generation counter is bumped

Speculative Transactions

mentat_with() wraps the transaction in a savepoint and rolls back after computing the result. This lets you see what a transaction would do without persisting it.

Pull Architecture

The pull API (pg_mentat/src/functions/pull.rs) implements Datomic’s declarative entity retrieval:

Pull pattern --> Parse --> Recursive fetch --> Cycle detection --> JSON assembly

Key implementation details:

  • Recursive navigation – forward refs (:person/friends) and reverse refs (:person/_friends) are followed to arbitrary depth
  • Cycle detection – a visited-entity set prevents infinite loops in cyclic reference graphs
  • Bounded recursion – {:person/friends 3} limits traversal depth
  • Component auto-expansion – attributes marked :db/isComponent true are automatically pulled recursively
  • Wildcard – [*] retrieves all attributes for an entity
  • Default values – (default :person/age 0) provides fallbacks for missing attributes
  • Limits – (limit :person/email 5) caps cardinality-many results

Schema Cache

pg_mentat maintains a process-local LRU schema cache (SchemaCache) per store that maps attribute keywords to their metadata (entid, value type, cardinality, etc.). This avoids repeated schema table lookups during query compilation.

Cache invalidation uses a generation counter stored in mentat.cache_generation: - After a schema-affecting transaction, the counter is bumped - Before query compilation, the local cache checks its generation against the table - If stale, the cache is reloaded from mentat.schema

This ensures cross-backend consistency when multiple PostgreSQL backends modify schema concurrently.

Multi-Store Architecture

pg_mentat supports multiple isolated stores within a single PostgreSQL database. Each store gets its own PostgreSQL schema (e.g., mentat_mystore) containing independent copies of all tables and indexes. The default store uses the mentat schema.

SELECT mentat.create_store('analytics');
SELECT mentat.t('analytics', '[{:db/ident :event/type ...}]');
SELECT mentat.q('analytics', '[:find ?e :where [?e :event/type "click"]]', '{}');