pg-clickhouse-decode.h

Decode ClickHouse Native blocks into PostgreSQL Datums. See setup and header.

Define PGCH_IMPLEMENTATION in exactly one translation unit. Define CHC_IMPLEMENTATION in same translation unit when using chunk source API.

Type mapping

Default PostgreSQL types come from pgch_pg_type_for. record and record[] describe anonymous composites, which cannot define table columns. Consumers choose a concrete fallback, such as text[] for Tuple and text[][] for Map and Nested.

Additional read targets list conversions beyond PostgreSQL explicit casts. An empty cell still allows those casts. Values must fit target types. Array elements convert individually; domains also enforce their constraints. Nullable and LowCardinality use inner type conversions.

Write conversions follow separate rules. Reading and writing may not preserve original types or values.

ClickHouse Default PostgreSQL Additional read targets Notes
Array(T) T[] One PG array type per depth
BFloat16 real
Bool boolean
Date date
Date32 date
DateTime timestamp with time zone time
DateTime64(P) timestamp(P) with time zone time P over 6 caps at 6
Decimal(P,S) numeric(P,S) xid8, oid8
Decimal32(S) numeric(9,S) xid8, oid8
Decimal64(S) numeric(18,S) xid8, oid8
Decimal128(S) numeric(38,S) xid8, oid8
Decimal256(S) numeric(76,S) xid8, oid8
Enum8 text bytea; input-compatible types Decodes label; PG enums with matching labels qualify
Enum16 text bytea; input-compatible types Decodes label; PG enums with matching labels qualify
FixedString(N) text bytea; input-compatible types Only bytea keeps trailing NULs
Float32 real
Float64 double precision
IPv4 inet
IPv6 inet
Int8 smallint boolean Zero is false; nonzero is true
Int16 smallint boolean Zero is false; nonzero is true
Int32 integer
Int64 bigint
Int128 numeric(39,0) xid8, oid8
Int256 numeric(77,0) xid8, oid8
IntervalDay interval smallint, integer, bigint Integers receive unit counts
IntervalHour interval smallint, integer, bigint Integers receive unit counts
IntervalMicrosecond interval smallint, integer, bigint Integers receive unit counts
IntervalMillisecond interval smallint, integer, bigint Integers receive unit counts
IntervalMinute interval smallint, integer, bigint Integers receive unit counts
IntervalMonth interval smallint, integer, bigint Integers receive unit counts
IntervalNanosecond interval smallint, integer, bigint Integers keep ns; interval truncates to us
IntervalQuarter interval smallint, integer, bigint Integers receive unit counts
IntervalSecond interval smallint, integer, bigint Integers receive unit counts
IntervalWeek interval smallint, integer, bigint Integers receive unit counts
IntervalYear interval smallint, integer, bigint Integers receive unit counts
JSON jsonb json, text, bytea; input-compatible types jsonb normalizes document
LineString path lseg lseg requires two points
LowCardinality(T) T
Map(K,V) record[] composite[], T[][], text One record per pair
MultiLineString path[]
MultiPolygon polygon[][]
Nested(…) record[] composite[], T[][], text One record per nested row
Nullable(T) T Sets nullable on the column
Point point
Polygon polygon[]
Ring polygon lseg lseg requires two points
SimpleAggregateFunction(f,T) T Stores values as T
String text bytea; input-compatible types bytea keeps raw bytes
Time time without time zone
Time64(P) time(P) without time zone P over 6 caps at 6
Tuple(…) record composite, T[], text; box, circle, line Match field order and types
UInt8 smallint boolean Zero is false; nonzero is true
UInt16 integer
UInt32 bigint
UInt64 numeric(20,0) xid8, oid8
UInt128 numeric(39,0) xid8, oid8
UInt256 numeric(78,0) xid8, oid8
UUID uuid

Input-compatible types parse text through their PostgreSQL input function. This includes PostgreSQL enums whose labels match ClickHouse Enum labels. bytea preserves raw bytes; other targets drop trailing NUL padding. Text targets apply configured encoding policy. JSON arrives as a document string.

For composite targets, match field order and types. T[] spreads tuple fields into array elements when each field converts to T and no field is an array. Map and Nested add an outer array dimension. Nested arrays must be rectangular. A text target renders a PostgreSQL record or array literal. Geometric tuple targets require matching coordinate shapes: two points for box, a point and radius for circle, and three coefficients for line.

Empty rings and lines decode as NULL because PostgreSQL polygons and paths require at least one point.

Supply decoded blocks

typedef struct pgch_block_source {
    void *ud;
    const chc_block *(*next_block)(void *ud);
    const char       *(*error)(void *ud);
} pgch_block_source;

next_block returns one block per call. Returning non-NULL transfers ownership to reader. Reader destroys block with pgch_alloc, so source must read it with same allocator.

Return NULL for end of stream. On failure, return NULL and make error return message. error must be callable before first block, after every next_block, and at end.

Reader runs chc_column_validate on every column of every block, so source need not repeat it. Violation sets reader.error and ends stream.

Handle transport recovery in your source, including discarding connections that cannot be reused after invalid data.

Supply Native byte chunks

typedef struct pgch_chunk_source {
    void *ud;
    bool (*next_chunk)(void *ud, const void **p, size_t *n, char **error);
    bool (*cancelled)(void *ud);
} pgch_chunk_source;

void pgch_reader_init_chunks(pgch_reader *r,
                             const pgch_chunk_source *src,
                             const chc_block_opts *opts);

Use chunk source when transport provides Native bytes instead of decoded blocks. next_chunk returns pointer and length. Keep bytes valid until next next_chunk call.

  • Set *n = 0 and return true for end of stream
  • Set *error and return false for source failure
  • Set cancelled to NULL when cancellation polling is not needed

Ending between blocks is clean end. Ending inside block produces truncation error. Pass NULL options to use pgch_block_opts_local.

Blocks use memory context active when pgch_reader_init_chunks is called. Keep this context alive until reader is finished. Callers can safely use a per-row context while reading rows.

Chunk source API is available when implementation translation unit defines both PGCH_IMPLEMENTATION and CHC_IMPLEMENTATION.

Read rows

Initialize in memory context that should own reader state and error text. Reader itself may be stack allocated.

Initialization reads first block and sets ncols and coltypes. Empty stream, zero columns, unsupported schema, or source failure leaves reader done. Check reader.error after initialization.

Call pgch_reader_next until false. Each successful call fills values and nulls, both ncols long. Consume row before next call. Reader advances across blocks and presents one continuous row stream.

Reader rejects unsupported column types, schema changes that would alter returned Datum shape, and columns failing chc_column_validate. Those failures set reader.error. Clean end leaves it NULL.

Value conversion can still raise PostgreSQL errors, including invalid JSON or numeric text, text outside database encoding, values outside selected target range, and payload that contradicts declared type.

pgch_reader_free releases current block and clears reader.error. Save error pointer first when it must survive that call; storage remains valid until initialization memory context is reset or deleted.

Convert into target PostgreSQL types

See Type mapping for supported targets and conversion rules.

Pass target atttypmod, or -1 when the target carries none. Array type modifiers apply to elements; pass them unchanged. Domains supply their own. The modifier applies as PostgreSQL applies it on assignment:

Target Effect
char(n) Pads to length
varchar(n) Rejects values beyond length
numeric(p,s) Rounds to scale
time(n), timestamp(n) Truncates to precision
domain Own modifier over base conversion, then constraints per element; NULL rows skip conversion, caller enforces NOT NULL

Use pgch_reader_convert_init to prepare each column before reading rows, including columns containing only NULLs. Use pgch_convert_init_type for a standalone ClickHouse type.

Set reader.encoding_check before preparing conversions to control invalid text handling. Default policy raises an error. See pgch_encoding_check in header for alternatives. Use pgch_parse_encoding_check to parse a string into a valid reader.encoding_check value.

All initialization functions allocate state in CurrentMemoryContext. Build state in context that outlives row loop. They return NULL when no conversion is required. pgch_convert accepts NULL state and returns input unchanged. pgch_convert_free releases top-level state; enclosing memory context owns associated allocations.

Fill target row

Use pgch_reader_fill to convert current row into caller arrays. values, nulls, and optional states must each hold r->ncols entries. Pass NULL for states, or use NULL entries for columns requiring no conversion.

pgch_reader_fill_map writes column i to dest[i]. Allocate output arrays for all destination attributes. Unmapped positions stay unchanged; initialize them before filling:

memset(slot->tts_isnull, true, desc->natts * sizeof(bool));
pgch_reader_fill_map(&reader, states, attnums,
                     slot->tts_values, slot->tts_isnull);

For text output, use pgch_value_to_cstring. It handles intermediate arrays and tuples and applies your encoding policy. Use bytea to preserve raw bytes.

Complete reader example

pgch_reader reader;

pgch_reader_init(&reader, &source);
if (reader.error)
    ereport(ERROR, errmsg("%s", reader.error));

void **states = palloc0(reader.ncols * sizeof(*states));
Datum *values = palloc(reader.ncols * sizeof(*values));
bool *nulls = palloc(reader.ncols * sizeof(*nulls));

for (size_t i = 0; i < reader.ncols; i++)
    states[i] = pgch_reader_convert_init(&reader, i,
                                         TupleDescAttr(desc, i)->atttypid,
                                         TupleDescAttr(desc, i)->atttypmod);

while (pgch_reader_next(&reader)) {
    pgch_reader_fill(&reader, states, values, nulls);
    consume_tuple(heap_form_tuple(desc, values, nulls));
}

if (reader.error)
    ereport(ERROR, errmsg("%s", reader.error));

pgch_reader_free(&reader);