Contents

• pg-clickhouse-encode.h
  • Type mapping
  • Create writer
  • Append PostgreSQL Datums
  • NULL arrays
  • Recover from row errors
  • Append arrays and tuples manually
  • Inspect and write buffered rows

pg-clickhouse-encode.h

Encode PostgreSQL values into ClickHouse Native blocks. See setup and header.

Define PGCH_IMPLEMENTATION in exactly one translation unit. Supply your own transport and query execution.

Type mapping

These defaults come from pgch_ch_type_for for non-nullable columns. Types without a dedicated mapping use String. Domains use their base type and type modifier. Arrays wrap the element mapping in Array, with nullable elements; the array itself cannot be nullable. Nullable scalar columns use Nullable where ClickHouse supports it.

PostgreSQL	Default ClickHouse	Notes
boolean	Bool
smallint	Int16
integer	Int32
bigint	Int64
oid	UInt32
xid8	UInt64
oid8	UInt64
real	Float32	BFloat16 drops low mantissa bits
double precision	Float64
numeric	Decimal256(38)	numeric_as_string selects String
numeric(12,6)	Decimal(12,6)	Precision selects Decimal width
text	String	low_cardinality selects LowCardinality(String)
bytea	String	Writes raw bytes
date	Date32
time	Time64(6)
timestamp	DateTime64(6, ‘UTC’)
timestamptz	DateTime64(6, ‘UTC’)
interval	String	Interval destinations require whole unit counts
uuid	UUID
json	String	json_as_json selects JSON
jsonb	String	json_as_json selects JSON
inet	String	Override with IPv4 or IPv6 matching address family
point	Point
lseg	LineString	Two points
path	LineString	Closed paths repeat the first point
polygon	Ring
box	Tuple(high Point, low Point)
circle	Tuple(center Point, radius Float64)
line	Tuple(a Float64, b Float64, c Float64)

Read conversions follow separate rules. Reading and writing may not preserve original types or values.

Explicit destination	Conversion
Integer	Accept compatible integer widths
String	Accept source type’s text representation
Interval	Interpret integers as destination unit counts
Other	Accept PostgreSQL explicit casts to destination’s PostgreSQL mapping

See Append PostgreSQL Datums for structural conversions and value constraints.

Create writer

Parse same ClickHouse declarations used by destination:

chc_type *type;
chc_err err = {};

if (chc_type_parse("Array(Int32)", 12, &pgch_alloc, &type, &err) != CHC_OK)
    pgch_raise(&err, ERRCODE_INVALID_PARAMETER_VALUE, NULL, "column \"items\"");

pgch_col col = {
    .name = "items",
    .name_len = 5,
    .type = type,
};

pgch_writer *writer = pgch_writer_new(CurrentMemoryContext, &col, 1);

Writer copies column names and borrows chc_type pointers. Keep types alive until pgch_writer_free. Writer owns a child memory context under parent; freeing writer or deleting parent releases writer state.

pgch_writer_new raises ERRCODE_FDW_INVALID_DATA_TYPE when any ClickHouse type cannot be encoded. Supported composite output is Array, Tuple, Map, Nested, and the geometric types built over them; Dynamic, Variant, and unsupported LowCardinality forms are rejected. SimpleAggregateFunction(f, T) writes as T.

Append PostgreSQL Datums

Call pgch_append_datum once for every column in every row. Column order within row does not matter. Pass source type OID as valtype; writer handles conversion.

For PostgreSQL arrays, pass actual array OID. To pass pgch_array, use ANYARRAYOID. Missing conversion raises ERRCODE_DATATYPE_MISMATCH.

Pass a Tuple as a composite or an array of fields. Pass a Map as an array of composites or a two-dimensional array of key/value pairs, and Nested the same way with one element per row. Composite attributes fill Tuple fields in order, skipping dropped attributes. Text elements are parsed as destination field types.

bytea values map to ClickHouse String and FixedString without text conversion. json and jsonb map to JSON, Object, or String. FixedString pads short values with NUL and rejects longer values. Enum requires a declared label. numeric scales to destination Decimal and rejects overflow. inet family must match IPv4 or IPv6.

interval conversion rejects precision loss, such as 18 months written as IntervalYear. Float32 and Float64 accept NaN and Infinity; Decimal rejects them.

NULL requires a nullable destination, except for arrays handled by policy below.

pgch_append_slot appends one row from slot. It skips dropped and generated attributes, matching pgch_structure_from_tupdesc. Build writer from same descriptor and options to preserve column order.

NULL arrays

Default PGCH_NULL_ARRAY_ERROR rejects NULL PostgreSQL arrays because ClickHouse cannot represent a nullable array. Call pgch_writer_set_null_array(w, PGCH_NULL_ARRAY_EMPTY) to write an empty array instead.

Recover from row errors

Zero-initialize pgch_checkpoint. Call pgch_writer_checkpoint before appending each row. Saving again replaces previous position. Successful appends need no further checkpoint calls.

If append fails, call pgch_writer_rollback to discard everything written since checkpoint.

Save checkpoints only at row boundaries, with no array or tuple open. Resetting writer or rolling back invalidates all saved checkpoints. Call pgch_checkpoint_free when checkpoint is no longer needed.

Append arrays and tuples manually

Open array, append one value per element, then close array:

pgch_array_begin(writer, col);
for (size_t i = 0; i < count; i++)
    pgch_append_datum(writer, 0, values[i], INT8OID, nulls[i]);
pgch_array_end(writer);

Open tuple, append one value per field left to right, then close tuple. pgch_tuple_end requires a value for every field:

pgch_tuple_begin(writer, col);
pgch_append_datum(writer, 0, name, TEXTOID, false);
pgch_append_datum(writer, 0, Int64GetDatum(count), INT8OID, false);
pgch_tuple_end(writer);

Nest these calls as needed. Write Map(K, V) as Array(Tuple(K, V)) and Nested(fields) as Array(Tuple(fields)).

While either context is active, pgch_append_datum ignores col and targets current array element or tuple field.

Close every context before building block.

Inspect and write buffered rows

pgch_writer_rows returns first column row count. pgch_writer_bytes returns buffered bytes across all columns. Use these to choose block boundary.

pgch_writer_flush appends serialized block to pgch_buf, then resets writer after a successful write. Pass NULL options to use pgch_block_opts_local.

pgch_buf out = {};

pgch_writer_flush(writer, &out, NULL);
send_native(out.data, out.len);
pgch_buf_reset(&out);

For another output destination, pass pgch_writer_build to chc_block_write. Built block borrows writer buffers; finish writing before calling pgch_writer_reset. Reset empties buffers and keeps storage for reuse.