Multi-Region Mesh & Anti-Entropy

The engine replicates state with a peer-to-peer TLS 1.3 gossip layer that converges a δ-CRDT across regions with no central sequencer. This page covers how the mesh is shaped, how peers authenticate before they trust one another, how the anti-entropy exchange minimizes bandwidth, and the measured convergence evidence — with an honest note on where the tail is still being hardened.

Region-aware gossip

By default the anti-entropy sweep is a full mesh: every node exchanges with every peer on each round. That is correct but quadratic — at 100 nodes it is roughly ten thousand connections per sweep. An opt-in region-aware mode reshapes the fan-out:

• Intra-region stays a full mesh, so convergence inside a region is one round.
• Inter-region uses a configurable fan-out (default 3) to distinct regions, chosen deterministically so coverage spreads every round instead of clustering.

This replaces cross-region full-mesh with a bounded fan-out — on the order of 35 peer-edges per round instead of ~9,900 at 100 nodes — while the merge stays lossless either way. Region-aware selection is opt-in and off by default; with it off, behavior is byte-identical to the plain full-mesh sweep.

Authenticated peer provisioning

A node must be able to verify a peer's identity before it trusts that peer's data. A pre-provisioned peer directory (--peer-dir) gives each node its peers' public keys up front, so a node authenticates a peer before the first connection rather than learning identities over a channel it has not yet verified. An optional auto-reconcile mode re-keys peers as they join or rotate, and a reconnect loop re-dials a peer that is not yet listening with bounded backoff.

Bandwidth-minimal anti-entropy

To converge, a node does not ship its full state to every peer. It exchanges a compact IBLT digest with a peer, subtracts the two digests, and peels the symmetric difference — so it ships only the entries the peer is missing. Bandwidth scales with |A−B|, not |A|+|B|. An earlier approach was removed because it computed the delta against an empty digest and so shipped everything; the corrected exchange reads the peer's real digest off the wire. The digest is fixed-size, so past a disclosed number of entries per node it saturates and the sweep falls back to a full send — convergence still holds; the bound is disclosed, not hidden.

The convergence evidence

The multi-region claim is measured, not asserted:

• 100 node processes on three machines, one per region — a 10,000-key write converges in 8.95 seconds on the published run, against a 10-second target; the decision log records seven consecutive runs before it, six between 6.6 s and 8.7 s and one miss at 10.15 s, with integrity verification armed and production checkpointing on. The fleet is one c7gd.8xlarge per region (AWS Graviton3, 32 vCPU, 64 GiB, 1×1900 GB local NVMe) in us-east-1, eu-west-1, and ap-southeast-2, carrying 34, 33 and 33 node processes.
• Crash recovery — a node killed mid-run recovers from its snapshot and its Merkle root re-equals the fleet.
• Memory-safe measurement — convergence is read with a sharded Merkle root (computed per shard) so that measuring a 100-node mesh does not exhaust memory. The earlier whole-state read did, and was replaced.

The write-ahead log's group commit — one fsync per batch instead of one per entry — is what made the sub-10-second convergence reachable at production durability. The background checkpoint is decoupled and delta-batched off the commit path, so durability no longer stalls convergence.

Honest status

• Convergence at 100 node processes on 3 machines: met on the published run. The sub-10-second target held on six of seven logged runs; one missed it by 150 ms.
• The convergence tail: the last one or two nodes occasionally straggle past 10 seconds (about one run in seven). This is a gossip-distribution property, independent of durability, and it is the open hardening target.
• Opt-in surface: region-aware gossip, digest exchange, and peer auto-reconcile are all off by default. The verified runs above opted into --region-aware gossip (inter-region fan-out 3) with --peer-dir provisioning; the digest exchange (--stratified-anti-entropy) and --peer-auto-reconcile stayed off. See Verification & Status for the full evidence ledger.