Core Architecture & Design
A deep dive into how the engine is structured and how data flows. This section walks the layered package architecture, then traces the request/data flow end to end, naming real packages, types, and functions.
The CRDT core (pkg/sync)
The heart of the engine is DeltaCRDTEngine (crdt.go:140). Its load-bearing fields:
shards []shardRoot+routeSeed maphash.Seed— the sharded root CAS, default 256 shards; eachshardRoot{ptr atomic.Pointer[HAMT]}is a per-shard CAS locus;routeShard(entityID)hashes the entity ID to pick the shard. This eliminates the single-root CAS bottleneck.lamportCounter atomic.Uint64+lastSavedCounter atomic.Uint64(Cache Line 1) — Lamport dot minting + monotone persistence watermark.arena *HamtArena,ebr *EBRManager(Cache Line 2, read-only after init) — off-heap allocation + safe memory reclamation.persistCh chan uint64(unbuffered) +persistWorkerWg+persistStopOnce+persistWorkerReady/persistWorkerParked(cap-1) — a decoupled persist worker; the onlysync.Mutex(persistMu) guards disk fsync in this worker, not the hot CAS path.deltaPool sync.Pool— zero-GC delta recycling.observedInboundRateBits atomic.Uint64— IEEE-754 bits of an EWMA inbound rate feeding the Lamport-skew bound.
The lattice element: CRDTEntry
CRDTEntry (hamt.go:35) is exactly 120 bytes, 8-byte aligned, zero padding (pinned by TestCRDTEntry_SizeAndAlignment). It encodes the full tri-temporal + spatial + causal-dot tuple:
PayloadDigest [32]byte @0 (SHA-256 of payload)
OriginNodeID [16]byte @32 (who originated the dot)
DotNodeID [16]byte @48 (who minted the dot)
DotCounter uint64 @64 (Lamport counter — the causal dot)
SystemTime int64 @72 (when the system observed the fact)
ValidTimeStart int64 @80 (when the fact became true)
ValidTimeEnd int64 @88 (when the fact ceased to be true)
AssertionTime int64 @96 (causal/assertion epoch)
DecisionTime int64 @104
H3Index uint64 @112 (spatial index)CausalDot{NodeID [16]byte; Counter uint64} (hamt.go:25) is the unique mutation-event id. CRDTDelta{OriginNodeID [16]byte; Entries Seq} (crdt.go:1515) carries a push-based iterator Seq func(yield func(entityID string, entry CRDTEntry) bool) — zero-alloc, no slice materialization — and (*CRDTDelta).Release() returns it to deltaPool.
The HAMT and the HamtArena
The HAMT (hamt.go:121) is a persistent, immutable, path-copying trie (Add-Wins Set keyed by entity ID); Set/Delete return a new root with path-copied nodes and structural sharing. Interior HamtNode carries refCount atomic.Int32, bitmap uint32, childrenPtr/entriesPtr NodePtr, a merkleHash [32]byte field, and nextFree NodePtr. That merkleHash is declared as a cached branch hash but is not wired into the root path — it is only ever zeroed, never read to serve a root. MerkleRoot() (hamt.go:273) therefore recomputes the root on every call: it walks every entry via ForEach (an O(N) pass over every dot), collects each (DotNodeID, DotCounter) pair, sorts them (O(N log N)), then SHA-256-hashes the concatenation. The 32-byte hamtLeaf rebuilds Go views via unsafe.Slice/unsafe.String over the arena.
The HamtArena (hamt_arena.go:168) is the mmap'd (MAP_ANON|MAP_PRIVATE) off-heap allocator: 17 size classes (1 node class + 16 var classes), nodeFreelist [256] and varFreelist [16][256] sharded Treiber free-lists (the three hottest CAS sites are sharded 256-way to defeat CAS-storms), four CacheLinePad-isolated route counters, and a bumpOffset atomic.Uint64. NodePtr is a uintptr — GC-invisible, the foundation of true zero-GC.
EBR + hazard pointers
EBR + hazard pointers live in reclamation.go: EBRManager.globalEpoch (CAS'd by AdvanceEpoch, on its own cache line), a 3-epoch × 256-shard retired ring retired [3][256]RetiredList with a 2-epoch grace window, and Participant.Enter/DetachAndProtect(slot,ptr) for hazard-pointer publication. Retired nodes reclaim only after the grace window proves zero lingering readers, making the slab free-lists ABA-immune (proven by TestHEBRDetachAllowsEpochAdvance).
The wire-integrity seam
The wire-integrity seam (crdt_reconstruct.go, crdt_reconstruct_skew.go) is the gate every inbound element crosses before Join: ReconstructEntry cross-validates PayloadDigest == SHA-256(payload) and reads all 12 contract fields off the capnp frame (returning typed WireIntegrityError with Kind ∈ WireIntegrityDigestMismatch/FieldUnread/DotOriginMismatch/LamportSkewPoisoning); it also adds the DotNodeID != OriginNodeID attribution check; ReconstructEntryWithSkewBound rejects far-future dots with saturation arithmetic, closing the far-future-dot and disk-state-poisoning attacks. ReconstructedEntry is returned by value to kill a per-element heap allocation.
Set reconciliation
Set reconciliation uses IBLT (iblt.go: XOR-accumulator buckets, 0.00% false-positive purity) and StrataEstimator (32 fixed IBLTs, 80 buckets, k=3) to compute the symmetric-difference size that drives DynamicIBLTSize(dEst) = dEst * ibltSafetyFactor floored at minDynamicBuckets=128. iblt_wire.go is a deliberate little-endian (non-capnp) codec to keep IBLT off the frozen capnp schema surface.
Storage & durability (internal/database + pkg/durability)
The durable tier accepts TriTemporalEvents (entity × system_time × valid_time × assertion_time + H3 + payload), buffers them in a jemalloc-backed pointerless SkipListArena, asynchronously flushes frozen arenas to per-entity Arrow IPC files, runs background L0→L1 compaction with tri-temporal dominance pruning, and answers bitemporal queries.
JemallocAllocator(memory_allocator.go): CGOmallocx(size, MALLOCX_ALIGN(64)|MALLOCX_ZERO)→ 64-byte cache-line aligned + zeroed; sized free viasdallocx;bytesAllocated atomic.Int64enforces the MemTable 256 MB ceiling. Implementsarrow.memory.Allocator.SkipListArena(skiplist_arena.go): pointerless, array-backed, lock-free.nodeSize=64(cache-line aligned),maxHeight=11,pValue=4(geometric 1/4 height prob),keySize=40. Concurrency viacasNodeNexton next pointers.Seek= Put descent without splice; proven correct but dormant (not wired intoscanWindowRecordBatch).- 40-byte composite key (
keySize=40):[hash16 | sysTime8 | validTimeStart8 | assertTime8]all BigEndian, sobytes.Compare== lexicographic == numeric.hash16= truncated SHA-256 of EntityID (128-bit). MemTable(memtable.go):mu sync.RWMutexguards the SkipListArena pointer swap (freeze→replace);flushSem chan struct{}(cap 4) bounds frozen arenas in flight (backpressure); async double-buffered flush — a full arena is frozen (immutable), a fresh one swapped in under the write lock, then the frozen one streams to L0 off the write path.L0Flusher(l0_flusher.go): serializes a frozen SkipListArena → per-entity Arrow IPC → S3. TheArrowSchemais 9 IPC fields:entity_id_hash(FixedSizeBinary 16), fourTimestamp ns UTCfields,h3_index(Uint64),payload_digest(FixedSizeBinary 32),entity_id(LargeBinary),payload(Binary).MaxValidTimeEndNs int64 = 9_000_000_000_000_000_000(9e18 ns ≈ year 2253) is the open-ended valid-time sentinel.L1Compactor(l1_compactor.go): per-entity L0→L1 merge +DominancePrune+ T_gc auto-inference.CompactionConfigcarriesL0FilesPerEntityTrigger(def 64),MaxL1FilesPerEntity(def 4),EnableDominancePruning(def false, opt-in),PruningHorizonInt64Ns(T_gc floor),PruneBackoffInt64Ns(the observed-frontier backoff).Resolver(query.go): bitemporalAsOf/Range+LiveSourcelive-merge.ResolverConfigcarriesLiveSource(nil = durable-only, byte-identical to the pre-live-source behavior),MaxL0Files(def 1000),MaxRangeRows(def 4096; 0 = UNLIMITED, deliberately not coerced up),EnableFirstSysSkip(def true; gates both file and manifest skips).L0Reaper(l0_reaper.go): cross-entity superseded-L0 disk-reclaim sweep, opt-in (--compaction-reap-enable, default false). Stages A–F verify each manifest'sl1Keyis still present viaDownloadbefore deleting any L0; any download failure ⇒ preserve.EpochCompactor(compactor.go): DEAD — zero production importers; retained only for scope hygiene. The real GC path is DominancePrune + L0 reaper, not tombstones.
pkg/durability is the crash-recovery surface:
Bridge(bridge.go): the write-through seam —PutLocaldoesengine.InsertLocal+ WAL fsync + (optional) checkpoint; the single chokepoint through which a locally-originated mutation enters both the in-memory HAMT and the fsync-per-mutation WAL. ACK-before-durability: a zeroCausalDotfromPutLocal(WAL fsync failed) → the control port returns 503, not a lying 200+zero-dot.OpenWAL/ReplayWAL(wal.go): a pure alias layer re-exportinginternal/chaosWAL with zero own logic — so the production WAL surface is only as proven as the chaos harness.RecoverEngine/RecoverEngineWithSnapshot(recovery.go): full WAL replay or bounded snapshot+tail.SnapshotStore.SnapshotExistsis the cheap probe gating the bounded branch;RecoveryWitnessreports which branch ran and why.SnapshotImage(snapshot.go): a dot-bearing snapshot image (snapshotMagic="SNSP", 120-byte big-endianCRDTEntrywire per record) + Arrow index → O(post-checkpoint) recovery instead of O(writes-since-boot).LocalFS(localfs.go): local-FS shim implementing the four S3 interfaces, used by tests and single-node deployments.
Mesh & replication (pkg/mesh + pkg/clock + pkg/admission)
pkg/mesh is the production peer-to-peer gossip layer over the TLS 1.3 transport. The Gossiper (gossip.go) owns the PeerSet, a payloadCache, the engine, the identity Directory, and ~10 setter-driven seams (SetBridge, SetBatchSize, SetStratifiedAntiEntropy, SetStratifiedFallbackReporter, SetDigestWaitTimeout, SetRoundReporter). AntiEntropySweep sorts peer IDs deterministically and, per peer, generates a sweep delta and ships it (batched via ShipBatch/shipBatchedDelta, or per-frame via shipDelta), then delta.Release()s.
pkg/clock's IngressHLCScalarCap (admission.go) is the first ingress gate — a Byzantine HLC physical-bound cap that drops future-skewed frames (incomingPhysicalUSec - localPhysicalUSec > 2000 us) in sub-µs before the ~60 µs Ed25519 verify. On accept it calls engine.AdvanceLamportTo(incomingLogical) unconditionally (the engine's max-CAS no-ops stale logicals). maxDriftEpsilon=2000 is unit-locked by a static guard plus compile-time array guards.
pkg/admission's PeerBucket (ewma.go) is the per-peer Sybil-burst rate-isolation token bucket — 16 shards each sync.Mutex + map[PeerBucketKey]*PeerEWMA, sharded by the low 4 bits of the first byte of the 32-byte Ed25519 pubkey. An attacker saturates only its own shard (TestPeerBucket_SybilIsolation). The EWMA (alpha=0.1) advances on Counter-delta (per-frame advancement, not absolute), so a fresh peer is never penalized.
pkg/mesh's ControlServer (control.go) exposes the JSON-over-mTLS control port: /v1/insert|get|query|range|merkle, /livecheck, /metrics. It honestly returns 503 (not 404) when the resolver is nil, and the 503 guard runs before param validation. handleGet uses a single-snapshot discipline (one State().Get, one selectLatestDot, one PayloadForDot) so payload and digest derive from the same entry (no TOCTOU).
Receive & transport (pkg/receive + pkg/transport)
pkg/receive's Receiver is the gate-stack composer and the first production caller of Join. The ingress pipeline is:
[wire]
→ FrameReader.ReadFrame (length-prefix reassembly, [uint32 frameLen BE][envelope])
→ 3-way dispatch (batch "SBAT" / digest "SDST" / relay 0x02|0x03)
→ RelayEnvelope.Open: readLastHop (O(1) raw-byte read) → readGateFields (v3 O(1) header mirror)
→ PeerBucket.Accept (3.1 rate, ~36 ns)
→ IngressHLCScalarCap.Admit (3.0 clock, calls engine.AdvanceLamportTo on accept)
→ RelayEnvelope.Open(maxHops) (3.2 depth O(1), then N outer Ed25519 Verifies)
→ Directory.Lookup (origin→pubkey)
→ identity.VerifyCRDTFrame (~60 µs inner origin verify)
→ crossCheckGateFields (the accept-path security guard)
→ engine.ApplyCRDTDeltaEvent / ApplyCRDTDeltaBatch (Join)Cheap gates run before expensive verify — a forged deep/rate/clock frame drops in nanoseconds with zero Ed25519 Verifies (the VerifyHookCount==0 tests). HandleBatchFrame amortizes one Ed25519 over N deltas (60.19 µs → 60.19/N µs/delta) and decrements the rate bucket once per batch on the origin's monotonic OriginSeq.
pkg/transport owns the egress zero-copy boundary (TransmitHeapBuffer: make([]byte,len) → copy(heap,mmap) → Pin(&heap[0])) — it never pins the mmap region (Pin on a non-Go address is a documented silent no-op, empirically proven by TestTransmitHeapBuffer_MmapPinIsNoOp_Physics), and a source guard (detectForbiddenPin) textually bans any Pin not on a make([]byte) var. TLSConnections (tls_transport.go) is 1.3-only mTLS (Min==Max==VersionTLS13, RequireAndVerifyClientCert) with SIGHUP leaf rotation. KernelFanout (ebpf_reuseport.go, //go:build ebpf_kernel) loads a live BPF_PROG_TYPE_SK_REUSEPORT program keyed on OriginNodeID via BPF_MAP_TYPE_SOCKHASH (cilium/ebpf v0.22.0, the first production-loadable eBPF dep). Route selection keys only on plaintext OriginNodeID, before any crypto (TestNoCryptoBeforeRoute).
Request/data flow (end to end)
Write / egress path (locally-originated mutation)
sdk.Client.InsertLocal(key,val) ──HTTPS/mTLS──▶ ControlServer.handleInsert
→ Gossiper.InsertLocalEvents (gossip.go:1214; NEVER engine.InsertLocal directly)
→ bridge.PutLocal(entityID, payload, entry) (bridge.go)
├─ sha256(payload) stamped into entry.PayloadDigest BEFORE engine folds it (an ordering guard)
├─ engine.InsertLocal → NextDot() → sharded root CAS → HAMT.Set (0 allocs/op) → EBR retire
└─ WAL.AppendMutations + f.Sync() (one fsync per batch; ACK only after durable)
→ payloadCache.record
→ (tick) Bridge.AppendCheckpoint → dot-bearing snapshot image + l0/{hex8}/{sysNs}.arrow
→ MemTable.Write → async double-buffered flush → L0Flusher.FlushArenaToIPC → Arrow IPC in S3/LocalFS
→ (scheduler) L1Compactor.Compaction → DominancePrune (opt-in) → L0Reaper.Reap (opt-in)Replication / ingress path (peer-originated delta)
peer TLS conn → FrameReader.ReadFrame → 3-way dispatch
→ PeerBucket.Accept → IngressHLCScalarCap.Admit → RelayEnvelope.Open(maxHops)
→ Directory.Lookup → VerifyCRDTFrame → crossCheckGateFields
→ engine.ApplyCRDTDeltaEvent / ApplyCRDTDeltaBatch
→ ReconstructEntry[WithSkewBound] (wire-integrity + skew bound)
→ Join (frozen merge-union per-shard CAS; crdt.go:1167)
→ HAMT path-copying Set → EBR retire retired nodes through 256-way sharded free-lists
→ onClockAdvance fires only when post > preAdvance → WAL.AppendClockAdvance (kills the fsync bomb on stale re-receive)Read path (/v1/query or /v1/range)
ControlServer.handleQuery / handleRange (503 if resolver nil, BEFORE param validation)
→ Resolver.AsOf(ctx, entity, validTime, txTime) / Range(ctx, entity, vLo, vHi, tx)
├─ QueryTxTimeFrontier = monotone atomic MAX of observed txTime (feeds T_gc inference)
├─ LiveSource.LiveRead(ctx, entityID, txTimeNs) (read-your-writes; EBR-pinned live HAMT)
│ → engineHAMTAdapter: ebr.Acquire → participant.Enter → Filter2 (SystemTime<=txTime) → defer Release
├─ durable tier: list L0+L1 per entity, skip files/manifests whose firstSys STRICTLY > txTime
├─ Filter1 (full-16-byte hash) · Filter2 (SystemTime<=txTime STRICT) · Filter3 (half-open valid-time) · Filter4 (entity-id collision guard)
└─ live vs durable dedup by (sysTime, digest); nil/empty live is NOT an error
→ response echoes PayloadDigest verbatim; deliberately NO payload field (digest-is-not-value; Law V)Gossip / anti-entropy (SweepLoop)
AntiEntropySweep (sorts peerIDs deterministically + topology.Select(ctx) region-aware fan-out)
per peer:
├─ oversend: GenerateDelta(empty IBLT) — one-round CRDT-idempotent convergence, pays N*entries verify
├─ batched: BuildCRDTDeltaBatch — one Ed25519 over N deltas (amortizes 60.19 µs → 60.19/N)
└─ stratified anti-entropy: register a receive channel → send StrataEstimator → block on recv (digestWaitTimeout)
→ GenerateDelta(remoteIBLT) for minimal delta ∝ |A−B| (the wire-exchanged remote IBLT digest); fallback to oversend on timeout
[the earlier GenerateDeltaStratified variant was removed — it subtracted an empty
remote IBLT it never populated, so its delta was byte-identical to oversend: no bandwidth cut]
shipBatchedDelta / shipDelta → peers.Publish (TLS) → delta.Release() (EBR epoch pin drop)
→ selectLatestDot total order: max DotCounter, ties → smallest DotNodeID (bytes.Compare) — deterministic across iteration order
→ stampConvergence (sweep 1 advances prevRoot, sweep 2 stamps convergence, sweep 3 does NOT re-stamp)Clock attribution
NextDot mints {NodeID; Counter} via a monotone CAS on lamportCounter; AdvanceLamportTo(remoteCounter) adopts a remote counter (monotone-CAS-shaped). On recovery, replay restores each mutation at its recorded dot — it never re-mints — and a foreign clock advance is itself recorded in the log (WALRecClockAdvance), so the recovered clock is restored rather than recomputed. The post-snapshot tail is selected by the log's record sequence against the snapshot's recorded cut, never by scalar clock arithmetic. This is what makes replay deterministic and the recovered Merkle root byte-equal to the live one.