Core Architecture & Design
A deep dive into how the engine is structured and how data flows. This section walks the layered package architecture, then traces the request/data flow end to end, naming real packages, types, and functions.
TestGate_FrozenMD5; everything else is a seam around it. The write path (insert → mint dot → live HAMT → WAL → checkpoint → L0 → compaction) and the read path (query → live HAMT + durable supremum) both pass through the core, which is why its allocation budget is zero. The CRDT core (pkg/sync)
The heart of the engine is DeltaCRDTEngine (crdt.go:135). Its load-bearing fields:
shards []shardRoot+routeSeed maphash.Seed— Phase 2.5a sharded root CAS, default 256 shards; eachshardRoot{ptr atomic.Pointer[HAMT]}is a per-shard CAS locus;routeShard(entityID)hashes the entity ID to pick the shard. This eliminates the single-root CAS bottleneck.lamportCounter atomic.Uint64+lastSavedCounter atomic.Uint64(Cache Line 1) — Lamport dot minting + monotone persistence watermark.arena *HamtArena,ebr *EBRManager(Cache Line 2, read-only after init) — off-heap allocation + safe memory reclamation.persistCh chan uint64(unbuffered) +persistWorkerWg+persistStopOnce+persistWorkerReady/persistWorkerParked(cap-1) — Phase 2.5c decoupled persist worker; the onlysync.Mutex(persistMu) guards disk fsync in this worker, not the hot CAS path.deltaPool sync.Pool— Phase 2.5b zero-GC delta recycling.observedInboundRateBits atomic.Uint64— IEEE-754 bits of an EWMA inbound rate feeding the Lamport-skew bound.
The lattice element: CRDTEntry
CRDTEntry (hamt.go:29) is exactly 120 bytes, 8-byte aligned, zero padding (pinned by TestCRDTEntry_SizeAndAlignment). It encodes the full tri-temporal + spatial + causal-dot tuple:
PayloadDigest [32]byte @0 (SHA-256 of payload)
OriginNodeID [16]byte @32 (who originated the dot)
DotNodeID [16]byte @48 (who minted the dot)
DotCounter uint64 @64 (Lamport counter — the causal dot)
SystemTime int64 @72 (when the system observed the fact)
ValidTimeStart int64 @80 (when the fact became true)
ValidTimeEnd int64 @88 (when the fact ceased to be true)
AssertionTime int64 @96 (causal/assertion epoch)
DecisionTime int64 @104
H3Index uint64 @112 (spatial index)CausalDot{NodeID [16]byte; Counter uint64} (hamt.go:19) is the unique mutation-event id. CRDTDelta{OriginNodeID [16]byte; Entries Seq} (crdt.go:1407) carries a push-based iterator Seq func(yield func(entityID string, entry CRRTEntry) bool) — zero-alloc, no slice materialization — and (*CRDTDelta).Release() returns it to deltaPool.
The HAMT and the HamtArena
The HAMT (hamt.go:115) is a persistent, immutable, path-copying trie (Add-Wins Set keyed by entity ID); Set/Delete return a new root with path-copied nodes and structural sharing. Interior HamtNode carries refCount atomic.Int32, bitmap uint32, childrenPtr/entriesPtr NodePtr, a cached merkleHash [32]byte (→ O(1) MerkleRoot()), and nextFree NodePtr. The 32-byte hamtLeaf rebuilds Go views via unsafe.Slice/unsafe.String over the arena.
The HamtArena (hamt_arena.go:162) is the mmap'd (MAP_ANON|MAP_PRIVATE) off-heap allocator: 17 size classes (1 node class + 16 var classes), nodeFreelist [256] and varFreelist [16][256] sharded Treiber free-lists (the three hottest CAS sites are sharded 256-way to defeat CAS-storms), four CacheLinePad-isolated route counters, and a bumpOffset atomic.Uint64. NodePtr is a uintptr — GC-invisible, the foundation of true zero-GC.
EBR + hazard pointers
EBR + hazard pointers live in reclamation.go: EBRManager.globalEpoch (CAS'd by AdvanceEpoch, on its own cache line), a 3-epoch × 256-shard retired ring retired [3][256]RetiredList with a 2-epoch grace window, and Participant.Enter/DetachAndProtect(slot,ptr) for hazard-pointer publication. Retired nodes reclaim only after the grace window proves zero lingering readers, making the slab free-lists ABA-immune (proven by TestHEBRDetachAllowsEpochAdvance).
The wire-integrity seam
The wire-integrity seam (crdt_reconstruct.go, crdt_reconstruct_skew.go) is the gate every inbound element crosses before Join: ReconstructEntry cross-validates PayloadDigest == SHA-256(payload) and reads all 12 contract fields off the capnp frame (returning typed WireIntegrityError with Kind ∈ WireIntegrityDigestMismatch/FieldUnread/DotOriginMismatch/LamportSkewPoisoning); Phase 2f adds the DotNodeID != OriginNodeID attribution check; Phase 2g ReconstructEntryWithSkewBound rejects far-future dots with saturation arithmetic, closing the Byzantine A1 far-future-dot and A4 disk-state-poisoning attacks. ReconstructedEntry is returned by value (FIX C, ADR-0014) to kill a per-element heap alloc.
Set reconciliation
Set reconciliation uses IBLT (iblt.go: XOR-accumulator buckets, 0.00% false-positive purity) and StrataEstimator (32 fixed IBLTs, 80 buckets, k=3) to compute the symmetric-difference size that drives DynamicIBLTSize(dEst) = dEst * ibltSafetyFactor floored at minDynamicBuckets=128. iblt_wire.go is a deliberate little-endian (non-capnp) codec to keep IBLT off the FROZEN capnp schema surface.
Storage & durability (internal/database + pkg/durability)
The durable tier accepts TriTemporalEvents (entity × system_time × valid_time × assertion_time + H3 + payload), buffers them in a jemalloc-backed pointerless SkipListArena, asynchronously flushes frozen arenas to per-entity Arrow IPC files, runs background L0→L1 compaction with tri-temporal dominance pruning, and answers bitemporal queries.
JemallocAllocator(memory_allocator.go): CGOmallocx(size, MALLOCX_ALIGN(64)|MALLOCX_ZERO)→ 64-byte cache-line aligned + zeroed; sized free viasdallocx;bytesAllocated atomic.Int64enforces the MemTable 256 MB ceiling. Implementsarrow.memory.Allocator.SkipListArena(skiplist_arena.go): pointerless, array-backed, lock-free.nodeSize=64(cache-line aligned),maxHeight=11,pValue=4(geometric 1/4 height prob),keySize=40. Concurrency viacasNodeNexton next pointers.Seek= Put descent without splice; proven correct (Day 23, ADR-0028) but dormant (not wired intoscanWindowRecordBatch).- 40-byte composite key (
keySize=40):[hash16 | sysTime8 | validTimeStart8 | assertTime8]all BigEndian, sobytes.Compare== lexicographic == numeric.hash16= truncated SHA-256 of EntityID (128-bit). MemTable(memtable.go):mu sync.RWMutexguards the SkipListArena pointer swap (freeze→replace);flushSem chan struct{}(cap 4) bounds frozen arenas in flight (backpressure); async double-buffered flush — a full arena is frozen (immutable), a fresh one swapped in under the write lock, then the frozen one streams to L0 off the write path.L0Flusher(l0_flusher.go): serializes a frozen SkipListArena → per-entity Arrow IPC → S3. TheArrowSchemais 9 IPC fields:entity_id_hash(FixedSizeBinary 16), fourTimestamp ns UTCfields,h3_index(Uint64),payload_digest(FixedSizeBinary 32),entity_id(LargeBinary),payload(Binary).MaxValidTimeEndNs int64 = 9_000_000_000_000_000_000(9e18 ns ≈ year 2253) is the open-ended valid-time sentinel.L1Compactor(l1_compactor.go): per-entity L0→L1 merge +DominancePrune+ T_gc auto-inference.CompactionConfigcarriesL0FilesPerEntityTrigger(def 64),MaxL1FilesPerEntity(def 4),EnableDominancePruning(def false, opt-in),PruningHorizonInt64Ns(T_gc floor),PruneBackoffInt64Ns(Day-22 backoff).Resolver(query.go): bitemporalAsOf/Range+LiveSourcelive-merge.ResolverConfigcarriesLiveSource(nil = durable-only, byte-identical to Day-26),MaxL0Files(def 1000),MaxRangeRows(def 4096; 0 = UNLIMITED, deliberately not coerced up),EnableFirstSysSkip(def true; gates both file and manifest skips).L0Reaper(l0_reaper.go): cross-entity superseded-L0 disk-reclaim sweep, opt-in (--compaction-reap-enable, default false). Stages A–F verify each manifest'sl1Keyis still present viaDownloadbefore deleting any L0; any download failure ⇒ preserve.EpochCompactor(compactor.go): DEAD — zero production importers; retained only for scope hygiene. The real GC path is DominancePrune + L0 reaper, not tombstones.
pkg/durability is the crash-recovery surface:
Bridge(bridge.go): the write-through seam —PutLocaldoesengine.InsertLocal+ WAL fsync + (optional) checkpoint; the single chokepoint through which a locally-originated mutation enters both the in-memory HAMT and the fsync-per-mutation WAL. ACK-before-durability: a zeroCausalDotfromPutLocal(WAL fsync failed) → the control port returns 503, not a lying 200+zero-dot.OpenWAL/ReplayWAL(wal.go): a pure alias layer re-exportinginternal/chaosWAL with zero own logic — so the production WAL surface is only as proven as the chaos harness.RecoverEngine/RecoverEngineWithSnapshot(recovery.go): full WAL replay or bounded snapshot+tail.SnapshotStore.SnapshotExistsis the cheap probe gating the bounded branch;RecoveryWitnessreports which branch ran and why.SnapshotImage(snapshot.go, Day-11): a dot-bearing snapshot image (snapshotMagic="SNSP", 120-byte big-endianCRDTEntrywire per record) + Arrow index → O(post-checkpoint) recovery instead of O(writes-since-boot).LocalFS(localfs.go): local-FS shim implementing the four S3 interfaces, used by tests and single-node deployments.
Mesh & replication (pkg/mesh + pkg/clock + pkg/admission)
pkg/mesh is the production peer-to-peer gossip layer over the Day-1 TLS 1.3 transport. The Gossiper (gossip.go) owns the PeerSet, a payloadCache, the engine, the identity Directory, and ~10 setter-driven seams (SetBridge, SetBatchSize, SetStratifiedAntiEntropy, SetStratifiedFallbackReporter, SetDigestWaitTimeout, SetRoundReporter). AntiEntropySweep sorts peer IDs deterministically and, per peer, generates a sweep delta and ships it (batched via ShipBatch/shipBatchedDelta, or per-frame via shipDelta), then delta.Release()s.
pkg/clock's IngressHLCScalarCap (admission.go) is the first ingress gate — a Byzantine HLC physical-bound cap that drops future-skewed frames (incomingPhysicalUSec - localPhysicalUSec > 2000 us) in sub-µs before the ~71.4 µs Ed25519 verify. On accept it calls engine.AdvanceLamportTo(incomingLogical) unconditionally (the engine's max-CAS no-ops stale logicals). maxDriftEpsilon=2000 is unit-locked by a static tooth plus compile-time array guards.
pkg/admission's PeerBucket (ewma.go) is the per-peer Sybil-burst rate-isolation token bucket — 16 shards each sync.Mutex + map[PeerBucketKey]*PeerEWMA, sharded by the low 4 bits of the first byte of the 32-byte Ed25519 pubkey. An attacker saturates only its own shard (TestPeerBucket_SybilIsolation). The EWMA (alpha=0.1) advances on Counter-delta (per-frame advancement, not absolute), so a fresh peer is never penalized.
pkg/mesh's ControlServer (control.go) exposes the JSON-over-mTLS control port: /v1/insert|get|query|range|merkle, /livecheck, /metrics. It honestly returns 503 (not 404) when the resolver is nil, and the 503 guard runs before param validation. handleGet uses a single-snapshot discipline (one State().Get, one selectLatestDot, one PayloadForDot) so payload and digest derive from the same entry (no TOCTOU).
Receive & transport (pkg/receive + pkg/transport)
pkg/receive's Receiver is the gate-stack composer and the first production caller of Join. The ingress pipeline is:
[wire]
→ FrameReader.ReadFrame (length-prefix reassembly, [uint32 frameLen BE][envelope])
→ 3-way dispatch (batch "SBAT" / digest "SDST" / relay 0x02|0x03)
→ RelayEnvelope.Open: readLastHop (O(1) raw-byte read) → readGateFields (v3 O(1) header mirror)
→ PeerBucket.Accept (3.1 rate, ~36 ns)
→ IngressHLCScalarCap.Admit (3.0 clock, calls engine.AdvanceLamportTo on accept)
→ RelayEnvelope.Open(maxHops) (3.2 depth O(1), then N outer Ed25519 Verifies)
→ Directory.Lookup (GAP-3 origin→pubkey)
→ identity.VerifyCRDTFrame (~60 µs inner origin verify)
→ crossCheckGateFields (§4 accept-path security tooth)
→ engine.ApplyCRDTDeltaEvent / ApplyCRDTDeltaBatch (Join)Cheap gates run before expensive verify — a forged deep/rate/clock frame drops in nanoseconds with zero Ed25519 Verifies (the VerifyHookCount==0 teeth). HandleBatchFrame amortizes one Ed25519 over N deltas (60.19 µs → 60.19/N µs/delta) and decrements the rate bucket once per batch on the origin's monotonic OriginSeq.
pkg/transport owns the egress zero-copy boundary (TransmitHeapBuffer: make([]byte,len) → copy(heap,mmap) → Pin(&heap[0])) — it never pins the mmap region (Pin on a non-Go address is a documented silent no-op, empirically proven by TestTransmitHeapBuffer_MmapPinIsNoOp_Physics), and a source guard (detectForbiddenPin) textually bans any Pin not on a make([]byte) var. TLSConnections (tls_transport.go) is 1.3-only mTLS (Min==Max==VersionTLS13, RequireAndVerifyClientCert) with SIGHUP leaf rotation. KernelFanout (ebpf_reuseport.go, //go:build ebpf_kernel) loads a live BPF_PROG_TYPE_SK_REUSEPORT program keyed on OriginNodeID via BPF_MAP_TYPE_SOCKHASH (cilium/ebpf v0.22.0, the first production-loadable eBPF dep). Route selection keys only on plaintext OriginNodeID, before any crypto (TestNoCryptoBeforeRoute).
Request/data flow (end to end)
Write / egress path (locally-originated mutation)
sdk.Client.InsertLocal(key,val) ──HTTPS/mTLS──▶ ControlServer.handleInsert
→ Gossiper.InsertLocalEvents (gossip.go:202; NEVER engine.InsertLocal directly)
→ bridge.PutLocal(entityID, payload, entry) (bridge.go)
├─ sha256(payload) stamped into entry.PayloadDigest BEFORE engine folds it (order tooth G08.e)
├─ engine.InsertLocal → NextDot() → sharded root CAS → HAMT.Set (0 allocs/op) → EBR retire
└─ WAL.AppendMutation + f.Sync() (fsync-per-mutation; ACK only after durable)
→ payloadCache.record
→ (tick) Bridge.AppendCheckpoint → dot-bearing snapshot image + l0/{hex8}/{sysNs}.arrow
→ MemTable.Write → async double-buffered flush → L0Flusher.FlushArenaToIPC → Arrow IPC in S3/LocalFS
→ (scheduler) L1Compactor.Compaction → DominancePrune (opt-in) → L0Reaper.Reap (opt-in)Replication / ingress path (peer-originated delta)
peer TLS conn → FrameReader.ReadFrame → 3-way dispatch
→ PeerBucket.Accept → IngressHLCScalarCap.Admit → RelayEnvelope.Open(maxHops)
→ Directory.Lookup → VerifyCRDTFrame → crossCheckGateFields
→ engine.ApplyCRDTDeltaEvent / ApplyCRDTDeltaBatch
→ ReconstructEntry[WithSkewBound] (wire-integrity + skew bound)
→ Join (FROZEN merge-union per-shard CAS; crdt.go:1089)
→ HAMT path-copying Set → EBR retire retired nodes through 256-way sharded free-lists
→ (Day-8.5) onClockAdvance fires ONLY when post > preAdvance → WAL.AppendClockAdvance (kills the fsync bomb on stale re-receive)Read path (/v1/query or /v1/range)
ControlServer.handleQuery / handleRange (503 if resolver nil, BEFORE param validation)
→ Resolver.AsOf(ctx, entity, validTime, txTime) / Range(ctx, entity, vLo, vHi, tx)
├─ (Day-22) QueryTxTimeFrontier = monotone atomic MAX of observed txTime (feeds T_gc inference)
├─ LiveSource.LiveRead(ctx, entityID, txTimeNs) (Day-27 read-your-writes; EBR-pinned live HAMT)
│ → engineHAMTAdapter: ebr.Acquire → participant.Enter → Filter2 (SystemTime<=txTime) → defer Release
├─ durable tier: list L0+L1 per entity, (Day-24/25) skip files/manifests whose firstSys STRICTLY > txTime
├─ Filter1 (full-16-byte hash) · Filter2 (SystemTime<=txTime STRICT) · Filter3 (half-open valid-time) · Filter4 (entity-id collision guard)
└─ live vs durable dedup by (sysTime, digest); nil/empty live is NOT an error
→ response echoes PayloadDigest verbatim; deliberately NO payload field (digest-is-not-value; Law V)Gossip / anti-entropy (SweepLoop)
AntiEntropySweep (sorts peerIDs deterministically)
per peer:
├─ Day-2 oversend: GenerateDelta(empty IBLT) — one-round CRDT-idempotent convergence, pays N*entries verify
├─ Day-5 batched: BuildCRCTDeltaBatch — one Ed25519 over N deltas (amortizes 60.19 µs → 60.19/N)
└─ Day-29 stratified: register recv chan → send StrataEstimator → block on recv (digestWaitTimeout)
→ GenerateDeltaStratified(remoteSE) for minimal delta ∝ |A−B|; M5 fallback to oversend on timeout
shipBatchedDelta / shipDelta → peers.Publish (TLS) → delta.Release() (EBR epoch pin drop)
→ selectLatestDot total order: max DotCounter, ties → smallest DotNodeID (bytes.Compare) — deterministic across iteration order
→ stampConvergence (sweep 1 advances prevRoot, sweep 2 stamps convergence, sweep 3 does NOT re-stamp)Clock attribution
NextDot mints {NodeID; Counter} via monotone CAS on lamportCounter; AdvanceLamportTo(remoteCounter) adopts a remote counter (monotone-CAS-shaped). WAL-replay invariant: replay seeds recoveredCounter = LamportHigh - len(Mutations) (NOT LamportHigh), re-running the minting to avoid double-minting. Foreign AdvanceLamportTo is recorded as WALRecClockAdvance=0x03 (Day-8.5) and replayed with the exact seed firstMutation.Counter-1.