Design Rationale & Philosophy
The engine is governed by five physical laws stated in SUPREMUM_STYLE.md and enforced by named test gates. Each law exists because a measured failure mode made it non-optional. This section ties each rationale to the real mechanism that implements it.
The five physical laws
Memory law
0 allocs/op on the hot path. One make() on the write path at the core rate (63M+ ops/s) = stop-the-world ~every 4ms.
Cache law
128-byte stride for every contended atomic. Two atomics on one cache line at 32 cores = a HITM storm = 1.1M ops/s, 1.6% efficiency vs the 63M+ core range.
Lock law
No sync.Mutex on the write path. CAS + EBR only. A futex is a scheduler stall, and a stall at the core rate (63M+ ops/s) is a cliff.
WAL law
Replay restores the recorded dot — it never re-mints. Each mutation is applied at its WAL-recorded (NodeID, Counter); the tail is cut by record sequence, not a scalar clock. Re-minting double-mints, and double-minting is corruption.
Honesty law
Report numbers, not adjectives — and report the layer. “The core microbench gate passes at 50,736,038 ops/s @32 cores; this tree re-measures 68,278,197; production ingest is 5.7–6.0M δ/sec” is a fact. Quoting any peak as “sustained throughput” is the round-up we forbid.
Why δ-CRDTs (vs operational logs)
An operational log conflates what happened with what the system believed. A δ-CRDT separates them: state is a join-semilattice element, and a delta is the minimal state-difference needed to bring a remote replica up to date. The merge operation is a pure function (Join, crdt.go:1167, frozen), so convergence is a lattice-join property — commutative, associative, idempotent — proven by the rapid-based property tests (TestCRDTJoinCommutativity, TestCRDTJoinAssociativity, TestCRDTJoinIdempotence, TestCRDTConvergenceMultiNode, TestCRDTJoinMonotonicGrowth) and proven to survive a lossy/duplicating/reordering/partitioned transport by internal/chaos's TestMerkleConvergenceAfterPartition (32 engines, asymmetric partition, 12 gossip rounds, byte-equal roots after heal) and TestConvergenceDeterminismAcrossRuns (two independent 8-node runs converge to the same root with dedup disabled — Join idempotence alone guarantees correctness).
The δ-prefix is what makes this affordable across a fleet: instead of shipping full state, the engine ships GenerateDelta(remoteDigest) (the set of entries the remote is missing, computed by subtracting the wire-exchanged remote IBLT digest and peeling the real diff — the minimal delta ∝ |A−B|). (An earlier variant computed its delta against an empty remote digest it never populated, so its "minimal delta" was byte-identical to a full send — no bandwidth cut. It was removed.) This is why the mesh can converge 1000 split events in ≤10 rounds over a real TLS 1.3 loopback (TestTwoNodeConvergence_InMemory).
A known, deliberately-not-fixed regression
Join is a merge-union on CausalDot — it drops nothing, ever. An eager last-writer-wins resolver that would drop the losing write was removed precisely because a dropped dot cannot re-merge across a foreign AdvanceLamportTo replay — wiring it into Join would be a convergence regression, not an optimization. The genuine lossless conflict resolution lives in the read path, which reads the full dot set via selectLatestDot (a two-dimensional pick: larger DotCounter, then bytes.Compare(DotNodeID)).
Why off-heap / zero-GC (the memory law)
Memory law: 0 allocs/op on the hot path. Every allocation on the hot path is a future GC pause. One
make()on the write path at the CRDT core rate (50.7M–68.3M ops/s, theHAMT.Setmicrobench) = stop-the-world ~every 4ms. Unacceptable.
The mechanism: NodePtr is a GC-invisible uintptr; all node/leaf/string/entry-array data lives in the mmap'd HamtArena; even the *HAMT wrapper is arena-allocated (allocHAMTWrapper); makeBinaryKey writes into a stack [8]byte then unsafe.String (safe because makeLeaf synchronously copies bytes into the arena before Set returns). The gate is TestHotPathZeroAllocations asserting HAMT.Set = 0 allocs/op. The raceEnabled build-tag pair lets it self-skip under -race (shadow-memory instrumentation inflates AllocsPerRun).
The same philosophy propagates outward: internal/database's JemallocAllocator (CGO mallocx with 64-byte alignment + zeroing) backs the SkipListArena and the Arrow IPC allocator; internal/telemetry's Counter is a 64-stripe LongAdder with zero hot-path allocations (all construction at init()); internal/crypto's MaskPII returns the input verbatim with an identical data pointer on the no-PII fast path (proven by TestMaskPII_NoPII_ReturnsInputUnchanged comparing unsafe.StringData); internal/network's S3 ShardedKey uses fixed-size stack buffers (zero heap escape, BenchmarkShardedKey).
Where the law does not bind, it is honestly waived: VerifyCRDTFrame/RejectSmallOrderKey allocate ~3 edwards25519.Points per call — explicitly justified because signature verification itself allocates far more; the sign path (SignCRDTFrame) allocates per call — it is not the CRDT core apply hot path (which is 50.7M–68.3M ops/s); FrameReader.ReadFrame does make([]byte, frameLen) per frame — the zero-alloc law applies to the engine hot path, not this wire edge.
Why 128-byte cache-line padding (the cache law)
Cache law: 128-byte stride for every contended atomic. Two atomics on one cache line at 32 cores = HITM storm = 1.1M ops/s (1.6% efficiency) vs the 50.7M–68.3M ops/s CRDT core range. This was measured.
The mechanism: every hot atomic is CacheLinePad-isolated. ElimSlot is exactly 128 B (two cache lines) verified compile-time via unsafe.Sizeof; secShard is exactly 128 B; EBRManager.globalEpoch is on its own line; internal/telemetry's counterStripe is exactly 64 B (//go:align 64, 56-byte lead pad + 8-byte atomic). The gate is TestMemoryLayoutAnalysis (layout_analysis_test.go), which enumerates field offsets and flags any line holding >1 contended field as "FALSE SHARING RISK". The spatial SPSC ring carries the same discipline across a process boundary: RingSlot is exactly 64 B (one L1 line) and RingHeader is exactly 192 B (3 cache lines) with 56/56/52-byte padding isolating WriterCursor, ReaderCursor, and the control word — no false sharing between the Go writer and the C++ reader.
The CAS-storm sharding is the same law applied to free-lists: the three hottest CAS sites — class-0 node freelist (was 92% of AllocNode CPU at 32c), the var freelist, and the EBR per-epoch retire head — are sharded 256-way with route counters dispersing the locus (TestNodeFreelistSharded proves a 1.5× cardinality speedup).
Why CAS + EBR and no mutexes on the hot path (the lock law)
Lock law: No
sync.Mutexon the write path. CAS + EBR only. A futex is a scheduler stall. A scheduler stall at the CRDT core rate (50.7M–68.3M ops/s) is a cliff.
The mechanism: Join/InsertLocal/NextDot use CAS + EBR only. The only sync.Mutex in the CRDT core (persistMu) guards disk fsync in the decoupled background worker, not the hot CAS path; stateViewMu guards only the lazy State() merged view (off the hot path). EBR (reclamation.go) gives safe memory reclamation without fences on the read path: globalEpoch is CAS'd by AdvanceEpoch; participants pin the epoch on Enter; RetireBlock cannot recycle a node onto a Treiber free-list while a participant holds an epoch pin → the slab free-lists are ABA-immune (proven by TestHEBRDetachAllowsEpochAdvance and the ABA suite). Hazard pointers (DetachAndProtect) publish a retired-node address so a concurrent reader's isHazardProtected check can rescue it.
The same law propagates: internal/telemetry's hot path is wait-free (each writer picks stripes[stripeIndex()] deterministically and CAS-loops only on that one cache line); internal/network's ClientPool is lock-free acquire via connSlot.inUse.CompareAndSwap(false,true); the spatial SPSC ring uses atomic.StoreUint32/LoadUint32 with release semantics and a phased procyield/sync.Cond wait (deliberately eradicating runtime.Gosched()). Where mutexes do appear, they are honestly off the hot path: PeerBucket's 16 shards (per-shard sync.Mutex), Directory's sync.RWMutex (read-dominated receiver path), payloadCache's sync.Mutex (a TOCTOU guard), the WAL's sync.Mutex (serializes appends + f.Sync()).
Why replay restores the recorded dot (the WAL law)
WAL law: replay restores the recorded causal dot — it never re-mints one. Each replayed mutation is applied at the exact
(NodeID, Counter)the log recorded for it, and the post-snapshot tail is cut by the log's record sequence, not by a scalar clock. Re-deriving the clock and re-minting assigns an already-used counter to a different mutation — a double-mint — and a double-mint is silent divergence.
The mechanism: the write-ahead log records each mutation with the full fidelity of its causal dot (the 0x04 full-fidelity record in pkg/durability/wal.go). On recovery, the engine replays each record at that recorded dot rather than calling the minting path again, and a foreign node's clock advance is recorded in the log too, so the recovered clock is restored, not recomputed. The tail after a snapshot is selected by the snapshot's recorded sequence cut — the point in the log the image is known to cover — so a record either is or is not in the tail with no dependence on clock arithmetic.
Two older schemes were measured and rejected because both break under concurrent mint/append reordering: re-deriving the seed as high-water − count, and re-minting from first-mutation-counter − 1. Both re-mint, and re-minting double-mints. The determinism contract is byte-identity: the Merkle root folds only DotNodeID + DotCounter under SHA-256, so an exact replay of the same recorded dots reproduces the identical root — proven by a recovery-determinism test that replays N mutations plus a checkpoint and asserts byte-equal roots. This exactness is also what makes the integrity checksums meaningful: because replay is exact, a corrupted record can be refused rather than silently absorbed.
Recovery is bounded, not linear in total writes: if a snapshot image exists, the engine seeds from it and replays only the records after its cut — O(post-checkpoint) instead of O(writes-since-boot) — and reports which branch it took and why.
Why tri-temporal + H3 spatial + post-quantum from day one
Tri-temporal is the moat: keying state by (system_time × valid_time × assertion_time) (plus decision_time on the wire) lets the engine answer "who owned entity X at valid-time V as the system knew it by tx-time T" — a query a mutable-row or log-only model cannot answer without an auxiliary history store. The durable tier materializes this as the 40-byte composite key and the 9-field Arrow IPC rows; the read path enforces four-record-batch filters (Filter1 hash, Filter2 SystemTime <= txTime STRICT, Filter3 half-open valid-time, Filter4 entity-id collision guard) with Range generalizing to interval-intersection (not point-in-window — a distinction that prevents a data-loss class). Dominance pruning is three claws of the tri-temporal lattice: C1 structural sweep order, C2 [vs,ve) containment, C3 the txTime <= horizon FLOOR guard — pure function DominancePrune(rows, horizon), idempotent, preserve-all the byte-identical default.
H3 spatial is a reserved spatial axis. The H3Index uint64 field is carried on every CRDTEntry (hamt.go:39, @112) and in the Arrow schema (h3_index Uint64), and round-trips on the wire — but H3 cells are not computed on any in-tree path. The in-tree Go producer — a memfd-backed SPSC shared-memory ring (internal/spatial/h3_spsc_ring.go, memfd_create(2) + MAP_SHARED cross-process, cache-line-padded) fronted by an EpochBatcher that batches coordinate submissions — ships in the repo but is currently unwired: internal/spatial is imported by zero production files (only its own unit tests exercise it). The C++ h3_worker consumer that would compute the cells is operator-supplied / out-of-tree and is not present in the repository. The only in-tree H3Index assignment is the synthetic key-derived uint64(k) << 8 in the internal/chaos partition harness. H3 today is a carried spatial axis with a placeholder producer, not a live native geospatial index.
Honest caveat — H3 worker
The C++ h3_worker binary is operator-supplied and absent from the repo. The in-tree Go SPSC-ring producer (internal/spatial) ships but is not yet wired into the engine's data path (no production importer — only its own unit tests). H3 is a reserved spatial axis, not a live index.
Post-quantum from day one is a hedge against harvest-now-decrypt-later — and it is in the default build, not a preview. The transport negotiates X25519MLKEM768 key exchange by default, and a hybrid Ed25519 + ML-DSA-65 signature is compiled in with a BOTH-required verify — opt-in at runtime behind --hybrid-sign/--hybrid-verify, so once enabled a classical or a quantum break alone cannot forge a frame. The classical Ed25519 verify gate (identity.VerifyCRDTFrame) is the production seam; a hedged (randomized-nonce) signer (eddsa_hedge.go) stays compatible with the unchanged verifier (the exact s·B = R + k·A construction of RFC-8032 §5.1.6). The size economics are measured, not assumed: an ML-DSA-65 signature is 3309 B versus Ed25519's 64 B — 51.7× — and a test enforces that ratio so the cost cannot drift silently.
Why a chaos test harness
internal/chaos exists because three invariants cannot be synthesized by external tools (AWS FIS, Chaos Mesh):
- Semantic Byzantine injection (
byzantine.go): a DotCounter ratchet towardMaxUint64re-signed with circl is cryptographically valid but semantically malicious — FIS cannot forge a valid signature on mutated material. The catch isadmission.PeerBucket.Accept, proven byTestByzantine_A2RatchetCaughtByAdmission(Max-ratchet dropped in 1 admit) andTestByzantine_A2RatchetIncrementalCaught(incremental drains withinmaxAdmits=5). - Merkle convergence under partition (
partition.go+virtualnet.go): an in-memoryVirtualNetsimulates partitions, drops, duplicates, reorders and drives realDeltaCRDTEngineanti-entropy —TestMerkleConvergenceAfterPartition(32 nodes, byte-equal roots after heal). - Process-crash survival from off-heap SIGSEGV (
supervisor.go+fuzzer.go+probe.go): a raw off-heap SIGSEGV in a child worker is recovered from the WAL without dropping an active TCP connection — becauserecover()cannot catch a SIGSEGV inmmap'd C-space (TestSIGSEGVSurvival, conditionally skipped pendingCHAOS_WORKER_BIN).
The harness also enforces no-fabrication checks via go/ast source scans: TestByzantine_NoVectorClockLamportTime bans deprecated identifiers, TestByzantine_UsesRandV2 requires math/rand/v2, TestByzantine_NoInPlaceDeltaMutation bans in-place delta mutation.
The honesty law
Honesty law: Report numbers, not adjectives — and report the layer. "The core microbench gate passes at 50,736,038 ops/s @32 cores; this tree re-measures 68,278,197 ops/s; production ingest is 5.7M–6.0M deltas/sec" is a fact. Quoting any single peak as "sustained throughput" is a hero-number round-up. "Blazing fast" is evidence of incompetence.
This is not a workflow rule but a load-bearing engineering discipline enforced by gates: hardware-honesty tests assert that a benchmark ran on the hardware its label claims, and forbid relabeling a 4-core number as a 32-core one. Checksum-pinning tests pin the byte-identity of the load-bearing core across changes so it cannot drift silently. Honest negatives are recorded verbatim: BenchmarkEBPFDelivery_vs_HashFallback_32c tolerates an eBPF-steer-slower-than-hash-fallback NEGATIVE at single-box scale; BenchmarkBatchedVerify records its negative as "ACCEPTED-with-NEGATIVE-perf". The SDK honestly reports the originator-vs-peer payload boundary (TestClientGetOnPeerReturnsDigestNotValue: a peer Get returns Payload=="" + PayloadDigest!="").