Design Rationale & Philosophy

The engine is governed by five physical laws stated in CLAUDE.md and enforced by named test gates. Each law exists because a measured failure mode made it non-optional. This section ties each rationale to the real mechanism that implements it.

M

Memory law

0 allocs/op on the hot path. One make() on the write path at the core rate (50.7–57.6M ops/s) = stop-the-world ~every 4ms.

→ TestHotPathZeroAllocations

C

Cache law

128-byte stride for every contended atomic. Two atomics on one cache line at 32 cores = a HITM storm = 1.1M ops/s, 1.6% efficiency vs the 50.7–57.6M core range.

→ TestMemoryLayoutAnalysis

L

Lock law

No sync.Mutex on the write path. CAS + EBR only. A futex is a scheduler stall, and a stall at the core rate (50.7–57.6M ops/s) is a cliff.

→ pkg/sync · reclamation.go

W

WAL law

Replay starts at LamportHigh − len(Mutations), not LamportHigh. Replay re-runs the minting — change this and you get double-minting.

→ TestStage6WALRecoveryDeterminism

H

Honesty law

Report numbers, not adjectives — and report the layer. “50,736,038 ops/s @32c is the CRDT CORE microbench gate-passing floor (range 50.7–57.6M; the 57.6M is a residency high, not sustained; production ingest is 1.0–3.1M δ/sec)” is a fact. Quoting 57.6M alone as “sustained throughput” is a hero-number round-up the post-mortem forbids.

→ TestGate_GearHonesty

Why δ-CRDTs (vs operational logs)

An operational log conflates what happened with what the system believed. A δ-CRDT separates them: state is a join-semilattice element, and a delta is the minimal state-difference needed to bring a remote replica up to date. The merge operation is a pure function (Join, crdt.go:1089, FROZEN), so convergence is a lattice-join property — commutative, associative, idempotent — proven by the rapid-based property tests (TestCRDTJoinCommutativity, TestCRDTJoinAssociativity, TestCRDTJoinIdempotence, TestCRDTConvergenceMultiNode, TestCRDTJoinMonotonicGrowth) and proven to survive a lossy/duplicating/reordering/partitioned transport by internal/chaos's TestStage6MerkleConvergenceAfterPartition (32 engines, asymmetric partition, 12 gossip rounds, byte-equal roots after heal) and TestStage6ConvergenceDeterminismAcrossRuns (two independent 8-node runs converge to the same root with dedup disabled — Join idempotence alone guarantees correctness).

The δ-prefix is what makes this planetary: instead of shipping full state, the engine ships GenerateDelta(remoteDigest) (the set of entries the remote is missing, computed via IBLT subtract+peel) or GenerateDeltaStratified(remoteSE) (minimal delta ∝ |A−B| via strata estimation). This is why the mesh can converge 1000 split events in ≤10 rounds over real TLS 1.3 loopback (TestTwoNodeConvergence_In_Memory, GATE C).

A known, deliberately-not-fixed regression

Join is MERGE-UNION on CausalDot, but LWWOperator.Resolve (elsewhere) drops the loser — a dropped dot cannot re-merge across a foreign AdvanceLamportTo. Per ADR-0033 this is a won't-fix in pkg/sync; the genuine lossless conflict resolution is the live HAMT read path shipped Day-27, which reads the full dot set losslessly via selectLatestDot and engineHAMTAdapter.

Why off-heap / zero-GC (the memory law)

Memory law: 0 allocs/op on the hot path. Every allocation on the hot path is a future GC pause. One make() on the write path at the CRDT core rate (50.7M–57.6M ops/s, the HAMT.Set microbench) = stop-the-world ~every 4ms. Unacceptable.

The mechanism: NodePtr is a GC-invisible uintptr; all node/leaf/string/entry-array data lives in the mmap'd HamtArena; even the *HAMT wrapper is arena-allocated (allocHAMTWrapper); makeBinaryKey writes into a stack [8]byte then unsafe.String (safe because makeLeaf synchronously copies bytes into the arena before Set returns). The gate is TestHotPathZeroAllocations asserting HAMT.Set = 0 allocs/op. The raceEnabled build-tag pair lets it self-skip under -race (shadow-memory instrumentation inflates AllocsPerRun).

The same philosophy propagates outward: internal/database's JemallocAllocator (CGO mallocx with 64-byte alignment + zeroing) backs the SkipListArena and the Arrow IPC allocator; internal/telemetry's Counter is a 64-stripe LongAdder with zero hot-path allocations (all construction at init()); internal/crypto's MaskPII returns the input verbatim with an identical data pointer on the no-PII fast path (proven by TestMaskPII_NoPII_ReturnsInputUnchanged comparing unsafe.StringData); internal/network's S3 ShardedKey uses fixed-size stack buffers (zero heap escape, BenchmarkShardedKey).

Where the law does not bind, it is honestly waived: VerifyCRDTFrame/RejectSmallOrderKey allocate ~3 edwards25519.Points per call — explicitly justified because signature verification itself allocates far more; the sign path (SignCRDTFrame) allocates per call — it is not the CRDT core apply hot path (which is 50.7M–57.6M ops/s); FrameReader.ReadFrame does make([]byte, frameLen) per frame — the zero-alloc law applies to the engine hot path, not this wire edge.

Why 128-byte cache-line padding (the cache law)

Cache law: 128-byte stride for every contended atomic. Two atomics on one cache line at 32 cores = HITM storm = 1.1M ops/s (1.6% efficiency) vs the 50.7M–57.6M CRDT core range. This was measured.

The mechanism: every hot atomic is CacheLinePad-isolated. ElimSlot is exactly 128 B (two cache lines) verified compile-time via unsafe.Sizeof; secShard is exactly 128 B; EBRManager.globalEpoch is on its own line; internal/telemetry's counterStripe is exactly 64 B (//go:align 64, 56-byte lead pad + 8-byte atomic). The gate is TestMemoryLayoutAnalysis (layout_analysis_test.go), which enumerates field offsets and flags any line holding >1 contended field as "FALSE SHARING RISK". The spatial SPSC ring carries the same discipline across a process boundary: RingSlot is exactly 64 B (one L1 line) and RingHeader is exactly 192 B (3 cache lines) with 56/56/52-byte padding isolating WriterCursor, ReaderCursor, and the control word — no false sharing between the Go writer and the C++ reader.

The CAS-storm sharding (Phase 2.5a.1/2.5b.1/2.5d) is the same law applied to free-lists: the three hottest CAS sites — class-0 node freelist (was 92% of AllocNode CPU at 32c), the var freelist, and the EBR per-epoch retire head — are sharded 256-way with route counters dispersing the locus (TestPhase25A1_NodeFreelistSharded proves a 1.5× cardinality speedup).

Why CAS + EBR and no mutexes on the hot path (the lock law)

Lock law: No sync.Mutex on the write path. CAS + EBR only. A futex is a scheduler stall. A scheduler stall at the CRDT core rate (50.7M–57.6M ops/s) is a cliff.

The mechanism: Join/InsertLocal/NextDot use CAS + EBR only. The only sync.Mutex in the CRDT core (persistMu) guards disk fsync in the decoupled background worker (Phase 2.5c), not the hot CAS path; stateViewMu guards only the lazy State() merged view (off the hot path). EBR (reclamation.go) gives safe memory reclamation without fences on the read path: globalEpoch is CAS'd by AdvanceEpoch; participants pin the epoch on Enter; RetireBlock cannot recycle a node onto a Treiber free-list while a participant holds an epoch pin → the slab free-lists are ABA-immune (proven by TestHEBRDetachAllowsEpochAdvance and the ABA suite). Hazard pointers (DetachAndProtect) publish a retired-node address so a concurrent reader's isHazardProtected check can rescue it.

The same law propagates: internal/telemetry's hot path is wait-free (each writer picks stripes[stripeIndex()] deterministically and CAS-loops only on that one cache line); internal/network's ClientPool is lock-free acquire via connSlot.inUse.CompareAndSwap(false,true); the spatial SPSC ring uses atomic.StoreUint32/LoadUint32 with release semantics and a phased procyield/sync.Cond wait (deliberately eradicating runtime.Gosched()). Where mutexes do appear, they are honestly off the hot path: PeerBucket's 16 shards (per-shard sync.Mutex), Directory's sync.RWMutex (read-dominated receiver path), payloadCache's sync.Mutex (Day-6.5 TOCTOU guard), the WAL's sync.Mutex (serializes appends + f.Sync()).

Why WAL replay starts at LamportHigh - len(Mutations) (the WAL law)

WAL law: Replay starts at LamportHigh - len(Mutations). NOT LamportHigh. Replay re-runs the minting. Change this and you get double-minting = data corruption.

The mechanism: InsertLocal re-stamps DotNodeID/DotCounter from NextDot(), so the recovered engine must start at the same initial Lamport to re-mint the same dots. RecoverEngine seeds rebuiltInitial = LamportHigh - len(Mutations) (recovery.go); the chaos harness proves it (TestStage6WALRecoveryDeterminism: N=64 mutations+checkpoint, recoveredRoot == liveRoot, recovered.LamportCounter() == rep.LamportHigh, per-mutation dot.Counter == initialCounter+i+1). The Day-8.5 refinement records foreign AdvanceLamportTo as WALRecClockAdvance=0x03 interleaved with mutations, so replay re-mints exactly with seed firstMutation.Counter-1 — closing the double-minting class for peer-driven Lamport jumps. The determinism contract is byte-identity: MerkleRoot folds only DotNodeID+DotCounter under SHA-256 (NOT maphash.Seed, which is non-serializable), so a fresh seed reproduces an identical root given identical (nodeID, lamport) replay.

Bounded recovery (Day-11) makes this O(post-checkpoint): SnapshotStore.SnapshotExists gates the bounded branch; if the dot-bearing snapshot image at ckpt/<LamportHigh> exists, seed from it and replay only the post-checkpoint tail; RecoveryWitness reports which branch ran and why.

Why tri-temporal + H3 spatial + post-quantum from day one

Tri-temporal is the moat: keying state by (system_time × valid_time × assertion_time) (plus decision_time on the wire) lets the engine answer "who owned entity X at valid-time V as the system knew it by tx-time T" — a query a mutable-row or log-only model cannot answer without an auxiliary history store. The durable tier materializes this as the 40-byte composite key and the 9-field Arrow IPC rows; the read path enforces four-record-batch filters (Filter1 hash, Filter2 SystemTime <= txTime STRICT, Filter3 half-open valid-time, Filter4 entity-id collision guard) with Range generalizing to interval-intersection (not point-in-window — the Day-8/13/14 data-loss class). Dominance pruning (Day-15, ADR-0020) is three claws of the tri-temporal lattice: C1 structural sweep order, C2 [vs,ve) containment, C3 the txTime <= horizon FLOOR guard — pure function DominancePrune(rows, horizon), idempotent, preserve-all the byte-identical default.

H3 spatial is carried on every CRDTEntry (H3Index uint64 @112) and in the Arrow schema (h3_index Uint64). Geocoding is offloaded to an isolated C++ worker over a memfd-backed SPSC shared-memory ring (internal/spatial/h3_spsc_ring.go) — cache-line-padded, MESI-coherent cross-process via MAP_SHARED, no syscalls/futexes/mutexes on the hot path — fronted by an EpochBatcher that batches coordinate submissions to amortize cross-process round-trips.

Honest caveat — H3 worker

The C++ h3_worker binary is operator-supplied; in-tree tests use mock Go consumers.

Post-quantum from day one is a hedge against harvest-now-decrypt-later. The Ed25519 verify gate (identity.VerifyCRDTFrame) is the production seam; a hedged (randomized-nonce) signer (eddsa_hedge.go) stays compatible with the unchanged verifier (proven by TestSignCRDTFrame_VerifiesUnderVerifyCRDTFrame — the construction follows the exact s*B = R + k*A equation per RFC-8032 §5.1.6). An ML-DSA-65 (FIPS 204) preview envelope (pq_mldsa.go, pq_preview build tag) is wired for promotion; its size economics are measured, not assumed: sig = 3309 B, pub = 1952 B = 51.7× sig inflation vs Ed25519 64 B on a 120 B frame — the datum behind the GO/NO-GO gate (TestVerdictMatrix_PQ enforces bloat-ratio >= 50×).

Why a chaos test harness

internal/chaos exists because three invariants cannot be synthesized by external tools (AWS FIS, Chaos Mesh):

1. Semantic Byzantine injection (byzantine.go): a DotCounter ratchet toward MaxUint64 re-signed with circl is cryptographically valid but semantically malicious — FIS cannot forge a valid signature on mutated material. The catch is admission.PeerBucket.Accept (gate 3.1), proven by TestByzantine_A2RatchetCaughtByAdmission (Max-ratchet dropped in 1 admit) and TestByzantine_A2RatchetIncrementalCaught (incremental drains within maxAdmits=5).
2. Merkle convergence under partition (partition.go + virtualnet.go): an in-memory VirtualNet simulates partitions, drops, duplicates, reorders and drives real DeltaCRDTEngine anti-entropy — TestStage6MerkleConvergenceAfterPartition (32 nodes, byte-equal roots after heal).
3. Process-crash survival from off-heap SIGSEGV (supervisor.go + fuzzer.go + probe.go): a raw off-heap SIGSEGV in a child worker is recovered from the WAL without dropping an active TCP connection — because recover() cannot catch a SIGSEGV in mmap'd C-space (TestStage6SIGSEGVSurvival, conditionally skipped pending CHAOS_WORKER_BIN).

The harness also enforces no-fabrication teeth via go/ast source scans: TestByzantine_NoVectorClockLamportTime bans deprecated identifiers, TestByzantine_UsesRandV2 requires math/rand/v2, TestByzantine_NoInPlaceDeltaMutation bans in-place delta mutation.

The honesty law

Honesty law: Report numbers, not adjectives — and report the layer. "50,736,038 ops/s @32c is the CRDT CORE microbench gate-passing floor (range 50.7M–57.6M; the 57.6M is a residency high, not sustained; production ingest is 1.0M–3.1M deltas/sec)" is a fact. Quoting 57,638,422 alone as "sustained throughput" is a hero-number round-up the cache-line post-mortem forbids. "Blazing fast" is evidence of incompetence.

This is not a workflow rule but a load-bearing engineering discipline enforced by gates: the gear-honesty teeth (TestGate_GearHonesty, TestBench_GearHonesty_4c, TestTrack36_GearHonesty, TestGateCedar_GearHonesty) assert NumCPU==4 else t.Skip, scan every .go in a package for a forbidden _32c tag, and forbid relabeling a 4c number as 32c. FROZEN-file MD5 teeth (TestGate_FrozenMD5, TestBench_FrozenMD5, TestTrack19_T7_FrozenByteIdentical_NoRePin) pin the byte-identity of load-bearing source across forks. Honest negatives are recorded verbatim: BenchmarkEBPFDelivery_vs_HashFallback_32c tolerates an eBPF-steer-slower-than-hash-fallback NEGATIVE at single-box scale; BenchmarkBatchedVerify records its negative as "ACCEPTED-with-NEGATIVE-perf". The SDK honestly reports the originator-vs-peer payload boundary (TestClientGetOnPeerReturnsDigestNotValue: a peer Get returns Payload=="" + PayloadDigest!="").