Verification & Status
This page is the engine's evidence ledger. Every capability claim on this portal traces to a test, a measured silicon run, or a checksum-pin — and the boundary of what is not yet proven is stated as plainly as what is. Nothing here is a projection.
Verified at silicon
Three guarantees define whether the engine is ready to trust with real state. Each is verified on real multi-node hardware under an orchestrated harness that injects partitions and kills nodes — not on a loopback simulation.
The fleet: 100 node processes on three machines — one c7gd.8xlarge per region, us-east-1 (34 processes), eu-west-1 (33), ap-southeast-2 (33) — each an AWS Graviton3 box with 32 vCPU, 64 GiB, 1×1900 GB local NVMe and Amazon Linux 2023 arm64. The NVMe is load-bearing: the write-ahead log fsyncs to local flash, not network storage.
| Guarantee | The claim | The measured result |
|---|---|---|
| Convergence | A 10,000-key write converges across 100 node processes on 3 machines in 3 regions in under 10 seconds | Met on the published run — 8.95s (t50 0.22s, t90 3.77s); the decision log records seven consecutive runs before it, six between 6.6s and 8.7s and one miss at 10.15s, with integrity verification armed and production checkpointing on |
| Crash recovery | Kill a node mid-run; it recovers and its Merkle root re-equals the fleet | Proven — bounded recovery from the snapshot horizon, root re-equals, with the payload-checksum integrity check armed |
| Durability | Every committed write survives a crash | Held on every run — the write-ahead log fsyncs once per batch before the write is acknowledged |
A run only counts when the integrity check that would catch a silent divergence is confirmed to be on. A pass with the check disabled is recorded as no evidence, not as a pass.
Work continues — not because these gates failed, but because met once and robust at the tail are different claims. The remaining edge cases are tracked openly below, not hidden.
What "verified" means here
The engine grades its own evidence and refuses to let a weaker tier stand in for a stronger one:
- Silicon — a real multi-node run on named hardware (instance class, CPU, and toolchain recorded). The strongest tier.
- Loopback — the real code paths over an in-process network. Proves logic, not wall-time.
- Unit / property — a single invariant, usually paired with a fault-injection test that deliberately introduces the defect to prove the guard would catch it.
Three disciplines keep the evidence honest:
- Frozen core. The merge law and the wire schema — the files everything else depends on — are held fixed: they change only through a deliberate, recorded decision, and every change is covered by property tests and a byte-identity review.
- Fault-injection proof. A guard that "would catch the bug" is proven by introducing the bug and watching the guard fire. An assertion that cannot fail is not treated as an assertion.
- Honest negatives. A benchmark that regresses is recorded verbatim, not re-run until it passes.
The depth of that verification is itself a measurable property of the tree: test, fuzz, fault-injection, and benchmark code outweighs the engine itself — roughly 1.6 lines of verification for every line of engine code. That ratio is a deliberate investment, not an accident of growth, and it is stated as a proportion precisely so it stays true as the codebase scales.
The operating envelope, measured
A number here always carries its layer, its range, and its provenance. A number missing any of the three is a hero number, and a hero number is forbidden.
| Layer | Measured | Provenance |
|---|---|---|
| CRDT core microbenchmark | 50,736,038 ops/s gate floor; this tree re-measures 68,278,197 ops/s @32 cores (the honest range is 50.7M–68.3M) | In-process HAMT.Set crucible — no crypto, network, TLS, or disk; c8g.8xlarge (Graviton4 Neoverse V2, CPU part 0xd4f), Go 1.26 |
| Production ingest | 5.7M–6.0M deltas/sec @32 cores (N=100–256; multi-origin ceiling 12.5M–14.7M) | the receive path, in-process — one Ed25519 verify per frame, envelope decode, CRDT apply; no socket, no disk |
| Hot-path allocation | 0 allocs/op | a zero-allocation test gate on the core data structure |
| Ed25519 verify | ~60 µs/op | the crypto-dominated ingress cost |
| ML-DSA-65 verify | ~73.7 µs/op | the hybrid gate runs it only after the classical check passes |
| Convergence, 100 node processes | 8.95s on the published run; six of seven logged runs 6.6–8.7s, one miss at 10.15s | 3× c7gd.8xlarge (Graviton3 + NVMe), one per region: us-east-1, eu-west-1, ap-southeast-2 |
| Checkpoint cost | commit-path checkpoint fsyncs 110,000 → 0 | the checkpoint was decoupled and delta-batched off the write path |
The hardening arc
These guarantees did not hold on the first attempt. Each issue below was surfaced by measurement, root-caused, fixed, and re-proven — most with a fault-injection test confirming the fix is load-bearing. This is the work that doesn't fit on a feature list, presented because the process is the point.
| Area | The issue | The resolution |
|---|---|---|
| Shutdown safety | Closing a node could free the off-heap arena while memory reclamation was still walking it — a use-after-free on shutdown | Shutdown now waits for in-flight readers to drain and flushes the clock's high-water mark before freeing; zero faults over a 40-minute 32-core stress run |
| Write latency | The write-ahead log held a lock across the disk sync, stalling commits under storage contention | Every append path now releases the lock before the sync |
| Convergence stalls | A 300-second stall traced — by measurement, not assumption — to the checkpoint writing one file per entity, synchronously, on the commit path | Checkpointing was moved off the commit path and delta-batched; the convergence target then passed |
| Read memory | A point read rebuilt the entire merged index on every call, exhausting memory under sustained reads | Reads now target a single shard — O(log n), flat memory |
| Heap correctness | A wrong-sized deallocation corrupted the allocator's metadata | Deallocation now derives the size from the allocator's own accounting, eliminating the class |
| Data integrity | The log and snapshot had no content checksum — on-disk corruption was silent | Every record and every snapshot now carries a CRC32C checksum; corruption fails loud instead of rebuilding from rotten bytes |
The honest boundary
What is not yet proven, stated as plainly as what is:
- The multi-region convergence tail. The convergence target is met, but the last one or two nodes occasionally straggle past the 10-second bar (about one run in seven). This is a gossip-distribution property, independent of durability, and is the open hardening target.
- The payload cache is unbounded. It has no global size bound yet; bounding it is planned work. The relay cache's unbounded-growth defect is fixed: retention is now refcount-bounded to live state (guarded by
TestRelayCacheBounded), though it still carries no fixed byte ceiling. - H3 spatial is a reserved axis, not a live index. The spatial field rides the wire and the durable tier, but the in-process indexing pipeline is built-and-unwired and its native worker is supplied by the operator. See Key Features.
- Cedar ABAC is a pinned dependency, not yet benchmarked or wired. No authorization engine is in the request path yet.
- eBPF ingress is opt-in. Kernel-level flow steering is proven on silicon but ships behind a build flag, off the default path.
- Process-crash survival is opt-in. The supervisor/worker recovery harness exists, but its worker binary is not in the default build, so that specific gate is off unless enabled.
The full readiness classification — verified, opt-in, dormant, bench-only, stub — is on the Integration page, and the engineering laws this evidence serves are on Design Rationale.