Integration & Usage
An honest readiness assessment: what is production-wired, what is opt-in, what is bench-only, and what is a stub — with the test that proves each claim.
The production binary: cmd/sovereign-node
cmd/sovereign-node/main.go is the single binary that wires every seam: the frozen δ-CRDT engine, the receive gate stack, the TLS 1.3 peer listener, the JSON/mTLS control port, the plain-HTTP /livecheck+/metrics ops surface, opt-in WAL durability + bounded snapshot recovery, the L0→L1 compaction scheduler with T_gc auto-inference, the L0 reaper, the OTel meter provider, and the read-your-writes live source adapter.
Defaults: defaultArenaSize=64 MiB, defaultAdmissionBudget=50 ms.
Flags
Identity & transport
--bind peer data-plane TLS listener (default 127.0.0.1:7430)
--peers comma-separated peer host:port list (optional @region suffix)
--tls-cert --tls-key --tls-ca mTLS leaf cert/key and CA bundle paths (required unless --selftest)
--node-id 16-byte node ID (hex); derived from --identity-seed if absent
--identity-seed Ed25519 seed for node identity (derives nodeID + leaf)
--selftest mint self-test certs, run a 5s smoke check, and exit (opens no control port)Client API & observability
--control-addr mTLS /v1/* control-port listener (OFF by default — required for the SDK)
--metrics-addr plain-HTTP /metrics + /livecheck listener (default 127.0.0.1:7431)
--pprof-addr loopback-only /debug/pprof listener (default = metrics port + 1000)
--diag-profile verbose diagnostic logging
--otel arm the OTel exporter at boot
--otel-interval OTel export intervalEngine & replication
--arena-mib HamtArena size in MiB (default 64)
--admission-budget-ns relay depth budget in nanoseconds (default 50ms)
--gossip-tick anti-entropy sweep interval (default 100ms)
--batch-size batched-send batch size (default 100, max 256)
--lamport-absolute-slack Lamport skew bound (default 1000)
--stratified-anti-entropy IBLT digest-exchange reconciliation (default off)
--region-aware region-aware fan-out (default off)
--self-region this node's region id (default 0)
--region-fanout inter-region fan-out (default 3)
--peer-dir pre-provisioned peer public-key directory
--peer-auto-reconcile re-key peers as they join/rotate (default off)Post-quantum & PKI lifecycle
--hybrid-sign sign deltas with Ed25519 + ML-DSA-65 (default off)
--hybrid-verify require hybrid Ed25519 + ML-DSA-65 on receive (default off)
--cert-rotation-enable auto-rotate the leaf before expiry (default off)
--cert-rotation-poll rotation check interval (default 1h)
--cert-rotation-lifetime leaf lifetime (default 720h = 30 days)
--crl-path certificate revocation list; revoked certs are rejected at the handshakeDurability & compaction
--wal-path WAL file path (enables durability)
--wal-checkpoint-interval checkpoint interval (default 1000)
--lsm-root durable-tier root (LocalFS root or S3 bucket)
--compaction-prune-enable opt-in dominance pruning (default false)
--compaction-prune-horizon-ns retention floor (ns)
--compaction-prune-backoff-ns observed-frontier backoff (ns)
--compaction-reap-enable opt-in L0 reaper (default false)
--compaction-reap-interval reaper sweep interval (default 5m)The control port is off by default
--control-addr is empty by default, so the SDK and the /v1/* API are disabled until you set it. And the read-history endpoints (/v1/query, /v1/range) return 503 unless both --wal-path and --lsm-root are set — they read the durable tier, not just the live state.
Key functions
parseFlags, resolveNodeID, parsePeers, run, acceptLoopWithDigest, serveConnWithDigest, convergenceGaugePoller, startLivecheck, startControlPort, mintSelftestCerts, buildNodeIdentity, compactionSchedulerLoop (hardcoded 30 * time.Second interval, off the write path), runCompactionSweep, reaperLoop, runReaperSweep, engineHAMTAdapter (implements database.LiveSource; the ONLY place the concrete engine is in scope for internal/database, avoiding an import cycle). armOTel (otel.go) arms the OTel MeterProvider gated on --otel; logOutputExporter writes OTel batches to the operator log stream (NOT a Prometheus bridge — the metrics bridge is separate).
The client SDK: sdk/sovereign
sdk/sovereign/client.go is a <50-line mTLS control-port client. Dial(addr, tlsCfg) forces Min==Max==tls.VersionTLS13 + ForceAttemptHTTP2:false; DialWithCerts(addr, certPath, keyPath, caPath, serverName) loads on-disk credentials.
type Client struct { httpClient *http.Client; baseURL string }
type GetResult struct { ... Payload string; PayloadDigest string ... } // originator-vs-peer boundary
type MetricSample struct { Name, Labels string; Value float64 }
type MetricSamples []MetricSample
func Dial(addr string, tlsCfg *tls.Config) (*Client, error)
func DialWithCerts(addr, certPath, keyPath, caPath, serverName string) (*Client, error)
func (c *Client) InsertLocal(key, val string) (dotHex string, err error) // POSTs /v1/insert; returns the write's causal dot
func (c *Client) Get(key string) (GetResult, error) // GET /v1/get — Payload only on originator; digest on peers
func (c *Client) MerkleRoot() (string, error) // GET /v1/merkle
func (c *Client) Status() (NodeStatus, error)
func (c *Client) Metrics() (MetricSamples, error) // parses /metrics, label-preserving
func (c *Client) RunDemo() error
func (c *Client) Close() error
func (s MetricSamples) Value(name string) (float64, bool) // false for 0/multiple samples
func (s MetricSamples) Samples(name string) []MetricSampleHonest boundary — originator vs. peer
GetResult.Payload is non-empty only on the originator (cache hit); peers return Payload=="" + PayloadDigest!="" because the engine stores only the PayloadDigest on a joined CRDTEntry (TestClientGetOnPeerReturnsDigestNotValue). The SDK does not claim linearizability — InsertLocal returns at LOCAL-apply; peer convergence is eventual (next gossip sweep).
Examples
examples/sdk/main.go— the canonical <50-line SDK example (main,run).examples/embed/main.go— the older lock-free stack export (ShardedStack/EliminationStack+ per-goroutineNewElimPRNG, 200k ops/worker, assertspushed - popped == drained). It proves external linearizable usability of the stack core, not the δ-CRDT HAMT engine — a historical artifact, not the current engine. RequiresGOMAXPROCS>=2.
Control surface
| Endpoint | Transport | Purpose | Status |
|---|---|---|---|
/livecheck | plain HTTP | liveness | Verified |
/metrics | plain HTTP | Prometheus scrape (sovereign_* labelled + supremum_* cumulative) | Verified |
/v1/insert | mTLS | local insert (→ WAL fsync → checkpoint → L0 Arrow) | Verified |
/v1/batch-insert | mTLS | batched insert — one WAL fsync per batch | Verified |
/v1/get | mTLS | single-shard point read (digest on peers, value on originator) | Verified |
/v1/query | mTLS | bitemporal AsOf (503 unless --wal-path + --lsm-root are set) | Verified |
/v1/range | mTLS | bitemporal Range (interval-intersection, capped by MaxRangeRows) | Verified |
/v1/merkle | mTLS | current root + commit stats | Verified |
/debug/memstats | plain HTTP | heap/GC/cache stats + the recovery witness | Verified |
/debug/pprof | loopback | Go profiler (bound to loopback, off the public path) | Verified |
handleInsert stamps an open-ended ValidTime default; handleQuery/handleRange echo PayloadDigest verbatim with deliberately no payload field (the index stores a sentry body; reporting one would fabricate a value that is not stored). parseQueryTime and dotHex are helpers; OpenEndedValidEndNs = 9e18 (year ~2253, deliberately NOT database.MaxValidTime.UnixNano() which overflowed).
TLS / cert provisioning
The dev-mesh CA (pkg/crypto/certgen.go: NewMeshCA → IssueLeaf → WriteCAPEM/Leaf.WritePEM) mints an Ed25519 self-signed CA (10-year, IsCA, CertSign|CRLSign, MaxPathLen=1) and 1-year server+client leaves (DigitalSignature, ServerAuth+ClientAuth, DNSNames {nodeID, localhost}); 62-bit crypto/rand serials; PEM to disk (CA pubkey at 0644 — CA private key stays in-process; leaf key PKCS8 at 0600). The binary's --tls-cert --tls-key --tls-ca point at these; SIGHUP reloads the leaf, the CA pool, and the CRL — so a revoked or expired certificate is rejected at the handshake with no listener restart.
Dev mesh only
This is a development CA, not production PKI. Leaf rotation and CRL revocation are wired (a revoked or expired cert is rejected at the handshake), but an offline root, intermediate CAs, HSM-backed key custody, and OCSP are not — supply your own PKI for production.
Honest readiness assessment
Status — research preview, not production-ready
The Sovereign Engine is a research preview, not production-ready software. The core is measured and verified on real hardware, but the engine is shared early so the design and the numbers can be checked — do not trust it with data you cannot afford to lose. The matrix below is the honest per-capability map of what is wired, opt-in, bench-only, or stub.
| Capability | Readiness | Evidence |
|---|---|---|
| δ-CRDT core (Join, HAMT, IBLT, EBR) | Verified | The merge law and wire schema are frozen and property-tested; property tests + partition-heal convergence; 0 allocs/op on the hot path |
| Per-shard point reads | Verified | O(log N) reads against one shard — retired the whole-state read that rebuilt the index on every call |
| Wire-integrity + skew bound | Verified | Digest, attribution, and Lamport-skew checks run before any merge |
| TLS 1.3 mTLS mesh + control port | Verified | TLS 1.3-only, mutual authentication; the client API rides the same trust root |
| Certificate + CA + revocation hot-reload | Verified | A revoked or expired certificate is rejected at the handshake; rotation needs no restart |
| Admission (rate + clock) | Verified | Per-peer Sybil-burst isolation plus a Byzantine clock cap, before the signature check |
| WAL + bounded snapshot recovery | Verified | Group commit — one fsync per batch; exact-dot replay; a recovered node re-equals the fleet |
| WAL + snapshot integrity (CRC32C) | Verified | Every record is checksummed; corruption fails loud and never replays silently |
| Checkpoint decouple + delta-batch | Verified | Checkpoint fsyncs moved off the commit path (110,000 → 0 per checkpoint) |
| Convergence — 100 node processes on 3 machines, 3 regions | Verified | The sub-10-second target is met on the published run (8.95 s) and on six of seven logged runs; one run missed at 10.15 s |
| Crash recovery + integrity verification | Verified | Bounded recovery re-equals the checkpoint root with the integrity check armed |
| Retention horizon (auto-inferred) | Verified | Tracks the live query frontier; never retreats below the operator-set floor |
| Read-your-writes | Verified | An insert is immediately queryable — no wait for the next checkpoint |
| Ed25519 verify + hedged signer | Verified | A canonical-verify gate plus a randomized-nonce signer that stays compatible with it |
| Post-quantum transport | Verified | X25519MLKEM768 key exchange on by default; hybrid Ed25519 + ML-DSA-65 signing is opt-in |
| Region-aware gossip | Opt-in | Intra-region mesh + inter-region fan-out; off by default |
| IBLT digest exchange | Opt-in | Ships only the minimal difference; the saturation bound is disclosed, not hidden |
| Peer provisioning | Opt-in | A pre-provisioned peer directory lets a node verify a peer before first contact |
| L0→L1 compaction + pruning | Opt-in | Off by default; the operator sets the retention horizon |
| L0 reaper | Opt-in | Off by default; never runs unless enabled |
| eBPF SK_REUSEPORT steering | Verified | Kernel-level flow pinning — 0/1000 mis-routes measured on silicon; ships behind a build tag |
| Zero-GC PII masking | Verified | Deterministic redaction on the write path with no allocation |
| Telemetry → Prometheus | Verified | New counters surface on /metrics automatically; the scrape reads cumulative values |
| Skip-list range iterator | Dormant | Proven correct but not yet wired into the durable scan path |
| EPOLLET Cap’n Proto ingestion | Partial | Proven end-to-end on a Unix socket; not the default ingest path |
| H3 spatial (C++ worker) | Partial | A reserved spatial axis on the data model and wire; the indexing pipeline is built but not wired in |
| aws-lc hedged bridge | Stub | A placeholder; the real CGO bridge is not built |
| Development PKI | Dev only | Rotation + revocation are wired; it remains a development CA, not production PKI |
| Cedar ABAC | Bench-only | Benchmarked, but no authorization engine is wired into the request path |
| Chaos harness | Bench-only | The verification layer; the process-crash survival test is opt-in |
Readiness is per-capability: a “Verified” badge means the capability is wired into the serving path and backed by a named test gate or a silicon run — not that the engine as a whole is production-ready. Sovereign Engine is a research preview, not production-ready software.
Additional disclosed debt
| Capability | Status | Evidence |
|---|---|---|
payloadCache | Verified (unbounded) | No eviction/LRU yet; documented debt |
| L0 file growth | Opt-in debt disclosed | One fresh l0/*.arrow per checkpoint, no merge/compaction by default; bounded by ResolverConfig.MaxL0Files (1000); TestQuery_L0FileGrowthDisclosed asserts the debt's existence |
| Postgres bulk load | Partial | InitializeSwarmPool + UnloggedStagingPromotion (uses a TEMPORARY table despite the name); no live COPY test — only SQL-string construction and config parsing are tested |
| S3 uploader | Partial | AWSS3Uploader with SHA-256 prefix sharding (65,536 partitions); multipart is sequential, not concurrent; no live Upload test (AWS creds assumed absent); database.S3Uploader conformance asserted only via compile-time var _ |
| Bench evidence tracks | Bench-only | pkg/codec120, pkg/durability120, pkg/pqecobench — no production importers; produce the byte-cost/latency CDFs behind the acceptance gates |
Getting started — operator
- Provision a mesh CA (dev): use
pkg/crypto'sNewMeshCA/IssueLeaf/WriteCAPEM, or supply your own PKI. - Derive node identity: pass
--identity-seed;nodeIDmust equalengine.localNodeID. - Start a node (see Quickstart).
- Opt into durability/compaction once you trust the T_gc floor; add the reaper once you trust your storage layer's existence-probe.
- Observe: scrape
/metrics; hit/livecheck; arm--otel. - Write/read: via the SDK over mTLS to the control port —
InsertLocal,Get,AsOf/Range.
Getting started — developer
Embed the stack core via examples/embed (the historical lock-free stack API) or drive a full node via examples/sdk (the canonical SDK path). For direct engine use, the pkg/sync public surface is:
eng, err := sync.NewDeltaCRDTEngine(nodeID, initialCounter, arenaSize)
dot := eng.InsertLocal(entityID, entry) // 0 allocs/op hot path
eng.Join(delta) // frozen merge-union
remoteDigest := eng.GenerateDigest() // IBLT
delta := eng.GenerateDelta(remoteDigest) // minimal missing set; defer delta.Release()
eng.ApplyCRDTDeltaEvent(wire) // wire-integrity + JoinFor durability, pkg/durability:
wal, _ := durability.OpenWAL(path)
bridge := durability.NewBridge(eng, wal, checkpointInterval)
dot, err := bridge.PutLocal(entityID, payload, entry) // InsertLocal + WAL fsync
bridge.AppendCheckpoint()
eng, wal, replayed, err := durability.RecoverEngine(nodeID, walPath, arenaSize)
eng, wal, _, witness, err := durability.RecoverEngineWithSnapshot(nodeID, walPath, store, arenaSize) // boundedFor queries, internal/database:
resolver := database.NewResolver(lister, downloader, alloc, bucket, database.DefaultResolverConfig)
row, err := resolver.AsOf(ctx, entity, validTime, txTime)
rows, ok, err := resolver.Range(ctx, entity, vLo, vHi, tx)
// LiveSource (read-your-writes): set cfg.LiveSource to an EBR-pinned adapter