Integration & Usage
An honest readiness assessment: what is production-wired, what is opt-in, what is bench-only, and what is a stub — with the test that proves each claim.
The production binary: cmd/sovereign-node
cmd/sovereign-node/main.go is the single binary that wires every seam: the FROZEN δ-CRDT engine, the receive gate stack, the TLS 1.3 peer listener, the JSON/mTLS control port, the plain-HTTP /livecheck+/metrics ops surface, opt-in WAL durability + bounded snapshot recovery, the L0→L1 compaction scheduler with T_gc auto-inference, the L0 reaper, the OTel meter provider, and the read-your-writes live source adapter.
Defaults: defaultArenaSize=64 MiB, defaultAdmissionBudget=50 ms.
Flags
--bind peer data-plane TLS listener (host:port)
--peers comma-separated peer host:port list
--tls-cert --tls-key --tls-ca mTLS leaf cert/key and CA bundle paths
--node-id 16-byte node ID (hex); derived from identity-seed if absent
--metrics-addr plain-HTTP /metrics + /livecheck listener
--control-addr mTLS /v1/* control-port listener (optional)
--arena-mib HamtArena size in MiB (default 64)
--admission-budget-ns relay depth budget in nanoseconds (default 50ms)
--gossip-tick anti-entropy sweep interval
--identity-seed Ed25519 seed for node identity (derives nodeID + leaf)
--selftest mint self-test certs (dev)
--batch-size Day-5 batched-send batch size (default 100, max 256)
--wal-path WAL file path (enables opt-in durability)
--wal-checkpoint-interval checkpoint interval (LamportCounter ticks)
--lsm-root durable-tier root (LocalFS root or S3 bucket)
--compaction-prune-enable opt-IN Level-2 DominancePrune (default false)
--compaction-prune-horizon-ns T_gc operator floor (ns)
--compaction-prune-backoff-ns Day-22 observed-frontier backoff (ns)
--compaction-reap-enable opt-IN L0 reaper (default false)
--compaction-reap-interval reaper sweep interval (default 5m)
--otel arm OTel MeterProvider at boot
--otel-interval OTel export intervalKey functions
parseFlags, resolveNodeID, parsePeers, run, acceptLoopWithDigest, serveConnWithDigest, convergenceGaugePoller, startLivecheck, startControlPort, mintSelftestCerts, buildNodeIdentity, compactionSchedulerLoop (hardcoded 30 * time.Second interval, off the write path), runCompactionSweep, reaperLoop, runReaperSweep, engineHAMTAdapter (implements database.LiveSource; the ONLY place the concrete engine is in scope for internal/database, avoiding an import cycle). armOTel (otel.go) arms the OTel MeterProvider gated on --otel; logOutputExporter writes OTel batches to the operator log stream (NOT a Prometheus bridge — the Day-18 bridge is separate).
The client SDK: sdk/sovereign
sdk/sovereign/client.go is a <50-line mTLS control-port client. Dial(addr, tlsCfg) forces Min==Max==tls.VersionTLS13 + ForceAttemptHTTP2:false; DialWithCerts(addr, certPath, keyPath, caPath, serverName) loads on-disk credentials.
type Client struct { httpClient *http.Client; baseURL string }
type GetResult struct { ... Payload string; PayloadDigest string ... } // Ruling 3 boundary
type MetricSample struct { Name, Labels string; Value float64 }
type MetricSamples []MetricSample
func Dial(addr string, tlsCfg *tls.Config) (*Client, error)
func DialWithCerts(addr, certPath, keyPath, caPath, serverName string) (*Client, error)
func (c *Client) InsertLocal(key, val string) error // POSTs /v1/insert (routes through Gossiper.InsertLocalEvents, NEVER engine.InsertLocal)
func (c *Client) Get(key string) (GetResult, error) // GET /v1/get — Payload only on originator; digest on peers
func (c *Client) MerkleRoot() (string, error) // GET /v1/merkle
func (c *Client) Status() (NodeStatus, error)
func (c *Client) Metrics() (MetricSamples, error) // parses /metrics, label-preserving
func (c *Client) RunDemo() error
func (c *Client) Close() error
func (s MetricSamples) Value(name string) (float64, bool) // false for 0/multiple samples
func (s MetricSamples) Samples(name string) []MetricSampleHonest boundary (Ruling 3)
GetResult.Payload is non-empty only on the originator (cache hit); peers return Payload=="" + PayloadDigest!="" because the engine stores only the PayloadDigest on a joined CRDTEntry (TestClientGetOnPeerReturnsDigestNotValue). The SDK does not claim linearizability — InsertLocal returns at LOCAL-apply; peer convergence is eventual (next gossip sweep).
Examples
examples/sdk/main.go— the canonical <50-line SDK example (main,run).examples/embed/main.go— the OLDER Phase-1 lock-free stack export (ShardedStack/EliminationStack+ per-goroutineNewElimPRNG, 200k ops/worker, assertspushed - popped == drained). It proves external linearizable usability of the stack core, not the δ-CRDT HAMT engine — a historical artifact, not the current engine. RequiresGOMAXPROCS>=2.
Control surface
| Endpoint | Transport | Purpose | Status |
|---|---|---|---|
/livecheck | plain HTTP | liveness | Production |
/metrics | plain HTTP | Prometheus scrape (sovereign_* labelled + supremum_* cumulative) | Production |
/v1/insert | mTLS | local insert (→ Gossiper.InsertLocalEvents → bridge.PutLocal WAL fsync → AppendCheckpoint → L0 Arrow) | Production |
/v1/get | mTLS | single-snapshot Get (digest on peers, value on originator) | Production |
/v1/query | mTLS | bitemporal AsOf (503 if resolver nil, BEFORE param validation) | Production |
/v1/range | mTLS | bitemporal Range (interval-intersection, capped by MaxRangeRows) | Production |
/v1/merkle | mTLS | current root | Production |
handleInsert stamps a Day-12.5 open-ended ValidTime default; handleQuery/handleRange echo PayloadDigest verbatim with deliberately no payload field (Law V — the index stores a sentry body; reporting one would be the "digest-is-not-value" fabrication). parseQueryTime and dotHex are helpers; OpenEndedValidEndNs = 9e18 (year ~2253, deliberately NOT database.MaxValidTime.UnixNano() which overflowed).
TLS / cert provisioning
The dev-mesh CA (pkg/crypto/certgen.go: NewMeshCA → IssueLeaf → WriteCAPEM/Leaf.WritePEM) mints an Ed25519 self-signed CA (10-year, IsCA, CertSign|CRLSign, MaxPathLen=1) and 1-year server+client leaves (DigitalSignature, ServerAuth+ClientAuth, DNSNames {nodeID, localhost}); 62-bit crypto/rand serials; PEM to disk (CA pubkey at 0644 — CA private key stays in-process; leaf key PKCS8 at 0600). The binary's --tls-cert --tls-key --tls-ca point at these; SIGHUP → tr.Reload() rotates the leaf (CA pool NOT reloaded).
Dev mesh only
This is a DEV mesh CA, not production PKI — offline root, intermediate CAs, HSM-backed key custody, OCSP/CRL revocation, and automated rotation are named as post-launch work (ADR-0006).
Honest readiness assessment
| Capability | Readiness | Evidence |
|---|---|---|
| δ-CRDT core (Join, HAMT, IBLT, EBR) | Production | FROZEN; rapid property tests + chaos mesh convergence; 0 allocs/op hot path |
| Wire-integrity + skew bound | Production | Phase 2c/2f/2g teeth; closes Byzantine A1/A4 |
| TLS 1.3 mTLS mesh + control port | Production | TestTLSHandshake_13_Only, /v1/* route teeth |
| Admission (rate + clock) | Production | TestPeerBucket_SybilIsolation; real EAGAIN-at-TCP is Track 2.1 (not yet shipped) |
| WAL + bounded snapshot recovery | Opt-in | --wal-path; TestStage6WALRecoveryDeterminism |
| L0→L1 compaction + DominancePrune | Opt-in | --compaction-prune-enable default false; ADR-0019/0020/0025 |
| T_gc auto-inference | Production | Day-22 ADR-0027; inferrer floors operator knob; retreats refused + counted |
| L0 reaper | Opt-in | --compaction-reap-enable default false; never auto-runs |
| Read-your-writes LiveSource | Production | Day-27 ADR-0032; insert→IMMEDIATE query→200; engine.State().Get is O(total entries) |
| SkipListArena.Seek | Dormant | Proven correct (Day-23 ADR-0028) but not wired into scanWindowRecordBatch |
| eBPF SK_REUSEPORT steering | Production | //go:build ebpf_kernel (opt-out); silicon tests t.Skip cleanly on capability-absent box |
| EPOLLET Cap’n Proto ingestion | Partial | Unix-socket E2E proven; binds but no message in TestEpollServer_BasicMessage; IPv6 unsupported |
| H3 spatial CRDT (C++ worker) | Partial | SPSC ring proven structurally; C++ h3_worker operator-supplied; in-tree tests use mocks |
| Ed25519 verify + hedged signer | Production | VerifyCRDTFrame + RejectSmallOrderKey; verifies under unchanged circl.Verify |
| ML-DSA-65 post-quantum | Preview | pq_preview build tag; no production imports; 32c re-run pending |
| aws-lc hedged bridge | Stub | aws_lc_hedged_stub.go is a panic stub; real CGO bridge deferred |
| Dev-mesh x509 PKI | Dev only | Not production PKI (no offline root/intermediates/HSM/OCSP/rotation) |
| Cedar ABAC | Bench-only | pkg/authorization has no non-test .go source; no production authorizer type or call site |
| Zero-GC PII masking | Production | internal/crypto/pii.go; memtable calls MaskPII per write |
| Telemetry → Prometheus bridge | Production | ADR-0023; SSoT-grows-auto; 19 instruments; real scrape cumulative-not-delta |
| Chaos harness | Bench-only | internal/chaos; SIGSEGV survival conditionally skipped pending CHAOS_WORKER_BIN |
| pkg/durability/wal.go | Alias layer | Pure re-export of internal/chaos WAL with zero own logic |
| EpochCompactor | Dead | Zero production importers; retained for scope hygiene; real GC is DominancePrune + L0 reaper |
Additional disclosed debt
| Capability | Status | Evidence |
|---|---|---|
payloadCache | Production (unbounded) | No eviction/LRU; ADR-0007 debt |
| L0 file growth | Opt-in debt disclosed | One fresh l0/*.arrow per checkpoint, no merge/compaction by default; bounded by ResolverConfig.MaxL0Files (1000); TestQuery_L0FileGrowthDisclosed asserts the debt's existence |
| Postgres bulk load | Partial | InitializeSwarmPool + UnloggedStagingPromotion (uses a TEMPORARY table despite the name); no live COPY test — only SQL-string construction and config parsing are tested |
| S3 uploader | Partial | AWSS3Uploader with SHA-256 prefix sharding (65,536 partitions); multipart is sequential, not concurrent; no live Upload test (AWS creds assumed absent); database.S3Uploader conformance asserted only via compile-time var _ |
| Bench evidence tracks | Bench-only | pkg/codec120, pkg/durability120, pkg/pqecobench — no production importers; produce the byte-cost/latency CDFs behind GO/NO-GO gates |
Getting started — operator
- Provision a mesh CA (dev): use
pkg/crypto'sNewMeshCA/IssueLeaf/WriteCAPEM, or supply your own PKI. - Derive node identity: pass
--identity-seed;nodeIDmust equalengine.localNodeID. - Start a node (see Quickstart).
- Opt into durability/compaction once you trust the T_gc floor; add the reaper once you trust your storage layer's existence-probe.
- Observe: scrape
/metrics; hit/livecheck; arm--otel. - Write/read: via the SDK over mTLS to the control port —
InsertLocal,Get,AsOf/Range.
Getting started — developer
Embed the stack core via examples/embed (historical Phase-1 API) or drive a full node via examples/sdk (the canonical SDK path). For direct engine use, the pkg/sync public surface is:
eng, err := sync.NewDeltaCRDTEngine(nodeID, initialCounter, arenaSize)
dot := eng.InsertLocal(entityID, entry) // 0 allocs/op hot path
eng.Join(delta) // FROZEN merge-union
remoteDigest := eng.GenerateDigest() // IBLT
delta := eng.GenerateDelta(remoteDigest) // minimal missing set; defer delta.Release()
eng.ApplyCRDTDeltaEvent(wire) // wire-integrity + JoinFor durability, pkg/durability:
wal, _ := durability.OpenWAL(path)
bridge := durability.NewBridge(eng, wal, checkpointInterval)
dot, err := bridge.PutLocal(entityID, payload, entry) // InsertLocal + WAL fsync
bridge.AppendCheckpoint()
eng, wal, replayed, err := durability.RecoverEngine(nodeID, walPath, arenaSize)
eng, wal, _, witness, err := durability.RecoverEngineWithSnapshot(nodeID, walPath, store, arenaSize) // boundedFor queries, internal/database:
resolver := database.NewResolver(lister, downloader, alloc, bucket, database.DefaultResolverConfig)
row, err := resolver.AsOf(ctx, entity, validTime, txTime)
rows, ok, err := resolver.Range(ctx, entity, vLo, vHi, tx)
// LiveSource (Day-27 read-your-writes): set cfg.LiveSource to an EBR-pinned adapter