The Loop Is the Product: Inside the Sovereign Intelligence Observatory

July 3, 2026

---

Every agent framework on the market answers the same question: how do you get a model to do a task. Almost none of them answer the question that actually determines whether your system gets better over time: what happened, in what order, under what confidence, judged by whom, and is that judgment still valid six months later.

The Sovereign Intelligence Observatory is a six-component, local-first Python system built to answer that second question. It doesn't wrap an LLM. It doesn't compete with LangGraph or CrewAI for orchestration mindshare. It sits downstream of whatever agent runtime you're already using and treats every decision that runtime makes as a first-class, versioned, queryable artifact. The project's own framing is blunt about it: intelligence isn't the weights, it's the accumulated decisions that shaped them, and if the loop is the product, observability of the loop is the operating system.

This post walks through the architecture at the code level -- the drift statistics, the sandboxing model, the ledger chain, the concurrency guarantees -- and makes the case for why this pattern matters for anyone building agents they intend to keep improving rather than keep re-prompting.

The core insight: recipes, not logs

Most agent systems produce logs. Logs are append-only text optimized for a human to read once, during an incident, and then forget. The Observatory instead produces recipes: structured, versioned artifacts that capture the complete decision context of a single agent run.

jsonCopy
1{
2  "recipe_id": "recipe-20240101-120000-abc123",
3  "objective": "classify_ai_paper",
4  "model": "qwen3.5",
5  "prompt_version": 5,
6  "memory_version": 12,
7  "retrieved_docs": ["doc_1", "doc_2"],
8  "reasoning_patterns": ["compare", "retrieve", "synthesize"],
9  "evaluation": {"score": 0.95, "reviewed_by": "expert"},
10  "outcome": "accepted"
11}

The distinction matters because a log is write-once and a recipe is a row in a schema. Once your agent's behavior has a schema, it can be indexed (SQLite FTS5 full-text search), diffed across prompt or memory versions, embedded and searched semantically (optional ChromaDB), streamed out as training data, and — critically — fed back into the system that decides whether your agent is getting better or worse. The Agent Recipe Compiler is the component that does the capturing; everything downstream consumes its output. The system frames this as the missing primitive most agent stacks never build, and the framing holds up: without it, "improving the agent" means eyeballing transcripts.

Architecture: six layers, one feedback loop

textCopy
1Agent
2 |
3 v
4Recipe Compiler ----------------------------------------+
5 |                                                      |
6 v                                                      |
7Expert Signal Router                                    |
8 |                                                      |
9 v                                                      |
10Autonomous Evaluation Loop                              |
11 |                                                      |
12 v                                                      |
13Tacit Judgment Extractor                                |
14 |                                                      |
15 v                                                      |
16Sovereign Apprenticeship Engine                         |
17 |                                                      |
18 v                                                      |
19Intelligence Observatory <------------------------------+
20 |
21 v
22Intelligence Timeline -> Actionable Insights

Each layer produces the input for the next, and the Observatory at the bottom folds everything back into a timeline that determines whether the whole loop is compounding or decaying. Six components, six SQLite databases in WAL mode, one FastAPI surface per component, 176 tests across 8 suites. Let's go through them in the order data actually flows.

1. Agent Recipe Compiler — the ledger of what happened

Every run gets ingested through POST /api/recipes, indexed with SQLite FTS5, and made available for full-text and (optionally) semantic search. It supports chunked streaming JSON export specifically so recipe history can be turned into fine-tuning data later without loading the whole table into memory. This is the layer everything else is built on top of, and it's deliberately boring: SQLite, JSON, HTTP. No vector database is required to get started; ChromaDB is dependency-injected and the system falls back to FTS5 silently if it isn't installed.

2. Expert Signal Router — deciding who judges the output

Recipes tell you what happened. They don't tell you if it was any good, and worse, they don't tell you who should be bothered to find out. The router implements a tiered confidence gate:

textCopy
1Agent Output
2    |
3    v
4Confidence >= 0.95?  --YES--> Auto-accepted
5    |
6    NO
7    v
8Confidence >= 0.80?  --YES--> Cheap evaluation
9    |
10    NO
11    v
12Expert review required

The thresholds aren't fixed. A dynamic calibration matrix adjusts them per objective based on historical error rate, so a task class that keeps fooling the cheap evaluator gets escalated more aggressively over time, and one that experts keep rubber-stamping gets cheaper to clear. Every expert decision the router captures becomes a labeled training example for the next tier down — this is the mechanism that lets human judgment gradually get absorbed into the automated evaluation layer instead of staying a permanent cost center.

3. Autonomous Evaluation Loop — catching drift before it becomes an outage

This is the layer I think is most underbuilt in the rest of the agent-framework ecosystem, and it's worth showing the actual math. Evaluation signals are defined as YAML specs with uncertainty bounds, synthetic test cases are auto-generated from production traffic, and every signal is checked for drift using two independent statistics that have to agree before an alert fires.

Two-sample Kolmogorov–Smirnov D-statistic, measuring how far apart two empirical distributions have drifted:

pythonCopy
1def _kolmogorov_smirnov_statistic(sample_a, sample_b):
2    combined = sorted(set(sample_a + sample_b))
3    max_diff = 0.0
4    for val in combined:
5        cdf_a = sum(1 for x in sample_a if x <= val) / len(sample_a)
6        cdf_b = sum(1 for x in sample_b if x <= val) / len(sample_b)
7        max_diff = max(max_diff, abs(cdf_a - cdf_b))
8    return max_diff  # threshold: 0.3

Population Stability Index, measuring binned proportion shift with Laplace smoothing so empty bins don't blow up the log:

pythonCopy
1def _population_stability_index(expected, actual, n_bins=10):
2    ...
3    for i in range(n_bins):
4        p_exp = (exp_counts[i] + 0.5) / (n_exp + 0.5 * n_bins)
5        p_act = (act_counts[i] + 0.5) / (n_act + 0.5 * n_bins)
6        psi += (p_act - p_exp) * math.log(p_act / p_exp)
7    return psi  # threshold: 0.25

Requiring both KS and PSI to cross threshold before flagging drift is a deliberate design choice against false positives — KS is sensitive to shape changes, PSI is sensitive to mass movement between bins, and real capability regressions tend to show up in both. There's also a validation guard that rejects synthetic or degenerate inputs before they can pollute the signal: if the last three scores for an objective are all identical, the new score is rejected outright, since real model output has variance and a suspiciously flat signal is more likely a broken pipeline than a stable one.

4. Tacit Judgment Extractor — mining expertise nobody wrote down

This is the component that answers a question most eval frameworks don't even ask: how do you capture the knowledge an expert isn't articulating while they review outputs? The extractor records text-based expert decision sessions, runs them through a local Ollama model to surface latent patterns, and converts the result into structured decision trees with condition, action, confidence, and rationale fields on every node.

Local LLM output is untrusted by default. The parser (_parse_llm_tree_response) runs a three-phase defense before it will build a tree from what the model returned: a balanced-brace scan that truncates at the last complete structure if the stream got cut off mid-object, a JSON decode that tries both bare-object and wrapped-array framing, and a schema conformance pass that silently drops any node missing a required field rather than propagating a malformed tree downstream. If Ollama isn't running at all, a rule-based fallback extractor keeps the pipeline alive. Nothing in this system assumes the local model is reliable; every consumer of local-model output is written as if it might return garbage, because eventually it will.

5. Sovereign Apprenticeship Engine — the missing middle between manual and autonomous

Most agent frameworks have exactly two operating modes: a human approves everything, or nothing gets approved at all. The Apprenticeship Engine implements a five-rung ladder instead:

1. Fully Supervised — 100% human oversight
2. Approve Dangerous — only flagged-dangerous actions reviewed
3. Approve Novel — only never-seen-before actions reviewed
4. Approve Uncertain — only low-confidence actions reviewed
5. Fully Autonomous — no human oversight

Promotion and demotion between rungs happen automatically based on tracked outcomes, and daily action budgets are compute-weighted — a monitored action costs 1.5x a routine one, which prices in the actual cost of the human attention it consumes. Demotion triggers a rollback freeze that clears accumulated autonomy debt and resets the budget, so an agent that regresses doesn't get stuck paying down a penalty indefinitely; it gets a clean restart at a more conservative rung. There's also a circuit breaker: when the federated outbox queue backs up past 50 pending items, record_action() starts returning circuit_breaked: true instead of raising, so a downstream outage degrades gracefully instead of cascading into the apprenticeship database.

6. Intelligence Observatory — GitHub Insights, for intelligence

Everything above feeds into the flagship component, which is the one users actually look at. It aggregates recipes, routing decisions, drift alerts, extracted judgment, and autonomy transitions into a single Intelligence Timeline, pre-rolled into weekly and monthly views so the dashboard doesn't have to scan the full table on every load. It flags obsolescent prompts using recency-weighted usage combined with trend scoring, and unused memories — documents that were ingested but never once retrieved, which in most RAG systems is silent, wasted storage that nobody notices until the index is too large to reason about. It correlates cheap evaluation signals against expert-reviewed ground truth to tell you which of your cheap signals is actually worth trusting. It ships as a Chart.js v4 dashboard served over GET /dashboard and a WebSocket stream at ws://host:port/api/observatory/stream that delta-encodes broadcasts so clients aren't re-downloading the full state every five seconds — only the changed top-level keys, with a full resync snapshot every sixth cycle so late joiners don't have to guess at missed deltas.

The parts that make this production software, not a demo

A lot of local-first tooling stops at "it works on my laptop." Three details in this repo signal it was built past that point.

Every SQLite connection runs in WAL mode with tuned pragmas — journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, an 8MB page cache — applied uniformly across all six component databases through a shared AsyncDatabase wrapper, so concurrent readers never block on a writer and a contended writer waits five seconds before failing instead of raising immediately. The concurrency test suite backs this up with 50 parallel writers against a single in-memory database and deliberate BEGIN IMMEDIATE contention tests.

Local model output never gets a free pass into execution. The ActionSandbox runs every extracted decision-tree action in a forked subprocess with a stripped environment (PATH=/usr/bin:/bin and nothing else), rejects 15+ dangerous code patterns before execution — eval(, exec(, __import__, subprocess., .__class__, .__subclasses__, getattr( among them — and enforces RLIMIT_NPROC, RLIMIT_NOFILE, RLIMIT_AS, and RLIMIT_CPU at the OS level with a hard timeout on top. That's the correct posture for any system where a local LLM's output can eventually become code that runs: assume it's hostile until proven otherwise, every time, not just on first install.

Cold storage is hash-chained, not just compressed. Recipes older than 90 days get compressed into gzipped CSV and SHA-256 hashed; each archive stores the hash of the archive before it, so the whole history forms a verifiable chain from genesis forward. verify_chain() walks the chain and flags a tampered archive two independent ways — its own hash won't match, and its successor's stored previous_hash link breaks. For a system whose entire value proposition is "trust the recorded history of what your agent did," that history needs to be tamper-evident, not just tamper-resistant, and this is the right primitive for that.

Why this is the actual thesis, not a feature list

Strip away the six components and the underlying claim is simple: the model is replaceable, the loop is not. You can swap Qwen for Llama for whatever ships next quarter and your capability curve barely notices, provided the recipe history, the drift baselines, the calibration matrix, and the extracted tacit judgment survive the swap. None of that state lives in a model checkpoint. It lives in the observability layer wrapped around the loop — which is exactly the argument for building that layer as durable, versioned, local infrastructure rather than as ephemeral logging you throw away every time you change providers.

This is also why the project pairs naturally with the rest of the sovereign stack rather than standing alone. The recipe format is the same discipline behind the Sovereign Memory Bank — treat every unit of agent experience as a structured, evolving artifact instead of a transcript. SovereignSpec applies the same discipline to code generation, where specs are living graph-grounded artifacts instead of prompts you retype. SynthInt applies it to persona-routed retrieval. The Observatory is the piece that closes the loop across all of them: it's the layer that tells you, with statistics instead of vibes, whether the rest of the stack is actually getting better.

Where the book comes in

If the architecture above is interesting to you and you want the reasoning behind why it's built this way — not just what the code does, but the design tradeoffs between local and cloud inference, how to build RAG pipelines that don't silently rot, how knowledge graphs beat flat vector stores for multi-hop reasoning, how to wire MCP servers into a fully local agent stack, and how to think about RLHF-style evaluation loops when you don't have a cloud lab's evaluation budget — that's the ground Sovereign AI: Building Local-First Intelligent Systems covers end to end.

The Observatory is a direct implementation of ideas from the book: the recipe-as-artifact model, the tiered evaluation philosophy, the autonomy ladder instead of a binary supervised/unsupervised switch. If you're the kind of engineer who reads a README like this one and immediately wants to know why the KS threshold is 0.3 and not 0.2, or why PSI needs Laplace smoothing, the book is where that reasoning is written out in full rather than left as a comment in the source.

Get the book on Amazon →

Try it

bashCopy
1git clone https://github.com/kliewerdaniel/sovereign-intelligence-observatory.git
2cd sovereign-intelligence-observatory
3python3 -m venv .venv && source .venv/bin/activate
4pip install fastapi uvicorn aiosqlite pydantic httpx pytest pytest-asyncio
5./run_tests.sh   # 176 tests, all passing, 0 warnings

No cloud APIs required to run the core loop. Ollama, ChromaDB, and Weights & Biases are all optional and guarded behind import checks — the system degrades gracefully to SQLite FTS5 and rule-based extraction if none of them are present. That's the sovereignty argument made concrete: the observability layer that decides whether your agents are improving shouldn't have a dependency on someone else's API staying up.

---

Intelligence is not the model. Intelligence is the accumulated decisions that shaped the model. Recipes are Git commits for intelligence.

Related reading: Sovereign Memory Bank · The Sovereignty Manifesto

Related projects: SovereignSpec · SovereignBank · SynthInt