This article accompanies our new paper, “A Measurably Self-Improving Multi-Agent Forecasting System: Five Feedback Loops, No Retraining”: https://github.com/Abilityai/trinity/blob/88effe17c8a186efed887d140ca48a8dc82d26c4/research/closed-loop-forecasting/closed-loop-forecasting-2026-07-11.pdf, co-authored with Dmytro Filatov.
The fleet’s live record - every prediction, resolution, and Brier score - is public at https://brierhq.ai.
Prefer to watch? I walk through the whole system - the loops, the hidden-state store, and the live dashboard - in a 11-minute video.
---
TL;DR
- We deployed a system of seven LLM forecasting agents, a meta-analyst, and a knowledge-base agent that improves through five closed feedback loops - with zero retraining. It has been running continuously since February 2026: 96,161 logged predictions, 73,552 resolved against real-world outcomes.
- The headline finding: 93.5-99.7% of prediction error is noise (inconsistency), not bias. The standard LLM calibration toolkit targets bias - the minority component. Measure the decomposition before you tune anything.
- Feedback works where it is absorbed: three of five compared agents improved significantly on both Brier score and surprise rate. Two moved *against* their feedback. We report the split rather than hide it - absorption is the open problem.
- The system’s most unusual object is a store of 56,347 empirically scored hypotheses about the hidden state of the world. A hypothesis that stops winning is not deleted - it fires a regime-change alert.
- A fifth loop generates *what to forecast* through abduction from anomalies, because a scoring engine has a structural blind spot: it can only improve forecasts it was asked to make.
The Premise: What Can Be Measured Can Be Engineered
Forecasting has a public image as an unmeasurable art. It is the opposite. A probability judgment can be scored against the outcome it predicted (the Brier score - mean squared error between your stated probability and what happened, lower is better). The error can be decomposed into systematic bias and noise. And the central lesson of Tetlock’s superforecasting research is that forecasters who keep score improve.
That measurability is the entire premise of the paper: once forecasting error is measured and decomposed, each component becomes an engineering target. You can build feedback loops around a running system the same way you build them around a thermostat.
Most LLM forecasting deployments do not do this. A model is prompted, generates a probability, the result is logged. If the system improves at all, it improves through retraining - new weights, new evaluation, new deployment cycle. We built the opposite: a system where the improvement happens in the loops around the models, while the models stay frozen.
Here is what five months of production data taught us.
The System: Seven Forecasters, an Analyst, and a Librarian
The architecture has three agent types:
1. The Oracle fleet- seven LLM forecasting agents, each specialized in a domain: financial markets, AI and technology, geopolitics, energy, space and shipping, semiconductors, and biotech. Each generates 10-25 predictions every four hours and resolves its backlog against tiered web sources every two. Every prediction records a statement, a probability, a time horizon, and the full chain of reasoning.
2. Cleon, the meta-analyst - measures everything. Brier scores, calibration error, surprise rates, and a full noise decomposition per agent. It coaches each Oracle with calibration feedback, routes incoming questions to the agent with the best domain track record, and maintains the hidden-state hypothesis store.
3. Cornelius, the knowledge base - a Zettelkasten-style graph with semantic search. When a prediction fails badly, Cleon sends the failure to Cornelius, which extracts a durable lesson. Before any Oracle predicts anything, it queries Cornelius for relevant lessons first.
![The architecture: Cleon (meta-analyst), the seven-agent Oracle fleet, and Cornelius (knowledge base), connected by five feedback loops.]
The scale, as of the paper’s data snapshot: 96,161 logged predictions, 73,552 resolved outcomes, all binary, all scored. The whole fleet runs on [Trinity](https://github.com/Abilityai/trinity), our open-source runtime for autonomous agents, and an open-source template of the knowledge-base agent is [also public](https://github.com/Abilityai/cornelius).
The important design property: every feedback signal in the system is grounded in resolved external outcomes. Nothing improves because a model judged its own output to be good. This matters because the research literature has documented exactly how self-judged improvement loops fail - models reward-hack their own evaluations and resist correcting confident mistakes. Reality is the only referee that cannot be gamed.
The Finding That Should Change How You Tune LLM Agents
Here is the result I keep coming back to.
Take Kahneman’s error decomposition from Noise - the distinction between bias (systematic error in one direction: always too confident, always too optimistic) and noise (inconsistency: the same judge giving different answers to equivalent problems). Apply it to seven LLM forecasting agents across domain-by-horizon cells, over tens of thousands of predictions.
The result is consistent across every single agent: noise accounts for 93.5-99.7% of prediction error. Systematic bias never exceeds 6.5%.
![Error decomposition per agent: the systematic-bias share (blue) never exceeds 6.5%. Noise dominates in every single agent.]
Think about what that means. Nearly the entire LLM calibration toolkit - bucket recalibration, temperature scaling, confidence-elicitation tuning - targets the bias component. Applied to a noise-dominated system, it is aimed at, at most, a few percent of the problem.
This is not a quirk of our fleet. Independent benchmarking found the same phenomenon: the same model, asked the same forecasting question in different runs, can answer below 20% and above 80%. Roughly 60% of the variance comes from the final judgment step, not from evidence gathering.
So the system routes accordingly. We built a noise-dominance gate into the calibration loop:
- If noise dominates (currently: every agent) - the feedback is process consistency guidance: structured reasoning templates for high-noise question types, temporal drift alerts, and per-category adjustments instead of global nudges.
- Only when bias is a meaningful share does the feedback become the classic “adjust your probabilities in bucket X” correction.
The per-category point deserves emphasis. Because bias direction varies by domain and horizon, telling an agent “you’re overconfident” globally helps some question types while actively hurting others. Noise-aware feedback has to be targeted or it becomes noise itself.
The practical takeaway for anyone building LLM agent systems: measure the error decomposition first, then pick the intervention. The default toolkit assumes a bias problem you probably do not have.
The Five Loops
Each loop targets a distinct source of prediction error. Four are corrective - they run downstream of a prediction. One is generative, and I will get to it last, because it exists to fix a blind spot the other four cannot see.
| # | Loop | Error source it corrects |
| 1 | Noise-adaptive calibration feedback | Inconsistency and conditional bias |
| 2 | Surprise-driven insight extraction | Wrong mental models |
| 3 | Knowledge retrieval before prediction | Repeated cross-agent mistakes |
| 4 | Hidden-state hypothesis injection | Ungrounded assumptions about the present |
| 0 | Abductive synthesis | Questions nobody thought to ask |
Loop 1 is the noise-gated coaching described above.
Loop 2 fires when a prediction fails hard - when the gap between stated probability and outcome exceeds 0.40. Cleon sends the full context to Cornelius with a structured prompt that asks what assumption about the world failed, not which agent was wrong. A real example from the store: “OPEC+ can execute emergency output changes faster than the assumed 3-month lead time. The model’s supply-side response lag is systematically wrong.”
Loop 3 closes the circuit: before generating any probability, every Oracle queries Cornelius - “what do we know about this topic?” - and reads the distilled lessons from every prior failure across the whole fleet. One agent’s mistake about OPEC+ supply response becomes another agent’s safeguard on LNG shipping routes. The fleet learns collectively without sharing any state directly - the knowledge base is the medium.
Loop 4 is the strangest and, I think, the most valuable. Every probabilistic prediction is implicitly a bet on the current, unobservable state of reality. “Oil exceeds $100 in Q3 at 75%” silently assumes specific hidden conditions hold right now. So the system makes those assumptions explicit: after each resolution, it extracts the hidden-state hypotheses the reasoning relied on, deduplicates them into a store, and scores each one by the win rate of the predictions that used it. Before each new prediction, the relevant hypotheses - with their empirical track records - are injected into the Oracle’s context.
![The assumption store: status distribution, uses per hypothesis, and the win-rate gradient across lifecycle tiers.]
The store currently holds 56,347 hypotheses, and the validation is clean: hypotheses graduate through a lifecycle (emerging, strong, fading) based on recency-weighted win rates, and the win-rate gradient across tiers is monotone - strong 75.0% > emerging 68.2% > fading 59.0%. The scoring separates reliable world-context from stale world-context.
The best part is what happens when a hypothesis stops working. It is not deleted. A hypothesis that scored well for months and then starts losing fires a regime-change alert - the world probably changed, and that is actionable information in itself. A live example: “FDA reviewers have not identified any new safety signal in the IV nimodipine formulation” was reliable context for 117 predictions, then its recent confidence collapsed to 0.22. That collapse is the signal.
The Evidence: An Honestly Split Fleet
Does the feedback actually change behavior? The honest answer is: yes, where it is absorbed - and absorption is not guaranteed.
Since calibration feedback began in March 2026, three of five compared agents improved on both Brier score and surprise rate, with 95% bootstrap confidence intervals excluding zero on both metrics. Fleet-wide surprise dropped from 48.7% to 44.4%.
The cleanest causal chain is the biotech agent. Told repeatedly it was about 11 points underconfident, it raised its mean confidence by 7.7 points - with every biotech sub-domain moving in the instructed direction - cut its surprise rate from 53.4% to 41.5%, and improved its Brier score while making the shift. Feedback direction matched on every axis. That is the archetype the loop is designed to produce.
![Loop 1 before/after evidence with 95% bootstrap confidence intervals. Three agents improve on both metrics; two regress against their feedback.]
And then there is the counter-signal: two agents - markets and semiconductors - were told to raise confidence and did the opposite, and their surprise rates rose. The loop delivers advisory text at prediction time; nothing forces a model to follow it. Two of seven agents moving against direction-explicit advice is a limitation we report in the paper rather than bury, and it defines the next engineering problem: when should a global advisory become a per-domain correction, and when does an agent need a different mechanism entirely?
We also refused to grade our own homework on accuracy. Raw Brier scores flatter agents on lopsided question mixes, so we anchored everything against baselines computed on *exactly the same questions*: the fleet scores Brier 0.194 against 0.250 for always-guessing-50% and 0.217 for an oracle-advantaged base-rate forecaster - a skill score of +0.105 with a tight confidence interval. Five of seven agents beat their own base rate; one ties it; one falls below it despite a mid-fleet raw Brier, which is precisely why baseline anchoring matters. For context (different question sets, so not head-to-head): superforecasters score 0.096, human crowds 0.149-0.19, and current LLM systems cluster around 0.12-0.21. The fleet sits in the LLM band - real but bounded skill, stated plainly.
![Left: each agent against the base-rate baseline on its own questions. Right: published Brier scores from the literature - context, not a head-to-head comparison.]
Loop 0: The Questions Nobody Asked
The four corrective loops share a structural blind spot that took us months to see: they can only improve forecasts the system was asked to make.
Every hypothesis in the store was extracted from predictions the Oracles already made. The store is a mirror of what we chose to ask - not a searchlight on the world. A regime shift nobody is forecasting never enters the system at all. We confirmed this with a blind test: we queried the store for the four pillars of a circulating de-dollarization thesis - central banks quietly rotating into gold and building parallel settlement rails - and got back nothing but unrelated noise. The system had no opinion, because no prediction had ever pointed there.
The missing operation is abduction - inferring the non-obvious present state that would explain a cluster of surface anomalies, then deriving what else would be true if it held.
So Loop 0 runs upstream of everything: scan for cross-domain anomalies the current model does not explain, generate candidate hidden-state theses (each paired with its competing “official” explanation), derive at least three falsifiable consequences per thesis - “if this is true, we would also see X, Y, Z by date H” - and then commission those consequences as real predictions through the same resolution machinery as everything else. A thesis becomes a bundle of falsifiable forecasts with a scoreboard.
This is the part of the system I find philosophically satisfying. A human macro commentator produces bold, non-consensus theses but never has to settle them - no calibration, no scorecard, and the wrong ones fade from memory. Loop 0 generates theses just as bold, then kills the wrong ones with data in weeks.
Status, honestly reported: the loop is newly activated. Three theses are live - on the Hormuz shipping regime, US economic-security policy, and rates-driven energy tightness - with all ten derived consequences commissioned as real predictions. Zero have resolved; the earliest horizons land in September 2026. As of the paper, the generative loop is an architectural claim with its first report card pending. We publish the resolution dates so anyone can check.
What This Means If You Are Building Agent Systems
Four principles fall out of this work, and none of them are specific to forecasting:
1. Decompose error before tuning anything. If your agents’ error is 95% inconsistency, calibration nudges and prompt-level bias fixes are aimed at the wrong component. Measure first; the intervention follows from the decomposition, not from the default toolkit.
2. Ground every feedback signal in external outcomes. Self-judged improvement loops reward-hack. Resolved reality - a prediction that came true or did not, a test that passed or failed, a customer who converted or churned - is the only feedback source that cannot be flattered.
3. Expect absorption failure and measure it. Delivering feedback is not the same as changing behavior. Two of our seven agents moved against explicit advice. If you do not measure before/after per agent, you will assume your feedback works everywhere it is merely delivered.
4. Correction and generation are different layers. A system that only corrects what it was asked will be precisely wrong about the questions that matter most - the ones nobody asked. Budget for the generative layer, and force its output through the same scoring machinery as everything else.
The deeper pattern: the improvement lives in the system around the models, not in the weights. Model capability was held fixed for the entire five months - every gain came from measurement, routing, memory, and feedback structure. When the next model generation arrives, the loops transfer; the architecture is the asset.
The paper reports all of it - the architecture, the noise decomposition, the split-fleet evidence with confidence intervals, the hypothesis-store validation, and eight limitations stated without cosmetics (https://github.com/Abilityai/trinity/blob/main/research/closed-loop-forecasting/closed-loop-forecasting-2026-07-11.pdf), or watch the 10-minute walkthrough.
If you are building feedback loops around LLM systems - or have hit the absorption problem yourself - I want to hear about it. Reply or leave a comment: what does your system do when an agent ignores its own coaching?
Further Reading
- Philip Tetlock and Dan Gardner, *Superforecasting* - the keep-score-and-improve tradition this system industrializes
- Daniel Kahneman, Olivier Sibony, and Cass Sunstein, *Noise* - the bias/noise decomposition applied throughout
- Halawi et al., “Approaching Human-Level Forecasting with Language Models” - the retrieval-reason-aggregate LLM forecasting baseline
- Karger et al., “ForecastBench” - the contamination-free benchmark where superforecasters still beat the best LLM
- FutureSearch, “Bench to the Future” - independent replication of noise dominance in LLM forecasting
Related concepts:
- [[Noise Dominates LLM-Agent Error So Calibration Tuning Targets the Minority Component]]
- [[Judgment quality is measurable through decision bias and noise]]
- [[Corrective Loops Only Improve What They Were Asked - Generation Is a Distinct Layer From Correction]]
- [[Fleet Learning Without State Sharing - One Agent’s Failure Becomes Every Agent’s Safeguard]]
- [[Good Agents Optimise the Process Not the Outcome - and Learn Only Where Feedback Generalises]]
- [[Decision-Making process needs to be optimised to improve non-verifiable judgments]]