For three weeks I ran a pilot AI automated research program to see if I could answer one big question: what does good deliberation actually depend on? AI agents, many running in parallel and isolated from each other, worked from what the models already know about roughly a hundred historical decisions, twelve fields that study inquiry and over a hundred and twenty analytical methods. They worked from memory rather than from source texts, and nothing was checked against originals. The most valuable thing it produced was a detailed map of what we still don't know.
This lab note explains how that research was run. Every post in the Mapping Deliberation series carries two short lines on how I know what it claims, and this is the page those lines point to. It is long on purpose, and it is built to be skimmed: the diagrams and the section headers carry the structure, and each section opens with its point. If you only read one screen, read the next one.
What this is, in one screen
What the research is. A pilot of automated research: dozens of AI agents, run in parallel and kept isolated from one another, reading what their models already know about how consequential decisions are made and how they go wrong, and turning that reading into a register of claims about deliberation that could each be tested and killed.
What it was for. I'm building a deliberation calculator, a machine that maps an open-ended problem, tests what the map contains and shows a human what survives. To build it I need to know what good deliberation is made of, and I wanted that answer assembled from history, theory and method together rather than from any one of them alone.
What it covered. About a hundred historical decisions that went wrong entered the program, and 69 of them, roughly two-thirds, survived the pruning that removed cases with too thin a record and were reconstructed in full. Alongside them: twelve fields that study inquiry, over a hundred and twenty analytical methods from about fourteen traditions, and my own controlled experiments with AI critics. Three weeks, September to October 2026. The subject is how groups and systems deliberate under uncertainty, and what transfers from people to machines; deliberative democracy was out of scope.
Where it stands. The reading phase is closed. The register exists, every claim in it carries a falsifier, and none has been tested against real outcomes. A second, pre-registered program that tries to do that testing is underway and not yet published; this page will carry its results, including the ones that kill claims made here.
Why I did it. Because I had assumed the question was settled, and I wanted to find out whether the methods we trust agree with each other, what each one quietly assumes about the people running it, and which of those assumptions survive when a machine runs the process instead of a person.
What I learned. Three things, in order of how much they changed my plans. The methods contradict each other in specific, nameable places, and each contradiction hides a condition that decides which side holds, which the methods themselves rarely state. Most of them depend on something a person in the room supplies without being asked, which a machine has to be given explicitly. And generating well-specified claims has become almost free, while finding out which of them are true has not moved at all. The posts in this series are built from the specific findings; this page is built from the method.
Why a lab note and not an essay. An essay argues for something I'm prepared to defend. A lab note records what was done, what it showed and what it did not, so that the essays in the series can point here instead of carrying their own methods section. It will be revised in place as the testing comes back, which an essay should not be.
The research program from four evidence streams to a decision, ending in an empty region marked "tested against outcomes - not yet"
The question, and why I thought I already knew the answer
The program asked what good deliberation depends on, and whether a machine could run it.
I'm building what I call a deliberation calculator: a machine that maps an open-ended problem, tests what the map contains, and helps a human see what survives. To build one, I need to know what a good deliberation is made of. When I started, I assumed that part was mostly settled. Decision science has studied judgment for decades, organizations already run red teams and pre-mortems, and intelligence agencies have formal methods for weighing rival hypotheses. I even wrote an essay arguing that we know how to think better and that it's just too expensive.
That assumption was reasonable, and in my experience it's the one most experienced decision-makers hold. The methods exist, they have names, and many of them have studies behind them. Plenty of people have catalogued these methods. What I hadn't seen was one pass that sets history, theory and method side by side, asks what each method assumes about the people running it, and checks where they agree. So the program set out to do that.
Four ways of looking at the same question
Each evidence stream was chosen because it sees something the others can't, and each has a blind spot the others are meant to cover.
Stream | What it covered | What it can tell you | What it can't |
Historical decisions | About a hundred consequential decisions that went wrong, chosen to exclude obvious incompetence or bad faith (business, military, statecraft, science, engineering, expeditions) | How real decisions fail under hierarchy, stakes and time pressure | What success looks like, because every case was chosen for its bad ending |
Theories of inquiry | Twelve fields that study how questions are asked and answered (the logic of questions, philosophy of science, decision theory, the value of information, argumentation, causal reasoning and others) | What a well-formed question and an adequate answer are supposed to be | Whether any of it holds up in practice |
Analytical methods | Over a hundred and twenty methods drawn from about fourteen traditions (hypothesis and evidence handling, clinical diagnosis, group aggregation, red teaming and dissent, wargaming, safety engineering, audit and diligence, and others) | What practitioners actually do, and what each method silently assumes | How often the methods work: many have little or no outcome testing, and most of the tests that exist are lab studies or narrow settings |
My own experiments | The controlled tests of AI critics I've already written about on this site | How an automated system behaves on one test task | Anything beyond one task, one test case and one family of models |
"Deliberation" here means how a group or a system reaches a decision under uncertainty. Deliberative democracy has its own large literature, and it was not a stream in this program.
The first three streams are about people. The fourth is about machines. That split matters later, because the question of what transfers from one to the other turned out to be one of the most interesting parts of the whole program.
What each stream can and cannot tell you, with the shared gap marked
Keeping my thumb off the scale
The agents working through the evidence were never given my hypotheses, so they weren't steered toward what I was hoping they'd find - with one exception, described below.
Before the run started, I wrote down what I expected the research to show and sealed it in a file that the reading agents were never given. My own experiments were held back too. The agents working through history, theory and method were blind to my bets and blind to each other, and their instructions were written in neutral vocabulary so that my terms (the words I use for my own machine) wouldn't steer what they noticed.
The reason is simple. A researcher who knows what he wants to find usually finds it, and so does a model that has been told what the researcher wants. The sealed file only became visible late in the program, to the stages that compared what the evidence said with what I had expected and then attacked the result.
The blinding wall: what each stage could and could not see, and where it leaked
From recall to claims
The findings were synthesized twice, independently, and the two versions were merged without forcing them to agree.
Each stream's output was consolidated by mechanism: instead of a list of facts, a set of recurring patterns (for example, "a warning existed and nobody with power to act received it"). In the historical stream, the most common pattern, in 30 of the 69 reconstructed cases, was a dissent dismissed because of who raised it rather than answered on its evidence, and many failures clustered where the right warning was already inside the process but never changed the decision. Like everything here, those are counts over cases recalled from memory and chosen for their bad endings, so they are hypotheses to test, not measures of how often it happens. Then two separate agents synthesized all three human-side streams into one picture, each reading them in the opposite order, because the first thing you read shapes how you read the rest.
The two syntheses agreed on most of the core findings and disagreed in specific, useful places. One reader, for example, found support for a claim in the theory stream that the other reader searched for and couldn't find. A reconciliation step merged the two while keeping every disagreement visible, with a note on whether it was a real difference in what the two claimed or just a difference in wording. Averaging them would have been tidier, and it would also have hidden exactly the places where the evidence is soft.
Only after that did the sealed file come out. One stage confronted the merged picture with my hypotheses and my experiments. One result surprised me: on each point checked, my short write-up of my experiments claimed less, or claimed more narrowly, than the experiments had actually shown. My summaries had drifted from the record, and it was the summaries that needed correcting.
Every claim gets a way to die
A claim entered the final register only if I could say, in one plain sentence, what result would prove it wrong.
The output of the program is a register of testable claims about deliberation, written for the testing that comes next. Each one carries its scope, how strong its support is, and a falsifier. Here's one in plain form:
Claim: when two runs of the same model agree, that agreement shouldn't count as corroboration. What would prove it wrong: if repeated runs of one model miss each other's errors as often as truly independent reviewers of equal skill do.
Claims that no possible test could kill were quarantined rather than admitted. Some were too vague to fail. Others were true by construction, which sounds like strength and is actually the opposite, because a claim that can't be wrong tells you nothing. One quarantined idea stays in view as a standing baseline: the possibility that process doesn't determine outcomes at all. Every theory of good deliberation in the register has to beat that before it deserves to be believed.
Each claim also went through what I think of as a translation test. Human methods rest on premises about people (fear of the boss, fatigue, a meeting that ends at five). For every method, the program asked what happens to that premise when a machine runs the process instead. Sometimes the reason for the method vanishes while the method is still useful for a different reason, and sometimes the method quietly depends on something only a person in the room supplies. That test turned out to produce some of the most surprising findings in the series.
How sure is each claim: an evidence ladder with the top rung still empty
A reviewer whose job was to break it
An adversarial reviewer attacked the whole program, and a separate agent decided each objection on its merits.
When the register was built, a reviewer agent was given one job: find where this research is most likely to be wrong. Its objections went to a different agent, which judged each one and fixed what could be fixed inside the run. Both were AI, like every other stage, which is itself one of the limits listed below. Some objections were accepted in full, many in part, and a few were accepted as real problems that can't be solved without new evidence.
The one objection the reviewer rated fatal - fatal to anything the register says about machines - still stands, and I think it's correct. Everything the register says with confidence about how an AI system deliberates rests on one experiment of mine, on a different task (finding planted flaws in a set of documents, which is a long way from making a decision), and the published research on how AI systems deliberate together was never read as a source. That objection is why the next step is a narrow reading of exactly that literature, and why none of the posts in this series claims a machine result beyond the one experiment it came from.
The leak I didn't plan for
The agents I had sealed off from my hypotheses could see a description of my own system, and later my own notes, through the environment they ran in.
I found this in the middle of the run. A routine vocabulary scan, which checked the agents' outputs for my private terms, came back with hits in one stream. Tracing them showed that the working environment had been exposing a description of my own system to the agents, and later my own notes. Nothing in the agents' instructions mentioned any of it. The blinding failed through the surroundings rather than through the prompt.
I switched those sources off, checked that the clean environment really was clean, and re-ran the affected work. Then a blind audit compared original outputs with clean re-runs of the same material, judged by an agent that didn't know which was which. At the level of individual readings it found no framing effect beyond the ordinary noise between runs. One consolidated summary did carry noticeably more of my framing, and it was replaced by its clean version before anything downstream read it. I've learned to read that carefully. "Not detected" is a statement about the audit, and it isn't a certificate that the work is clean, because a small effect, or an effect on the direction of the claims rather than their framing, could sit below what the audit could see. So every finding that points the way the leaked material pointed is tagged, and those are the ones to test first.
Timeline of the exposure: from first leak to scan, clean re-run and audit
How it closed
I closed the program instead of digging deeper, because the bottleneck had moved from ideas to evaluation.
Before closing, I had the program write down - before any test results exist - what would have to be true to justify another, deeper round of research, and I weighed the program's own list of limitations. More reading of the same kind would add more claims of the same kind, recalled by the same family of models, mostly from decisions that ended badly. What the register needs now is testing. So the research phase closes with one narrow reading pass on the published work about AI deliberation, and after that the work moves to building instruments that can test the claims.
There's something I find genuinely strange in that outcome. A system that can generate dozens of carefully specified, falsifiable claims about deliberation in two weeks has made the generating part almost free. The hard part, the part no amount of recall can do, is finding out which of them are true.
What this did not show
This research mapped the territory carefully, and it proved nothing yet. I'd rather say that plainly than let the scale of the program imply more than it earned.
- Nothing was tested against outcomes. No claim in the register has been checked against real decisions and how they turned out. Every claim is a hypothesis with a test attached.
- The sources were recalled from memory. The agents worked from what one family of AI models remembers about history, theory and method. Not one cited study, case fact or figure was verified against its original text. Directionally I expect most of it to hold, and individually any of it could be wrong.
- The cases were chosen because they ended badly. There is no matched set of similar decisions that went well, so no warning sign from the historical stream has been shown to separate good decisions from bad ones. Post No. 2 is about why that matters.
- The machine evidence is one experiment on a different task. It studied finding flaws in documents, graded mostly within the same family of models it was testing.
- Agreement between streams counts for less than it looks. The streams share literatures, the same stock stories turn up in several of them, and they were all read by models from one family. Where three streams agree, that's often closer to two independent witnesses than three.
- One draw of each consolidation. When some of the work was re-run after the leak, the groupings shifted noticeably. The findings that survived both versions are the ones I lean on, and I haven't yet measured how stable the rest are.
- The leak can't be fully ruled out. As described above, "not detected" isn't "clean".
- My own expert read never entered the record. The program had checkpoints built for exactly this, three "attention maps" sent to me during the run: each one named the few questions where my hands-on read would matter most, with an estimate of the minutes each would take, and the program then carried on without waiting for an answer. I answered none of them, so every one of those questions was settled on the program's default reading. That may have been a mistake, and it is one I keep coming back to.
A second program, pre-registered before its first call and designed around the flaws listed above, is underway. Its main job is the one this program could not do: set decisions that went wrong beside matched decisions that went well, with the outcomes hidden from the readers, and see which of the register's claims separate the two. It is not yet published. When it is, this page will say what it found, including what it killed.
How to read each post in this series
Every post makes one counter-intuitive claim about deliberation and carries a short "How I know" note (in the carousels, a concession slide).
The note answers four questions in two or three sentences:
- Which streams pointed the same way. History, theory, method, or my experiments, and how many of them.
- How independent they were. Whether they share a literature or a source, in which case I count them as fewer.
- What would prove it wrong. The falsifier, in plain words.
- Where it stands. Every claim in this series is untested against outcomes unless the post says otherwise.
If a post seems to claim more than its note supports, the note wins, and I'd like to hear about it.
The series on one map: where deliberation quality might live, with the region nobody has examined
The series
- No. 0 How the research behind this series was run (this note)
- No. 1 We don't actually know how to think better
- No. 2 Disasters alone can't tell you what good judgment looks like
- No. 3 Ten advisors who agree may be one opinion
- No. 4 The most dangerous flaw is the one nobody wrote down
- No. 5 Why "only ask what could change the decision" fails
- No. 6 The end of the meeting was doing the deciding
- No. 7 A review that can't stop anything is theater
- No. 8 Set the bar before you see the result
- No. 9 Thinking is getting cheap while checking stays expensive
Published so far: No. 0, No. 1 and No. 2. The rest follow over the coming weeks. This page updates in place as posts publish and as the testing produces results, including results that kill claims made here.
The question that lingers
Most of the claims in the register can be tested on problems with planted answers, where you know in advance what a good deliberation should find. The decisions that matter most don't work like that. Their outcomes arrive years later, mixed with luck, or never arrive in a form anyone can score. So the question I'm carrying into the next phase is this: how do you test a theory of good deliberation against decisions whose answers never come back?