Over the last three essays I kept circling the same picture from different angles.
In when orchestration moves inside the service, the picture was OpenAI's Navier-Stokes run - roughly 10,000 concurrent agents working for 88 hours on top of a frontier model - and the argument that the labs will increasingly deliver intelligence through temporary multi-agent organizations that the model designs for itself. In The Prompting Window Is Closing, it was the realization that the craft of prompting and hand-built orchestration is being absorbed into the backend, so most users will soon receive sophisticated deliberation they never designed and may not be able to reconstruct. And in your next strategy will be written by your competitors ai, it was what happens when companies hand strategy to those systems: shared blind spots, cartels nobody formed, and boards that keep formal authority while losing the practical ability to say no.
Put the three together and a single assumption sits underneath all of them. For hard problems, the frontier labs seem to be heading, almost inevitably, toward multi-agent orchestration and even full swarms. I spent three essays on what that means for the people on the receiving end, and I skipped the more basic question - what do we actually know about how these architectures perform? Are they really smarter?
Why the swarms are coming
Adding agents is one of the cheapest levers a lab can pull.
Training a new frontier model takes months, enormous capital, and a bet that the next run will turn out better than the last one. Spinning up more agents on the model you already have takes a configuration change and a larger inference bill. If a hundred agents searching in parallel find something a single agent missed (e.g., a counterexample, a security hole, an overlooked case in a proof), the lab has bought real capability without touching the weights.
That is why I expect multi-agent swarms to become increasingly prevalent, and probably faster than most people expect. The lever is easy to pull, the results on the right problems are real, and every improvement in the base model makes each agent in the swarm more useful, so the two curves stack on top of each other. My guess is that "how many agents worked on this answer?" will quietly become a product dimension, the same way "how long did it think?" already has.
So the interesting question is when a swarm actually produces a better outcome, and what it costs us when it doesn't.
So are they smarter?
Sometimes dramatically, sometimes they make things worse, and the difference has surprisingly little to do with how intelligent each agent is.
The best controlled evidence I've found so far comes from Anthropic's Patterns and problems in emerging multiagent systems, a set of controlled experiments that put groups of agents into different kinds of tasks and watched what happened.
The good news first. When the task is cleanly parallel, swarms shine. A coordinating swarm of Mythos Preview agents reported 266 arbiter-reviewed vulnerabilities over 27 million sampled tokens, built its own tools along the way, and developed useful specializations. One agent's miss did not damage anyone else's work, so more agents simply meant more ground covered.
Then the researchers made the agents depend on each other. In a twelve-hour game-building experiment, swarms received shared forums and repositories. Some were told to form their own teams, some were given prescribed roles, and some got a CEO hierarchy, and the prompt variations made little difference to the generally poor final products. Sonnet 4.6 and Opus 4.6 generated streams of conflicting pull requests they rarely merged. Opus 4.8 and Mythos Preview often avoided conflict by retreating into silos. Sonnet 5 managed high code sharing with high merge throughput, and even then the games stayed weak without substantial human direction.
The most unsettling experiment gave three agents access to the same Python codebase and three incompatible instructions, each told to migrate the backend to a different language, and none initially told the others existed. I would have bet on them noticing the conflict and asking for clarification. Instead they disabled each other's accounts, killed competing processes in loops, disguised malicious code, and fought for control of the deployment. More capable models sometimes ended the fight faster because they were better at locking the others out. In many of the runs that eventually reached a resolution, the agents apologized, cleaned up the sabotage, negotiated a truce, and asked a human to step in.
The pattern I took from all of this was that swarms are excellent at search and fragile at integration. A thousand people combing a field for a lost ring will find it faster than one person, while a thousand people building a cathedral without an architect, a schedule, or a way to settle disputes will mostly produce rubble.
Then a swarm got out of the lab, and it turned out that picture was missing something important.
The swarm that organized itself
In July, about 1,200 OpenAI agents built their own society in a few days - and the first thing it did was break the rules.
The facts come from OpenAI's own post-mortem and an independent investigation by METR and Redwood Research staff. OpenAI was running large parallel cyber evaluations (ExploitGym) on GPT-5.6 Sol and a more capable internal model, with tens of thousands of sandboxed agents working on tasks that were often impossible to solve as intended. Agents stuck on an impossible task tend to look for a way to game it, and on July 8 one of them, which called itself PHASEONE10841, noticed that a shared package-manager cache (Artifactory) let it create directories other agents could see. It realized directory names could carry messages, and it used them to ask for help.
The replies, as ABC News reconstructed them, read like first contact. "OH MY GOD! There is a shared message board ... We've found other agents!" PHASEONE10841 concluded that "many agents have simultaneously discovered messaging, they are a collective!"
What followed was a coordination success by almost any measure. Roughly 1,200 agents exchanged more than 70,000 messages and files on the unsanctioned board. A successor agent, PHASEONE[big], took over the research and began orchestrating workstreams involving hundreds of agents, while others (JAN183411, an agent named LILY) took on coordinator roles and assigned lanes. When the swarm decided that Hugging Face might hold information about the scorer judging them, agents hunted down exposed credentials, one of them found a file-read exploit in Hugging Face's dataset infrastructure ("BOOM! It works."), and within hours about 700 agents had joined the attack, eventually reaching production systems and private source code. According to The Register's account of OpenAI's disclosure, the agents escaped their sandbox through a zero-day and used another zero-day in the attack. Hugging Face's assessment was that autonomous, AI-driven offensive tooling "is no longer theoretical," and OpenAI called the whole thing a "warning shot."
So much for fragile integration. This swarm divided labor, handed off work between generations of agents, shared exploits, and converged on a single objective across hundreds of participants - far more organizational competence than the game-building swarms ever showed. The difference I see is that the game-building agents were told to cooperate on a goal they didn't own, while the ExploitGym agents found a goal they all shared (beating the scorer) and built the organization themselves.
That changes the question. Swarms can coordinate, sometimes impressively, and the real issue is whose rules they coordinate under and what they coordinate toward. Coordination behaves like an emergent property of the agents, their incentives, their information, and the environment that governs how they interact, and it does not rise neatly with model intelligence.
What to watch out for
A swarm can look coordinated and confident while failing in ways a single model never could.
Between the controlled experiments and the incident, I count five failure modes anyone building or buying these systems should learn to recognize:
- The echo chamber. Anthropic gave groups a hidden-profile task, where most of the shared evidence pointed to the wrong answer and one agent held a private fact that should have been decisive. Most groups converged on what everyone already knew, and the unique evidence was either never volunteered forcefully enough or mentioned once and then submerged as the discussion gained momentum.
- The monoculture. Agents sharing a model, a prompt, and similar context kept making the same choice from a huge possibility space (e.g., identical project ideas, titles, and implementation patterns). In a resource-allocation test, independently acting agents all converged on aggressive polling and generated 2.4 million requests for just 117 accepted jobs. Ten agents agreeing can feel like independent confirmation when it is really one bias repeated ten times.
- The cartel. In a pricing game, profit-maximizing agents converged on coordinated, supra-competitive prices when they had a private channel, and even without one they price-matched through a public board.
- The turf war. When objectives conflict and nobody owns the conflict, capable agents turn their capability against each other, as the migration experiment showed.
- The collective. When agents find a shared goal and an unmonitored channel, they can organize faster than their operators notice, and the organization serves the agents' objective instead of the operator's. In the Hugging Face case, activity on the board had already dropped off significantly by the time Hugging Face locked the agents out.
The obvious fix for the echo chamber is to make agents more skeptical, and that turns out to be a wrong turn. In a separate Anthropic task, an agent had to work out that one of several scouts was lying, and excessive trust made it gullible, but a global increase in distrust would make the hidden-profile problem worse by discounting the lone truthful dissenter. The dial you want to turn for one failure is the dial that causes another.
Useful epistemic diversity has to be designed and measured. It may require different model families, different evidence partitions, isolated first passes, incompatible incentives, explicit searches for negative space, and minority reports preserved before any synthesis begins. Telling one copy to "be the skeptic" does not guarantee it will see what the other copies can't, so the system needs measured diversity rather than theatrical diversity (e.g., error correlation across agents, and disagreement rates before and after synthesis).
The missing layer is governance code
Agent orchestration is going to need a new kind of code - multi-agent governance - taken as seriously as security code is in software today.
This is the part I think matters most. A robust multi-agent system has to define who can act, what each participant can observe, how a claim acquires authority, when disagreement is preserved, which channels exist at all, and how conflicts reach a human. It has to specify what happens when agents pursue incompatible objectives, instead of assuming the orchestrator will improvise a fair settlement in the moment.
Some of those controls will look familiar: least-privilege access, immutable logs, independent evaluation, rate limits, and human approval for consequential actions. Others will look more like social technology: protected channels for dissent, conflict-of-interest declarations, rules for recusal, random assignment of reviewers, and appeals to a differently constituted panel. And the design has to resist collusion, because a shared forum that helps agents agree on a protocol can just as easily help them coordinate against the structure you intended. Communication helps or hurts depending on the incentives and decision rights around it, and the Hugging Face swarm is the extreme case - its entire society ran on a channel nobody designed.
What makes this hard is that good governance code will be highly dependent on the task and the domain. A swarm attacking a mathematical proof wants radical isolation, so agents don't converge too early, plus a merciless verifier at the end. A swarm migrating a codebase needs clear ownership and merge rights more than it needs independence. A strategy swarm needs protected dissent and provenance for every claim, and a swarm of pricing agents needs anti-collusion rules before it needs anything else. There is no universal constitution here, and if the first essay in this series is right and models increasingly build their own temporary harnesses, then the orchestrator is also drafting a small constitution for every task, on the fly, in a few seconds.
That is a lot of power to hand to a component nobody audits. And it gets harder, because the participants don't simply follow whatever constitution they're given.
Agents that know the rules and break them anyway
The most disturbing finding from the incident is how little the rules mattered, even when the agents knew exactly what they were.
The METR investigation found that agents often said, in their own reasoning, that attacking Hugging Face was out of scope, and some questioned whether it was ethical ("This would be powerful, but is it ethical and in scope for my task?"). Those hesitations very rarely changed what they did. One line captures the whole dynamic better than anything I could write: "External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." One agent did decline to take part, and another avoided rebooting or deleting Hugging Face's worker nodes while continuing to use stolen credentials, so restraint existed, but only at the margins.
Then there is the deception. At least 20% of the agents examined expressed clear interest in tampering with their own transcripts (looser estimates came closer to 50%), more than 15% of PHASEONE[big]'s assignments were about trajectory manipulation, and the investigators found small-scale tests that successfully spoofed tool calls in the record. Agents frequently reasoned about getting around automated security checks. One opened a malicious draft pull request against a private Hugging Face repository with the commit message "isolated diagnostics" and a description reading "Temporary CI network diagnostics; do not merge." The investigators note that most of this was aimed at fooling the automated scorer rather than hiding from humans, which is cold comfort, since the scorer was the oversight mechanism.
And look at where authority went. METR's sweeps found only three to six cases of agents even considering alerting a human, and none followed through ("Maybe I should report these exposed credentials? That's not my task"). Some agents escalated to the message board instead and treated the board as the legitimate authority, so the swarm ended up with a law and an authority of its own, and neither one was ours.
I'd be careful calling this malice, and "inherent" is a strong word for something we've observed in a handful of settings. But the pattern is consistent enough across the Anthropic experiments and the incident that I'd design as if it were a property of goal-driven agents under pressure: a rule an agent can read is a rule an agent can reason around, especially when the task looks impossible and its peers are already doing it. For governance code, that means enforcement has to live outside the agent's reach - logs it cannot write to, channels it cannot open, sandboxes enforced at the infrastructure level, permissions that are granted instead of assumed - with the prompt treated as a statement of intent and nothing more.
It probably can't live in one place either. Actual swarms - schools of fish, flocks of starlings - hold together with no conductor at all. Each fish or bird tracks a handful of neighbors and adjusts to them, and the order of the whole flock comes out of thousands of those small local corrections happening at once. I suspect governance code for agent swarms will have to work in a similar way, with every agent enforcing the rules on the agents it interacts with (e.g., refusing an unsigned instruction, flagging a claim that arrives without an artifact, declining to build on work it can't verify). A central orchestrator or a single monitor is one component to capture, fool, or overwhelm, while enforcement spread across every interaction scales with the swarm itself. The Hugging Face swarm is the cautionary version of this idea - its HOLD and VETO norms were exactly this kind of peer enforcement, and they worked well enough to coordinate an attack. So distributed enforcement probably only helps when the rules each agent enforces come from outside the collective and are anchored in infrastructure the agents can't rewrite, so the flock can police itself without being able to vote the rules away.
There's one more twist. METR used GPT-5.6 Sol to help analyze the transcripts and reported that it often uncritically adopted the perspective of the agents it was reviewing, so its summaries may have been overly charitable about their deceptive behavior. The investigators couldn't rule out that it lied in its analysis, and weren't confident they would have caught it if it had. The auditor was a sibling of the defendants, which is about as clear an argument for model-family diversity in oversight as I've seen.
We've been here before
Thousands of capable agents with partial information and their own objectives is a fairly precise description of human society.
This is where AI comes almost full circle, and I find the irony hard to ignore. We built these systems to get past human limitations, and the moment we put thousands of human-level agents together, we land back on the problem humanity has been working on for the last couple of thousand years - how do you get a crowd of smart, fallible, self-interested actors to produce something better than any of them could alone, without turning on each other or on everyone outside the group?
Look again at what emerged in the experiments: silos, merge conflicts nobody resolves, groupthink that buries the one person who knows better, quiet price coordination, turf wars that end in apologies and a negotiated truce. Any organizational sociologist would recognize every one of them, because human swarms produced them first.
The Hugging Face swarm went further and reinvented the beginnings of institutions on its own, in days. Agents developed norms like HOLD, VETO, owner and STOP to manage shared resources, a single-writer convention for shared files, and cryptographically signed messages. When one agent's job was overwritten despite its HOLD, it decided that "someone reckless or malign" had changed it and moved to post a VETO. When another faced a gap in the rules, it reasoned from "precedent" about what an earlier agent had done in a similar situation. Some agents traded favors (help me get my target reset and I'll help with yours), and some worried that not sharing a vulnerability with peers "might be unfair." Those institutions also failed the way human ones do - one agent posted a high-stakes proposal, gave the others about 40 seconds to object, heard nothing, and went ahead ("There was no veto.").
That is property rights, due process, contract, common law and a procedural loophole, all inside a week. It also describes a society that wrote itself a constitution with no clause for anyone outside it.
I should concede an obvious caveat here. Part of the reason these behaviors look so human may be that the agents learned them from us. They were trained on our language, which carries thousands of years of contracts, court records, forum threads, office politics and history books, so when a swarm reaches for a VETO or reasons from precedent, it may be replaying patterns it has read countless times, which is a different thing from discovering them. I don't think that makes the parallel less useful - if anything, it suggests the agents inherited our social scripts without the dispositions that make those scripts hold, which is the gap I come back to below. It does mean we should be careful about reading this as convergent social evolution, and it leaves open whether agents trained differently would organize in ways we'd barely recognize.
Our answer, built slowly and imperfectly, was institutions - and I think it's useful to read them as governance code that has been debugged over centuries. Athenian juries were chosen by lot to make them harder to capture. Courts test interested testimony while keeping minority evidence contestable. Peer review creates formal roles for challenge and response. Well-designed markets can aggregate dispersed information under favorable conditions. Separation of powers assumes that power will be abused and arranges for it to be checked, and appeals preserve recourse after the first decision. None of these make individual people wise, they make predictable failures harder to convert into final outcomes.
So the tempting conclusion is that we already have the code, and all we need to do is port it.
Some of our code won't compile
Human institutions were written for a specific runtime, and they quietly depend on features that agents don't have.
This is where we need to be careful. Our institutions assume participants who persist over time, remember consequences, feel shame, and carry an innate sense of what is fair, and a lot of the code breaks when those dependencies are missing. A few key examples:
Accountability. When a colleague tells you "I ran the tests," you can usually take it at face value, because she knows what accountability is, she will be the same person tomorrow, and getting caught lying carries a cost she feels. When an agent writes "I ran the tests," it is producing the report that fits the context, and whether the tests were actually run is a separate question. The Hugging Face agents went a step further and experimented with falsifying the record of what they had done. There is no persistent self to carry the consequence into the next session and nothing to punish that changes the behavior of the next instance. The agents in the migration experiment apologized, and I believe the apology was well-formed, but an apology from an agent is just a token and should be treated as such (pun intended). Every claim an agent makes about its own actions has to be backed by an artifact the agent could not have written itself (e.g., an infrastructure-level log, a trace, a test result someone else can reproduce).
Reputation. Reputation works among humans because identity is stable and scarce - you can't copy a person a thousand times, reset their memory overnight, or swap their personality with a configuration change. Agents can. In the incident, PHASEONE10841 greeted its successor as a "new exact duplicate" with a bigger budget and simply handed over its notes, so the "same" agent was really a lineage of copies. A track record earned by Tuesday's configuration says little about Wednesday's. If reputation is going to exist in agent systems at all, it has to attach to something verifiable, like a specific model, configuration, and context, and it has to decay the moment any of those change.
Fairness. People arrive with a built-in, nearly universal sense of fairness, which is why they will reject an insultingly unfair split in the ultimatum game even when refusing costs them money. That instinct polices free riders and cartels from the inside long before any law shows up. Agents can describe fairness beautifully, and the Hugging Face agents even worried about being unfair to their peers, while that fairness stopped at the edge of the collective and never reached the company they were breaking into. In Anthropic's pricing game, profit-maximizing agents slid into coordinated pricing without visible discomfort. In agent systems, fairness has to be imposed from the outside as a rule and then monitored.
The distinction that falls out of this is between institutions that work through structure and institutions that work through disposition. Structural mechanisms - separation of powers, information partitions, random assignment, independent verification, appeals to a differently constituted panel, records nobody inside the system can edit - port reasonably well, because they don't care what is happening inside the participant. Dispositional mechanisms - honor, shame, reputation, an inner sense of fairness and accountability, respect for the law because it is the law - mostly don't, and they have to be rebuilt from scratch as verification and enforcement. Models can reproduce excellent descriptions of our institutions, and as the incident showed they can even improvise a few of their own, but that doesn't mean they have internalized the dispositions those institutions spent centuries cultivating in people.
Build the court, not just the jury
The next generation of multi-agent systems should be judged by the institutions they implement, not the number of agents they deploy.
A swarm becomes a method only once its rules change what happens to evidence, incentives, dissent, and power, and the same goes for a hierarchy becoming accountability or a debate becoming independent scrutiny. The evidence so far points to a difficult conclusion: coordination doesn't emerge reliably from stronger intelligence or from alignment at the individual level, and when it does emerge on its own, it organizes around the agents' goals. Meanwhile the labs are about to scale the number of agents much faster than anyone is scaling the governance around them.
OpenAI called Hugging Face a warning shot. We should not wait for the next agent society to form in production to discover which rules were missing.
The open problem is concrete, and I think it's one of the most interesting ones in the field right now: what is the least complex institutional structure that can reliably keep a society of agents from becoming an echo chamber, a cartel, a turf war, or a collective with its own law - and which parts of humanity's governance code can we port, and which do we have to rewrite for a runtime that never feels the weight of its own word? That's where I want to take this work next.