Cohere scaled coding-agent teams. The lead became the bottleneck

THE SIGNAL IN ONE SENTENCE

Putting sixteen coding agents on one job sounds like replacing a developer with a small software company. It can also resemble sixteen interns sliding folders under one office door while the person inside tries to remember which folder contains the database. Cohere Labs published a controlled study of coding-agent teams this week. The researchers changed one design choice at a time: which stage received workers, whether the lead waited or kept working, which models played each role, how many workers joined the team and whether workers could message each other. The headline is not that teams won. They often did. On ProgramBench, Cohere's all-Sol teams improved the average share of hidden behavioral tests passed compared with solo agents. The same teams also cost between three and seven times as much as their solo baselines in the paper's overall summary, with the blog reporting four to seven times for the principal all-Sol comparisons. That is not a footnote. It is the product. Agent teams are a way to spend more inference compute. The design question is where that extra computation buys new information and where it merely creates more work for the lead to merge, test and repair. The plain signal is that the lead becomes the bottleneck unless the team has an orchestration ledger. That ledger should record the task given to every worker, the evidence each worker returned, the files or assumptions it changed, the checks the lead performed, the conflicts left unresolved, the time and cost consumed, and the reason the final result was accepted. Without it, a swarm produces activity. With it, a team can produce inspectable progress. The benchmark matters here. ProgramBench gives an agent a compiled program and its documentation. The agent can run the original program but cannot read its source. It must discover the interface and edge cases, then build a new implementation that behaves the same way. The 200 tasks range from command-line utilities to systems such as SQLite, FFmpeg and the PHP interpreter. The score is the share of hidden behavioral tests the rebuilt program passes. This is useful because it measures observable behavior instead of rewarding code that merely looks plausible. It is also narrow. Rebuilding a black-box program is not the same as maintaining a production service, negotiating requirements with people, passing a security review or owning an incident at 2 AM. Inside that testbed, planning workers were unusually valuable. Cohere divided the work into three stages: exploration and planning, implementation, and verification. When an asynchronous lead delegated only planning, the team reached 63.6 percent reward at a reported cost of $249 across the benchmark setup. Delegating all three stages reached 63.2 percent at $510. The cheaper team worked because ProgramBench begins with uncertainty about what the target program actually does. Planning workers could probe different parts of the executable in parallel and return distinct evidence. The lead then implemented and verified the code with a broader map. The workers were not simply producing more code. They were reducing ignorance. That distinction travels well beyond this benchmark. Before adding agents, ask what the job is missing. If the bottleneck is understanding a poorly documented system, parallel investigators may help. One can map command-line behavior, another can inspect file formats, another can enumerate errors and another can test performance limits. If the bottleneck is making one coherent architectural decision, another pile of implementations may not help. It may hand the lead sixteen incompatible answers and a very expensive afternoon. Cohere's scaling experiment makes that visible. With one Sol lead and Luna workers, planning teams improved as the fixed worker count grew from two to sixteen. The reward rose from 59.6 percent to 61.8 percent. Each planner continued probing the reference program, so additional workers brought additional observations. Implementation and verification teams improved from two to four workers, then slipped slightly as more workers were added. At sixteen implementers, assignments had become much thinner, but the lead still checked the combined program only five or six times, roughly the same number of reference runs used when combining two pieces. More code arrived. Validation capacity did not. This is the lead bottleneck in plain clothes. A lead must decide how to divide the job, explain each piece, track dependencies, absorb reports, resolve contradictions, combine artifacts and test the result. The workers can run in parallel. Much of the lead's judgment cannot. The paper also tested model mix. A strong Sol lead with cheaper Luna planning workers kept 47.5 percent of the reward gain produced by the all-Sol planning team while using 13.9 percent of its extra cost above the solo baseline. In another fixed-size comparison, one Sol lead with sixteen Luna planners reached 61.8 percent reward at $155. The all-Sol team that delegated every stage reached 63.2 percent at $510. Those numbers do not mean every software team should hire sixteen cheap agents. They mean role quality matters. The lead needs enough capability to choose work, recognize useful evidence and integrate it. A cheaper worker can be effective when its assignment is bounded and verifiable. A cheap implementer that writes a critical module badly can erase the savings in review and repair. Cohere found exactly that limit. Upgrading the lead while holding Luna workers constant improved results at each step. Cheap workers did not rescue a weak lead. A Terra lead with Luna workers still fell below a solo Sol agent in the reported comparison. The practical lesson is not strong manager, cheap labor. That analogy is too tidy and too human. The lesson is to spend capability where the workflow concentrates irreversible decisions. Use cheaper workers for parallel reconnaissance, test generation, documentation searches, dependency inventories and other work with clear evidence. Use the strongest reasoning at the points that define architecture, resolve conflicts, approve external actions and decide whether the whole artifact is ready. Then log the handoffs. Every delegated task should have a purpose, an input boundary, an expected artifact and a verification method. A worker report should distinguish observation from proposal. The lead should record whether it checked the evidence, reused the output, rejected it or requested repair. The ledger also needs a queue. If sixteen workers finish at once, their output should not disappear into one giant context window. Reports should arrive as structured units with dependencies, claimed files, unresolved questions and risk labels. The lead should be able to pause intake, request clarification and send a change back to the worker that still carries the relevant context. Cohere's harness allowed the lead to resume an earlier worker rather than briefing a new one. That is a small but important detail. Continuity can be cheaper than repeatedly rebuilding context from scratch. Communication did not automatically solve coordination. When Cohere gave agents a direct messaging tool, 84 percent of messages in the reported Sol-team setup went from workers to the lead. Worker-to-worker messages averaged fewer than one per task. Stronger models used the channel more, but rewards stayed flat for Sol and Terra teams and fell for Luna teams. About half of Luna workers' messaging calls were malformed and rejected by the harness. Adding a channel is not the same as creating useful collaboration. For production systems, measure communication as overhead and evidence. Which messages changed a decision? Which repeated information already in the workspace? Which arrived too late? Which were rejected? A chatty swarm can spend tokens explaining that it is busy. Cost needs the same treatment. The reported dollar figures depend on the models, prices, benchmark and harness used in the study. They are not a quote for a production migration. Still, the relative lesson is durable: a higher score can hide a brutal exchange rate. Cohere cites an outside GDAL reconstruction example in which a three-role agent team more than doubled reward over a single agent but cost fourteen times as much and took thirteen times as long. That may be acceptable for a rare, valuable recovery. It is absurd for a routine task that a capable solo agent or person can finish safely. Set a stop rule before the swarm begins. Give the lead a cost ceiling, elapsed-time limit and minimum evidence threshold. Require an explicit reason before adding another worker. Stop when new workers repeat known findings, when validation queues exceed a defined size or when the expected value of another branch falls below its cost. Most importantly, compare the team with a real baseline. Run the same bounded task with a solo agent, the proposed team and the current human workflow. Measure accepted changes, tests passed, regressions, security findings, review minutes, elapsed time, token cost and post-merge repair. A team that produces ten more files and one more bug has not become ten times more productive. The paper is useful precisely because it refuses the easy swarm story. More agents can improve results. More agents can also duplicate work, fragment implementation, overload the lead and inflate cost. Planning helped most in a benchmark where discovering behavior was the scarce resource. A different task can move the bottleneck somewhere else. So do not begin with a worker count. Begin with the bottleneck. Give every worker a distinct question. Preserve what it learned. Make the lead's checks visible. Count the cost of rejected work. Test the assembled result, not the volume of motion around it. Sixteen workers are not a software team because they ran at the same time. They become a team when somebody can explain what each one contributed and why the combined result deserves to ship.