Protein agents could find the record and lose the geometry. Report the harness with the score

THE SIGNAL IN ONE SENTENCE

Ask an AI agent how a protein structure was measured and it may fetch the answer from a database record. Ask the same agent to calculate a torsion angle from the atoms in that structure and the confident tone may stay exactly the same while the reliability changes underneath it. That is the gap at the center of ProteinGeoBench, a new open benchmark and October 19 preprint from researchers at Protege DataLab and Heimdall Bio. The benchmark contains 131 structural-biology tasks. Each task gives an agent one or more protein structure files and a natural-language question. The agent gets a shell, Python, structural-biology libraries and, unless an experiment removes it, network access. It writes a free-text answer to a fixed file. An executable verifier extracts the requested result and assigns a score. The tasks cover four families. Metadata lookup asks for recorded facts such as experimental method, resolution or an accession. Sequence validation checks chains and residue-level claims. Numeric calculation asks for a quantity derived from coordinates, such as a distance, angle, clash or contact. Concept reasoning asks the agent to interpret information across structures. All four can look like protein questions in a benchmark table. They are not the same job. The plain signal is this: report the harness with the score. A model name and one aggregate number do not tell a scientist whether the agent retrieved a fact, parsed coordinates correctly, used a structural library, reached the internet, got lucky on one run or benefited from a grader that awarded partial credit. ProteinGeoBench makes that problem visible with unusual detail. The authors ran 4,249 rollouts across four frontier models on the full suite. They also report 4,486 additional graded trials across 26 models in a broader sweep and numeric extension. On the main comparison, Claude Opus 5 averaged 0.816, GPT-5.6-sol 0.812, Grok 4.5 0.785 and Gemini 3.1 Pro Preview 0.707. The paper treats the leading three as unresolved rather than forcing a podium out of a narrow spread. The reported intervals separated each of those three from the fourth model, but did not cleanly order the leading group. That is good benchmark hygiene. The family breakdown is more useful than the league table. Three of the four models scored higher on metadata lookup than numeric calculation. Claude Opus 5, for example, averaged 0.907 on metadata and 0.716 on numeric tasks. Gemini was the exception, with numeric calculation slightly above its other families. Numeric tasks were only 28 of the 131 questions. They still accounted for six of the twelve tasks that all four frontier models failed. In other words, numeric work was 21.4 percent of the suite and half of the shared failure set. The benchmark does not show that agents cannot do protein geometry. It shows that a subset of coordinate problems remained unusually difficult under these configurations. One dihedral-angle task is a perfect little warning label. The reference answer was negative 42.90 degrees. In six of the seven recoverable failing submissions from the 26-model runs, agents returned positive 42.9 degrees. The magnitude was right. The sign was wrong. That can happen when atom order is reversed, a handedness convention changes or a shared calculation mistake propagates. The task named the atoms in order, so the sign difference was not excused by an unspecified order. The final number alone still could not reveal which step failed. You need the trace. Did the agent identify the requested atoms? Did it load the intended chain and residue? Did it call a library routine or manually parse fixed-width columns? Which convention did the function use? Did the agent read its own output before writing the answer file? Did it mention both signs and let the verifier choose the closest number? The paper found three credited submissions on that task that included both signs. The verifier looked for the closest number, so an answer could earn the accuracy component without committing to one unambiguous result. That is not a scandal. It is a measurement detail that belongs beside the score. The same is true of partial credit. Some tasks gave 0.10 for completeness and 0.90 for accuracy. A non-empty but entirely wrong answer could therefore score 0.10 instead of zero. Across the 4,249 main rollouts, 10.2 percent landed exactly on 0.100 while 1.9 percent landed on zero. If two benchmarks use different reward floors, their means are not automatically on the same scale. Run count changed the ranking story too. The authors examined 99 tasks with eight completed runs for all four main models. With one sampled run per model and task, the exact eight-run reference ordering reappeared in 65.2 percent of bootstrap replicates. A different model ranked first 23.9 percent of the time. Agreement rose as the evaluation budget increased, reaching 97.6 percent for the exact ordering at eight sampled runs. That does not make the eight-run ordering a universal truth. It shows that one-shot rankings can be noisy even when the benchmark itself has strong split-half reliability. The software wrapper mattered as well. The reference setup used terminus-2, a terminal agent running through the Harbor framework. The researchers then swapped in each provider's own command-line agent on a selected 15-task slice. Scores changed from minus 0.071 to plus 0.086 depending on the model and harness pairing. Only one reported interval excluded zero, and the slice was too small to rule out moderate effects in either direction for several comparisons. The direction was not consistent. One model scored lower with its own command-line agent. That is precisely why the harness belongs in the receipt. An agent harness decides how instructions are packaged, which tools exist, how files are exposed, how long the model can work, when output is truncated and what context returns after a command. Calling two systems by the same model name does not make those conditions equal. Network access created another layer. On the 15-task access slice, blocking eight structural-database domains lowered mean reward from 0.752 to 0.662 in the paper's main treatment. Removing the network entirely produced a larger loss in an earlier experiment, but that comparison was graded in a different pass. The authors explicitly warn against treating the difference as a clean decomposition of where the network benefit came from. The rest could include package installation, documentation, mirrors or other online tools. An offline score is not a direct measurement of what the model memorized. This is the part benchmark summaries tend to sand off. Model, task set, harness, access and grader all interact. ProteinGeoBench varies several of them, but it does not place every factor in one fully crossed experiment. The full model comparison uses 131 tasks. Harness and access tests use a selected 15-task slice. The broad model sweep uses a different 20-task subset, followed by a numeric extension. Those effect sizes describe their own task sets. They cannot tell us whether model choice matters more than harness choice, network access or benchmark composition in general. The authors say so clearly. That restraint should become part of ordinary evaluation practice. Every scientific-agent score should ship with a compact receipt. Name the exact model endpoint and collection dates. Record the harness and its version. List the task IDs and family counts. State how many runs were launched, graded, timed out and excluded. Preserve reasoning effort, temperature, time limit, CPU, memory and dependency versions. Describe the network policy and which domains were blocked. Identify the coordinate source. Pin the grader and show its partial-credit rules. Report uncertainty over tasks and runs. Keep answer files and execution traces for failures. Then break the score down by the kind of work the system performed. For geometry tasks, add a more specific trace. Record the structure file hash, model or experimental source, chain, residue, atom order, coordinate frame, units, library call and sign convention. Save the raw value before rounding. If the agent reports alternatives, require it to select one and explain why. That is not paperwork for its own sake. A structural biologist deciding whether to trust an agent needs to know whether the system can compute on a new in-house structure that does not exist in a public database. Strong lookup performance on known entries cannot answer that question. ProteinGeoBench includes thirty structures that were not in public databases when the evaluations ran: twenty tasks use a predicted agatoxin structure and ten use chimeric cadherin structures from a molecular-dynamics pipeline. Metadata-stripped files and restricted-access conditions approximate parts of that private-data setting, but the authors correctly call them approximations. The open repository is a useful contribution here. It publishes all 131 task directories, executable verifiers, model configurations, documentation, known issues and a canary string intended to help detect benchmark contamination. The repository is licensed under CC BY 4.0. That makes inspection possible. It also starts the contamination clock. Once task questions and ground truth are public, future models may encounter them during training or retrieval. Later evaluations should preserve the public suite for reproducibility while adding protected or newly created holdouts for claims about generalization. The study has other limits. The task families and difficulty tiers were assigned by domain experts among the authors, not independently cross-labelled. A small judged component used an LLM evaluator, and agreement among three judge models was low on the tested subset. The main findings are author-run results from a new preprint. Some comparisons have limited power, revised graders or different collection settings. None of that makes the benchmark useless. It makes the receipt essential. The tempting headline is that one protein agent beat another. The more durable result is that a single mean can hide the difference between finding a record and doing geometry, while an apparently stable model name can hide different tools, access rules and evaluation budgets. Report the score. Then report the work that produced it.