The agent got the simulation right and the answer wrong. Name the authoritative file

THE SIGNAL IN ONE SENTENCE

An AI agent can run the right scientific simulation, produce the right number and still give you the wrong answer. That is not the usual hallucination story. The model did not invent a result because it lacked a tool. It used the tool. The simulator completed. The correct value existed in the workspace. The failure came later, when the agent had to decide which file counted as evidence. A new October 19 preprint from researchers at the University of California, Riverside and the University of California, Irvine makes that failure unusually visible. The researchers gave language-model agents access to PHREEQC, a U.S. Geological Survey program for aqueous geochemical calculations. They tested 158 multiple-choice questions whose answers had been rechecked by actually running the simulator. PHREEQC is a useful test bed because it is deterministic and its reports can be traced. It is also wonderfully unglamorous. The task is not to write a sonnet about groundwater. It is to run the model, find the right quantity and report it at the precision the question expects. The plain signal is this: context management is part of scientific accuracy. When long tool output is placed directly in the model's context, the model sees the report. When that output is stored in a file and represented by a table of contents, the agent must decide what to open. That saves context space. It also creates a new choice: which evidence should the agent trust? In the study, that choice mattered. The authors report that GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 percentage points, respectively, when the simulator report was delivered through retrieval rather than inline. Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.4 showed no detectable net cost in the headline comparison. Those numbers do not mean retrieval is generally bad or that one vendor has solved the problem. They mean the interface changed behavior, and the effect depended on the model. The researchers went further than comparing final scores. They re-ran 11,007 logged simulator calls and separated the path to an answer into three stages. Build asks whether the agent's simulation produced the keyed value. Exposure asks whether that value reached the agent through a tool result, read or search. Closure asks whether the agent answered correctly after seeing the value. That decomposition is the important contribution. It prevents every wrong answer from being dumped into one mysterious model-error bucket. If build failed, inspect the simulation input. If exposure failed, inspect retrieval. If closure failed, inspect how the model interpreted and transformed the evidence it saw. For GPT-5.1 and GPT-5.2, the study found that the correct value was produced about as often across interfaces. The larger loss arrived after computation. Under retrieval, the agents often skipped PHREEQC's canonical report and read a shorter file they had created through the simulator's selected-output mechanism. That file held the quantity with more digits. More precision sounds better until the answer choices are printed at a particular precision. The agents frequently truncated rather than rounded, landing on the neighboring option. In one example, a value of 2.75065 should map to 2.751. The agent called it 2.750. The simulator had done its job. The agent had created an alternate representation, preferred it over the canonical report and mishandled the final transformation. The authors call this evidence substitution. It is a useful name for a broad operational risk. An agent may have a vendor invoice, a spreadsheet export and a summary it wrote itself. It may have a lab instrument file, a cleaned table and a notebook cell. It may have a court filing, a search result and a generated case note. All three can contain related values. They do not have equal authority. Retrieval gives the model access to information. It does not automatically teach the model which artifact governs the answer. The most practical intervention in the paper was also the least glamorous. The researchers explicitly told the agent to answer from the simulator's report. That instruction reduced report skipping to under 2 percent and raised accuracy by 6.3 to 7.9 points over ordinary retrieval for the three GPT models in the intervention. For GPT-5.1 and GPT-5.2, directly including reports that fit had a similar effect. Telling the agents merely to round correctly did not help. The error was not fixed by adding a generic arithmetic reminder. It was fixed by naming the authoritative evidence. That should become a design rule for agent systems. Every tool call that produces consequential output should declare its authoritative artifact. The declaration can be simple: this report is the source of record for final values; this other file is a convenience view; this generated note is interpretation. The agent should preserve the artifact's tool version, input hash, run time and relevant line or field. The final answer should point back to that locator. Do not make the model infer authority from filename length, directory position or how easy a file is to read. Build the evidence hierarchy into the harness. There are at least four useful labels. Source of record is the artifact whose value governs the answer. Derived artifact is a transformation such as a filtered table, converted unit or rounded value. Working note is material created for planning or convenience. External reference is supporting information that does not replace the source of record. Those labels should travel with the data, not sit in a forgotten system prompt. If a derived artifact disagrees with the source, the system should stop and surface the mismatch. If a calculation changes precision, the final response should show the original value, transformation rule and displayed result. If the source cannot be reopened, the answer should not quietly borrow confidence from a cached summary. This is especially important when long output is hidden behind indexes and ranged reads. An index can tell the agent where a section begins. It does not prove the agent opened the section, found the intended field or used it in the final answer. A good trace records those events separately. That is where build, exposure and closure become an operations dashboard rather than just research terminology. For each failed task, ask: Did the tool produce the correct evidence? Did the agent retrieve that evidence? Did it answer consistently with what it retrieved? If a team only watches final accuracy, improvements at one stage can hide failures at another. The paper reports exactly that pattern for GPT-5.4: better computation under retrieval offset an evidence-use loss of roughly ten points on numeric items, leaving the net change statistically indistinguishable from zero. A flat score can conceal a moving problem. The study has limits. It is a preprint, not a universal law of agent design. It uses one deterministic scientific simulator, a multiple-choice benchmark, a fixed tool interface and specific model versions. Some reported contrasts are not statistically resolved after correction. Real deployments may involve noisy instruments, ambiguous documents and humans who disagree about which record is authoritative. Those limits make the operating lesson more useful, not less. If authority is contested, the system should expose that contest. If evidence is noisy, the provenance chain matters more. If models respond differently to the same retrieval design, teams need model-specific tests rather than a single context strategy copied across every endpoint. The temptation in agent engineering is to celebrate successful tool use. The tool ran. The file exists. The answer arrived. This paper shows why that is one checkpoint too early. The real question is whether the final claim stayed attached to the artifact that earned it. Name the authoritative file. Preserve the exact field. Record what the agent opened. Show the transformation. Make disagreements visible. The simulation receipt is not complete when the tool finishes. It is complete when a reader can follow the answer back to the right evidence without having to trust the agent's memory of what happened.