NumTrace caught finance agents with the right number in the wrong account. Audit the binding

THE SIGNAL IN ONE SENTENCE

Finance loves a correct number. It loves that number considerably less when it belongs to somebody else. That distinction sits at the center of NumTrace, a new benchmark for auditing financial analyst agents. The authors tested whether agents could answer questions from SEC EDGAR filings and retirement-plan documents while preserving the provenance of every numerical claim. Their central finding is a useful little horror story: an agent can retrieve digits that really appear in an authoritative filing, reproduce those digits correctly and still attach them to the wrong company, concept, period, unit or group. The paper calls this a right-number-wrong-account error. In accounting, the binding is the answer. Consider the paper's worked example about a group of co-filing investment managers associated with Point72. The question asks for an aggregate holding across the group. One tested agent returns 256 shares. Those digits exist in the filing, so a simple value checker can wave the answer through. But 256 belongs to one manager's row. The group total requested by the question is 2,854,645 shares. The number is real. The answer is wrong. This is the kind of failure that can survive a superficial citation check. A reviewer clicks the filing, searches for the digits and finds them. The source looks official. The agent looks careful. The binding error hides in the distance between what the number says and what the question asked. NumTrace is designed to inspect that distance. The benchmark contains 540 questions covering 2,163 documents from a frozen SEC EDGAR collection. The documents include 2,132 filings and 31 retirement-plan documents, divided into 41,726 searchable chunks. The questions span 12 families and produce 2,306 answer slots. The authors added 420 questions centered on Form 13F institutional holdings and Item 402(i) pay-versus-performance disclosures. Each numerical claim receives three separate verdicts. The value check asks whether the digits and arithmetic agree with the cited source. The binding check asks whether the figure belongs to the correct entity, concept, period, unit and scope. The support check asks whether the cited document is authoritative and actually supports both the number and its binding. That separation is more useful than a single accuracy score. If value fails, the agent may have copied or calculated incorrectly. If value passes but binding fails, the agent found a real number and assigned it to the wrong account. If value and binding pass but support fails, the answer may be right without a defensible source trail. The authors evaluated Claude Opus 5.5, GPT-6 Astra and DeepSeek-V4.1-Flash inside the same four-stage workflow: retrieve, extract, calculate and synthesize. The agents could issue up to six BM25 search queries and received the top five chunks for each query. They had no additional browsing or external tools. Across the three systems, the authors report that right-number-wrong-account errors accounted for 67.5 to 81.8 percent of failing claims. On the co-filing-group questions, around 91 percent of claims passed the value check, while fewer than half passed the binding check. Those percentages are the headline. They are not the whole result. The paper also shows why the benchmark itself needs an audit trail. NumTrace's reference answers come from 216 deterministic rules. Twenty-two follow external authority. The other 194 encode author design choices in situations where reasonable alternatives may exist. An independent programmer reproduced 443 of 444 sampled answer slots, which is strong evidence that the rules were implemented consistently. It does not turn every design choice into the only defensible interpretation. The automatic scorer also made mistakes. In an expert review of 360 sampled claims, the weighted expert binding-pass rate was 88.0 percent, compared with 90.6 percent from the scorer. The authors estimate that the scorer produced false failures and false passes, with false failures especially important when interpreting the raw failure counts. The reviewers saw two language-model labels before making their decision, which the paper acknowledges could have anchored their judgment. The dramatic co-filing result rests on nine usable groups. That is enough to expose a failure mode. It is not enough to estimate every agent's production error rate across the financial industry. Retrieval was another constraint. In an offline study, the search system surfaced every needed chunk for 57 percent of the 512 questions with scorable evidence. Coverage fell to 10 percent for the 13F top-holdings family. Some downstream errors therefore begin before the model reasons about the number. The model comparison is not perfectly like for like either. Two systems ran through command-line interfaces and one through an API. Each agent received one scored run per question. A result can describe these tested configurations without becoming a timeless ranking of three model families. That is the caution. The practical lesson still travels. Every number produced by a financial agent should carry a small passport. The passport should record the source document, source span, entity, concept, period, unit, scope and derivation. It should also record where the claim first became visible in the pipeline. If the wrong entity entered during retrieval, the repair is different from a synthesis-stage rewrite that dropped the word aggregate. A financial memo should not accept a naked number merely because it matches digits in a filing. Imagine an agent says revenue rose to $4.2 billion. The release gate should ask: revenue for which legal entity? Under which accounting concept? For what quarter or year? In what currency and scale? Consolidated or segment level? Reported, adjusted or calculated? Which filing and which exact span? Was the number copied, converted or derived? If the agent cannot answer, the number is not ready for the client deck. This is especially important for filings with repeated figures. A 10-K may contain the same amount in a table, note, comparison and reconciliation. A 13F may repeat a security across managers that share a filing. A pay-versus-performance disclosure may place company-selected measures beside standardized compensation figures. Exact-string matching is weakest precisely where a document is numerically dense. The control can be built into the workflow. At retrieval, preserve document identity and table coordinates. At extraction, require a typed record instead of a loose string. At calculation, save the operands, formula and units. At synthesis, block a claim whose binding fields are missing or inconsistent. At review, show the source span and the claimed binding side by side. The human reviewer should not have to reverse-engineer the agent's memory. Teams can also use deliberately adversarial tests. Ask for a subsidiary when the parent appears nearby. Ask for a quarter next to a year-to-date total. Ask for shares when market value sits in the adjacent column. Ask for one filer and then for a co-filing group. Change the unit scale. Move the date. If the agent returns the same digits, the test has found a binding problem. This is not merely a finance issue. Medical agents can attach a real lab value to the wrong patient or date. Legal agents can attach a real citation to a different jurisdiction. Supply-chain agents can attach a real quantity to the wrong warehouse. Scientific agents can attach a real measurement to the wrong experimental condition. Provenance is often treated as a link. NumTrace argues, in effect, that provenance is a structured relationship. The source matters. So does the route from the source to the claim. That makes the final interface important. A provenance record that exists only in a hidden trace will not help the analyst signing the report. Show the binding receipt where the decision happens. Highlight uncertainty. Make conflicting fields visible. Let the reviewer reject the claim without rewriting the whole document. And keep the model honest about what it did not find. An agent that says it cannot establish the group total is annoying for five minutes. An agent that confidently supplies one manager's holding as the group total can be annoying for considerably longer, possibly with counsel present. NumTrace is a new preprint, not a certification regime. Its document snapshot, question families, retrieval stack, scorer and single-run design limit what the reported percentages can prove. Independent replications, broader filing types, repeated runs and operational studies would make the evidence stronger. Still, the paper gives financial teams a sharper acceptance test than 'the number appears in the filing.' Ask whether the digits are right. Then ask whether they belong there. The second question is where the account begins.