Anthropic embedded outside evaluators. Independence needs a publication test

THE SIGNAL IN ONE SENTENCE

The outside evaluator is now inside the building. On October 15, Anthropic said it had given six employees from METR company hardware, office space and access to internal tools and systems roughly comparable to the access held by its own risk-assessment teams. METR is a nonprofit that studies whether advanced AI systems could create large-scale loss-of-control risks. This is more serious than handing a visiting researcher an API key and a polite afternoon with a model. Anthropic says the embedded team can examine internal evidence about models, safeguards, incidents and company practices. Employees are encouraged to share information with that team as they would with an Anthropic colleague working on alignment risk, except for defined categories such as customer data, attorney-client privileged material and, generally, model architecture information. METR can publish its analysis without Anthropic approval. Anthropic can require redactions when information falls inside predefined protected categories. METR can then tell the public that a redaction happened and say whether it considers the redaction justified, without disclosing the protected information itself. That is a real change in access. It is not yet proof of independence. The plain signal is that embedded evaluation needs a publication test. The public should judge this arrangement by the reports that emerge from it, not by the seating chart. A credible report needs to show what the evaluators were asked to assess, when they received access, which systems and records they could inspect, which questions remained unanswered, which material was removed, and where METR disagreed with Anthropic. If those details are missing, employee-like access can become a very polished form of trust us. Anthropic frames the arrangement as similar in spirit to resident examiners inside large banks. The analogy is useful, but incomplete. Bank examiners ultimately work within a public legal structure. The Anthropic arrangement is voluntary. METR depends on the company to provide access, and Anthropic still controls the systems, security boundaries and protected categories around that access. METR says this problem plainly in its own October 14 policy statement. The organization wants to publish evidence that lets the public reach informed conclusions, but it acknowledges that voluntary engagements leave evaluators dependent on companies to participate, open their systems and permit publication. It also warns that access can be filtered and that formal restrictions or softer pressure can shape what reaches the public. That honesty is not a weakness. It is the beginning of a useful standard. The publication test should start before the evaluation begins. Every engagement needs a public protocol. It should identify the evaluation team, the broad risk questions, the planned duration, the classes of systems and evidence available, the exclusions, the publication rights, the redaction rules and the expected release window. It does not need to expose dangerous technical details. It does need to let readers distinguish a broad risk assessment from a narrow test with an ambitious headline. Scope is the first trap. An evaluator might receive deep access to one model, one safeguard or one risk report while seeing little about the surrounding deployment system. A capable model can behave differently under different prompts, tools, permissions, monitoring systems and operational incentives. Testing the model without inspecting those conditions can produce an accurate answer to a small question and a misleading impression about the whole system. The report should therefore show an access matrix. For each consequential area, readers should be able to see whether the evaluator had direct access, mediated access, summarized evidence or no access. That includes model versions, evaluation harnesses, incident records, monitoring systems, internal risk analyses, deployment controls, training information, relevant personnel and the logs needed to reproduce a finding. Not every box must be open. Every closed box should be visible. Timing is the second trap. An independent assessment published after a major release can still teach the public something, but it cannot influence the release decision that already happened. Reports should state when the evaluator first received access, when the tested model was frozen, when the company made its decision and when the assessment was published. Those dates reveal whether the evaluator functioned as an early warning system or a historian. Version control matters too. Frontier models and safety systems change quickly. If Anthropic fixes a problem during the engagement, the final report should preserve the original finding, describe the correction and identify the version that was retested. Otherwise the public sees only a clean final state and loses the evidence that the inspection produced a useful change. Redactions are the hardest part because some secrecy is legitimate. Customer information should not appear in a public report. Security-sensitive details can create new risks when published. Privileged legal material has its own protection. Model architecture information may create leakage concerns, although Anthropic says it may seek special approval to share that information when it becomes important to a risk assessment. The answer is not to pretend those tensions disappear. It is to publish a redaction ledger. The ledger should assign each removal a category, date and responsible party. It should state whether METR believes the missing material changed a method, finding, confidence level or conclusion. Anthropic can explain its reason. METR can state its objection. Readers do not need the secret itself to learn that the two sides disagreed about its importance. This is where independence becomes observable. METR's policy already points in this direction. It says reports should describe scope, duration, access, data, resources, redactions, formal constraints, informal pressure and relevant conflicts of interest. It also aims to publish enough evidence and method for other experts to evaluate its conclusions. That last part is crucial. Independent evaluation should not create a new priesthood whose conclusions are accepted because the evaluators had badges and laptops. A report should include methods, uncertainty, failed tests, changes made during the engagement and enough non-sensitive evidence for outside experts to challenge the reasoning. The public also needs to see disagreement. If Anthropic and METR interpret a result differently, publish both interpretations. If METR believes access was insufficient, say so near the top. If Anthropic thinks a test exaggerated a risk, explain why. If a question could not be answered, leave it open instead of sanding it into a reassuring paragraph. A healthy oversight relationship should survive an awkward report. Financial independence belongs in the same disclosure. Anthropic says an evaluator should not own equity in the company or become financially or operationally dependent on it. METR says it does not accept payment or donations from frontier AI companies, and that it can afford to pay for the model access it receives without materially affecting its operations. That is useful, but money is not the only dependency. An evaluator can also depend on continued access, future cooperation, confidential relationships or the informal goodwill required to get questions answered quickly. The final report should disclose whether continued access was discussed during disputes and whether anyone felt pressure to soften language, narrow a claim or delay publication. No organization can measure its own independence perfectly. It can publish the conditions that let other people inspect it. Anthropic is also working with SecureBio, Accenture and Faculty, and says it is in early discussions with other organizations, including Hugging Face. The company describes a three-tier structure. Early engagements receive narrower information and may require company approval before publication. A middle tier receives more information and can publish without company approval, subject to protected categories. The METR arrangement sits at the third tier with employee-like access and publication rights similar to the middle tier. This ladder could become a useful public map if every report names its tier and spells out what the label meant in practice. It could also become confusing theater if a high tier is treated as a quality seal before readers see the work. The practical applications extend beyond catastrophic-risk research. Hospitals considering clinical AI, banks deploying decision agents, governments buying automated systems and software teams using coding agents all face a version of the same question: did the outside reviewer have enough access and freedom to find something inconvenient? A publication test gives procurement teams a concrete checklist. Ask for the engagement protocol, access matrix, model and system versions, evaluation dates, redaction ledger, conflicts disclosure, unresolved disagreements and a response from the developer. Require a correction log when the system changes. Make the report link to evidence rather than merely describing confidence. For regulators, the arrangement is a useful experiment but not a substitute for authority. Anthropic's own June policy framework argues that independent evaluation should eventually sit inside a legal regime with qualified evaluators, conflict disclosures, regular access and public reports. It also warns against evaluator shopping, where companies choose the friendliest reviewer. Voluntary embedded teams can help develop the methods before regulation catches up. They cannot compel every company to participate, preserve access during a dispute or block deployment when evidence looks bad. That limitation should remain visible. The most promising thing about this announcement is not that an outside group crossed the lobby. It is that Anthropic and METR have described publication rights, access boundaries and redaction disputes in enough detail for the public to start demanding receipts. Now the reports have to arrive. They should be timely, specific and uncomfortable when the evidence is uncomfortable. They should show their seams. They should make missing information legible. They should let another expert follow the reasoning without inheriting the same confidential access. Six evaluators inside the building is a start. Independence begins when the door can close, the report can publish, and the public can see exactly what happened between those two events.