OpenAI made its Evals platform read only. Export the evidence before shutdown

THE SIGNAL IN ONE SENTENCE

A dashboard can look permanent right up until somebody puts a date on the door. OpenAI has put a date on its Evals platform. The company's current deprecation page says existing evals became read only on October 31. The Evals dashboard and API are scheduled to shut down on November 30. This is not a surprise announcement. OpenAI says it notified developers about the deprecation on June 3. The current event is the freeze. If your team has evaluation definitions, test data, graders, completed runs or review history living in the product, the useful work now is not admiring the old charts one last time. It is making the evidence portable before the remaining window closes. OpenAI's migration guide recommends Promptfoo, an open-source command-line tool and library for evaluating and red-teaming AI applications and agents. The guide says the migration is a manual recreation of the evaluation. It does not require or promise an automatic OpenAI Evals export feature. That sentence deserves a highlighter. Manual recreation means the old green score does not simply teleport into a new system with all of its meaning intact. An evaluation result is the product of several moving pieces. There is the task definition. There is the test data. There is the prompt or application version. There is the model and its configuration. There are the graders and thresholds. There are raw outputs, errors, usage records and human annotations. There is also the environment in which tools, retrieval and other dependencies behaved a certain way. Lose one piece and the score can become a souvenir. Keep the pieces together and the run becomes evidence. That distinction matters because teams use evals to make decisions that reach well beyond a developer dashboard. They decide whether a new model is safe enough to ship. They decide whether a support bot became less accurate after a prompt edit. They decide whether an agent still respects tool boundaries. They decide whether a cheaper model is good enough for a workload. They decide whether a failure was fixed or merely moved out of view. OpenAI's working guide describes an eval using two core ingredients: a data source configuration that defines the test-data schema, and testing criteria containing graders that determine whether the output is correct. Its datasets guide adds the human layer. Annotations can serve as ground truth, diagnose prompt failures and help align automated graders with expert intent. Those ingredients should become the spine of an evaluation evidence ledger. Start with identity. Give every evaluation a stable internal ID that does not depend on a dashboard URL. Record the old OpenAI eval ID, run IDs and dataset references, but treat them as source references rather than the permanent address. Name the owner, application, decision supported by the eval and date range in which the evidence was considered valid. Then preserve the task definition. What behavior was the system supposed to perform? Which inputs were in scope? Which outputs counted as correct, acceptable or harmful? Which known limitations were deliberately excluded? A title such as customer support quality is not a specification. Save the input schema, expected output shape, instructions, variables and any tool or retrieval setup required to reproduce the task. Next, freeze the test data. Export or reconstruct every test case your organization is allowed to retain. Save the exact version, order, labels, annotations, provenance, permissions and redaction state. Compute a hash for the dataset file so a later rerun can prove whether it used the same cases. Do not casually copy sensitive production material into a new repository. The migration is not permission to expand who can see customer conversations, medical details, employee records or security traces. Preserve access controls, retention rules and deletion obligations along with the content. If a record cannot leave its existing boundary, preserve a reference and a documented method for regenerating or securely retrieving it. A portable ledger can point to controlled storage without becoming a data leak in a nicer folder. Now save the graders. OpenAI's Evals guide shows that testing criteria can include exact string checks and other graders. The deprecation page says graders documented for eval workflows are part of the transition. For each grader, record its type, configuration, prompt, model, threshold, reference fields and aggregation rule. If a model judges another model's output, save that judge model's exact identifier and settings too. An average score of 0.84 is not reproducible if nobody knows whether the grader changed, the judge model changed or the pass threshold moved from 0.80 to 0.85. Human review needs equal care. Save the annotation label, critique, reviewer role, review date and adjudication rule. Do not publish private reviewer identities when a stable pseudonymous identifier or role is enough. The purpose is to preserve how expert judgment entered the result, not to build a surveillance file on the experts. Then capture the run configuration. Record the model and snapshot when available, endpoint, prompt version, system instructions, sampling settings, tool definitions, retrieval index version, code commit, dependency versions, region and time. For agents, add sandbox policy, approval rules, available tools, network access, memory state and termination conditions. For every case, save the raw output before reducing it to a score. Also save errors, refusals, timeouts, tool calls, citations, latency and usage when the system exposes them. Aggregate results are convenient for comparison. Raw records are what let a person investigate why the aggregate moved. This is where the ledger earns its name. One folder or database row should connect the definition, data version, grader version, run configuration, raw evidence, human review and final decision. The decision is important. Did the team ship the model? Did it block the release? Did it accept a known regression? Who approved that exception, for how long and with what monitoring? An eval that never reaches a decision record is a lab result left on the printer. OpenAI's migration guide maps the old concepts into Promptfoo concepts. Test data remains test data. Prompts and providers become portable configuration. Graders become assertions and metrics. Runs move from a managed platform workflow into local or continuous-integration execution. The guide also warns that the replacement does not simply reproduce the old interface. That is healthy honesty. The first run in the new system is not proof of equivalence. It is a migration test. Recreate a representative slice first. Run it without cached results. Compare the prompts, inputs, outputs and scores case by case. OpenAI specifically warns that similarity-based scoring may not produce identical numerical results across systems and that manually recreated graders, especially model judges, should be validated before teams rely on them for regression decisions. So keep a migration comparison sheet. For each grader, note whether it is exact, approximately equivalent or intentionally changed. Record expected score tolerance. Investigate cases that flip from pass to fail or fail to pass. If the new system fixes an old flaw, document the break instead of bending the replacement until it recreates the flaw. The practical applications are straightforward. A product team can put portable eval configuration beside application code and run it on every release candidate. A safety team can preserve adversarial cases, refusal criteria and human adjudication without depending on one vendor's interface. A procurement team can compare providers with the same test cases while keeping provider-specific settings explicit. An incident-response team can return to the exact evidence used to approve a model version after a failure appears in production. An auditor can trace a public claim back to the cases, graders and review decision that supported it. None of that makes an eval automatically trustworthy. Test sets can be narrow, stale or contaminated. Automated graders can reward superficial patterns. Model judges can inherit bias and change when their underlying snapshot changes. A high average can hide a catastrophic failure in a small subgroup. Human annotations can disagree. Raw production behavior can escape every lab scenario. Migration can add a second layer of uncertainty. The old and new systems may format messages differently, call tools differently, retry errors differently or aggregate scores differently. A fresh run can also hit a model that changed after the original evidence was collected. That is why the ledger should separate three questions. What did the old run show? What does the recreated run show? Why do they differ? Do not erase the first answer just because the second tool is newer. Do not treat the old result as sacred just because it came first. Keep both, record the gap and decide which evidence is fit for the next decision. The shutdown date creates urgency, but urgency is not a reason to export blindly. Verify which API calls and dashboard views still work for your organization. Check whether related files, prompts, datasets and grader artifacts have separate lifecycles. Confirm what your contract and retention policy allow you to keep. Test the migration before the last week. OpenAI's documents do not promise a one-click archive containing every dashboard view, completed run and linked artifact. Assume you need an inventory. List each eval. List its runs. List the evidence required to explain each important decision. Mark what has been preserved, what has been recreated and what remains available only through the legacy system. Then have a second person open the ledger on a clean machine and rerun one evaluation. If that person cannot find the data, identify the grader, select the model and understand the result without asking the original operator to remember a hidden setting, the migration is not done. Dashboards are useful. They are also rented rooms. The evidence should be able to leave through the front door before the lights go out.