CWE-bench v2 opened 100 security fixes. Keep the public and hidden exams separate

THE SIGNAL IN ONE SENTENCE

A public benchmark is an exam with the answer key sitting nearby. That is not automatically a problem. It can be a very good way to find bad questions, broken graders and security tests that reward the wrong fix. CWE-bench v2 leans into that tradeoff. Collinear AI has released a public set of 100 defensive software tasks through Hugging Face. Each task gives a coding agent a real open-source repository containing one security weakness and a short instruction. The agent must find the weakness, repair it and preserve normal behavior. The set spans 91 Common Weakness Enumeration categories across Go, Java, JavaScript, C++, C, Python, Rust and TypeScript. The problems are packaged as Harbor tasks with Dockerized environments, instructions, reference fixes and executable verifiers. The grader is deliberately blunt. A patch receives a reward of one only if every tested exploit is closed and every legitimate behavior still works. One failed check makes the reward zero. If the environment cannot be evaluated, the trial is void rather than quietly scored as a failure. That is a useful correction to benchmarks that ask another model whether a patch looks convincing. Executable tests can show that a known attack no longer works and that expected behavior survived. They cannot show that the patch is secure against every attack nobody put in the test. Collinear says every task was reviewed by hand. The team also says it checked whether the instruction sounded realistic, whether the required behavior could be inferred from it, whether the verifier tested behavior rather than one specific code shape, whether vulnerable starter code failed, whether the reference fix passed and whether a memorized upstream patch would be rejected. That last check is especially interesting. Real vulnerability fixes often already exist upstream. A model may have seen the exact patch during training or retrieve it from the public internet. A benchmark that accepts only a memorized patch would confuse recall with security reasoning. CWE-bench tries to reduce that shortcut inside its public tasks. The larger problem remains: once the task, repository snapshot, verifier and reference material are public, model builders can train against them, tune prompts around them and inspect failures until the score rises. The benchmark is supposed to become easier after publication. That is what public debugging is for. Collinear maintains a separate held-out set and asks labs to contact the company for evaluation. The separation matters, but outsiders cannot inspect a secret set with the same confidence they can inspect the public one. We do not know from the public materials whether the hidden tasks match the same difficulty, repository mix, language distribution or review quality. So keep two exams and give them different jobs. Use the public set as a workshop. Let teams reproduce results, inspect graders, file bugs, measure regressions and learn exactly which vulnerability classes remain difficult. Publish the harness, costs, attempts and failure traces. Use the hidden set as a thermometer. Rotate it, quarantine its tasks and keep model developers away from the cases before scoring. Report its construction method, category distribution and review process without exposing the actual repositories, instructions or exploit probes. Do not blend the two scores into one friendly number. A public score measures capability on known material. A hidden score estimates transfer to unfamiliar material. The gap between them is part of the result. Every run also needs an exam receipt. Preserve the exact model snapshot, system prompt, agent harness, tool permissions, network policy, time and token budgets, repository commit, task version, attempt count, cost, grader version and raw verifier logs. Record when the task first became public and whether the model's training cutoff came before or after that date. That does not prove contamination, but it keeps an important source of uncertainty visible. The new public leaderboard reports results from 11 models. Collinear says Gemini 4 Argon leads with 67.8 percent pass at one and 81.0 percent pass at four, just ahead of Claude Opus 5.5 at 67.3 percent and 78.0 percent. The authors say 95 of the 100 tasks were solved by at least one tested model, while the leader failed 19 tasks on every attempt. Those are author-reported benchmark results, not production security guarantees. Model versions, pricing, harness behavior and leaderboards can change. Four attempts can also hide useful variance. A model that succeeds once after three unsafe patches is different from a model that produces a defensible patch every time. Report the entire attempt distribution. Show pass at one, pass at four, cost, time, invalid trials and how often a patch closed the known exploit while introducing a new problem. Keep the failed diffs, not just the winning one. The public dataset says agent execution runs without network access except for the permitted model API host. That is a sensible evaluation boundary. Production coding agents may have package registries, documentation, issue trackers, secrets and deployment tools. A clean benchmark sandbox does not reproduce that attack surface. Before trusting an agent on a live security fix, add human review, static analysis, dependency scanning, fuzzing, secret scanning and a staged rollout. Test nearby variants that are not in the benchmark. Ask another reviewer to attack the patch instead of merely admiring it. The verifier should be the beginning of the review, not the judge's gavel. Open exams make the field better because everyone can find their flaws. Hidden exams keep the score honest because the model cannot rehearse the exact questions. Security evaluation needs both, clearly labeled and stubbornly separate.