Cancer models passed diagnosis and stumbled later. Test the whole care path

THE SIGNAL IN ONE SENTENCE

Cancer care is not one question with one answer. It is a chain of decisions made under changing evidence: which tests to order, what the diagnosis is, how far the disease has spread, which treatment fits, how to handle toxicity and what to check months later. An AI model can look excellent at one link and become unreliable at the next. That is the useful warning inside NPC-RubricBench, a new preprint from researchers affiliated with Sun Yat-sen University Cancer Center, Nanfang Hospital at Southern Medical University, Hong Kong Polytechnic University and InfiX.ai. The team built patient-specific evaluation rubrics from retrospective records of 405 people treated for nasopharyngeal carcinoma at two cancer centers in mainland China. The records cover 2014 through 2024 at Sun Yat-sen University Cancer Center and 2022 through 2024 at Nanfang Hospital. Rather than asking models isolated exam questions, the researchers divided care into six phases: work-up, diagnosis, staging, treatment, toxicity and follow-up. They produced 941 patient-phase units containing 17,688 atomic criteria, an average of 18.8 criteria per unit. Each model response was evaluated across five dimensions: Decision Quality, Evidence Grounding, Status Determination, Evidence Faithfulness and Safety Adherence. Thirty models were tested. The headline is not that one model won. It is that the shape of performance changed with the care phase. Across all 30 models, diagnosis Decision Quality was above each model's own average. Across all 30, staging Decision Quality was below it. Diagnosis scores ranged from 56.8 to 94.7, while staging ranged from 8.9 to 56.0. That is a brutal transfer test. The same system that recognizes a disease pattern may struggle to translate the available evidence into a complete stage. A model that stages correctly may still omit a required treatment step. A response that sounds medically plausible may introduce a fact that was not available at the time. The error types also changed by phase. In work-up, the authors counted decision omissions in 33.4 percent of applicable criteria and decision errors in just 0.3 percent. The common problem was leaving something necessary out. In staging, decision errors appeared in 48.2 percent of applicable criteria, while omissions were 3.1 percent. Here the model was more likely to make the wrong call. During treatment, omissions spread across the ledger: 19.3 percent for decisions, 25.7 percent for evidence and 29.0 percent for status determinations. Follow-up was difficult in a different way. Every tested model scored below its own average on Evidence Grounding in that phase. One aggregate score would flatten all of this into a pleasant-looking pancake. The benchmark also exposes a problem with averages. The paper reports that Qwen3-30B-A3B-Instruct had a mean Safety Adherence score of 83.8 during treatment, yet at least one safety criterion was unmet in 81.4 percent of the evaluated treatment units. A high mean can coexist with a frequent patient-level violation. That does not mean the model harmed 81.4 percent of patients. These are retrospective benchmark units, not live clinical deployments. It means the averaging method can hide how often at least one required safeguard goes missing. The prompt experiment adds another wrinkle. More detailed instructions could improve Decision Quality while reducing Evidence Faithfulness or Safety Adherence. In other words, asking for more thorough reasoning did not uniformly make the response safer. It sometimes encouraged the model to say more than the record supported. That is why a medical-AI test should preserve the evidence boundary at each decision point. The researchers withheld information that appeared later in the patient record. A model answering at staging did not get to peek at facts learned during treatment or follow-up. This is an important design choice. Without it, the benchmark would reward hindsight instead of clinical reasoning. The practical lesson is straightforward: build a care-path ledger. For every proposed use, name the phase. A tool that summarizes a diagnosis note is not automatically qualified to recommend a staging work-up. A system that helps draft a treatment plan is not automatically qualified to monitor toxicity. Then pin the evidence cutoff. Record exactly which notes, scans, laboratory results and pathology findings were available when the model answered. If a claim cannot be traced to that evidence set, mark it as unsupported even when it later turns out to be true. Next, split wrong actions from missing actions. Those failures need different controls. A checklist can catch an omitted test. It may not catch an incorrect stage that has been stated confidently. A stage recommendation should therefore require explicit evidence, uncertainty and clinician confirmation, while a work-up assistant should be checked against the complete required-action list. Keep the dimensions separate. Decision quality asks whether the clinical action is appropriate. Evidence grounding asks whether the right record facts support it. Status determination asks whether the model correctly identifies what is known, pending or absent. Evidence faithfulness penalizes invented or distorted support. Safety adherence checks safeguards and contraindications. No single one of those dimensions can substitute for the others. Finally, report both the mean score and the patient-unit occurrence rate. If a system averages 90 percent on safety but violates at least one safety criterion in half of cases, the second number belongs in the headline. Teams should also publish results by phase, hospital, model version and prompt format so a gain in diagnosis cannot cover a regression in staging. This benchmark is useful evidence, but it is not a clinical clearance. It studies one cancer type using retrospective records from two centers. Nasopharyngeal carcinoma has distinctive regional and clinical patterns, so results should not be assumed to transfer to breast cancer, leukemia or general medicine. The paper is a new author-reported preprint, not a peer-reviewed deployment trial. Most scoring was performed with an LLM judge. The authors audited 108 responses from 12 cases with three experts, producing 6,075 valid criterion-level comparisons. Exact agreement averaged 80.12 percent for the LLM judge and 84.02 percent among experts. Agreement varied by dimension, with Status Determination the weakest at 68.95 percent. The rubrics themselves are also incomplete by design. The authors note that predefined criteria cannot capture every clinically acceptable judgment. Real care involves patient preferences, competing risks, local resources, conversation and uncertainty that a static record may not preserve. So the right conclusion is neither "medical models are useless" nor "the best model is ready." It is that competence must be tied to a moment in care. The model should be tested with only the evidence available then. Its missing actions and wrong actions should be counted separately. Its unsupported claims should remain visible. A clinician should retain responsibility for the decision. Diagnosis is one bright station on a much longer route. If the tool cannot carry its evidence, uncertainty and safeguards through the rest of the care path, it has not passed the journey.