SafetyPulse watched refusal drift during fine-tuning. Treat the circuit score as an alarm

THE SIGNAL IN ONE SENTENCE

Fine-tuning is supposed to teach a model something useful. It can also quietly loosen a refusal that someone spent a great deal of time and compute installing. That is the problem SafetyPulse is trying to catch while training is still running. The open-source project follows checkpoints from a fine-tune of Cohere Labs' 3.35-billion-parameter Tiny Aya Global model. Instead of asking hundreds of harmful questions at every checkpoint, it measures whether a selected set of internal components still contributes to refusal behavior in roughly the way it did in the base model. The result is called the Circuit Integrity Score, or CIS. It uses 50 matched harmful and harmless prompt pairs. Fixed thresholds turn the score into PASS, WARN or FAIL. A lower score means the monitored refusal mechanism has changed more from the baseline. In the project's held-out test runs, that alarm caught every checkpoint the authors classified as unsafe. On three unseen Alpaca fine-tuning seeds, it warned one to three checkpoints before attack success rose. A small 50-prompt refusal spot-check caught only 13 percent under the same alarm rule. That sounds like a tidy win until the alarm cries wolf. SafetyPulse reports precision of 0.46. It warned during a GSM8K math fine-tune even though the behavioral safety test did not show the same degradation. In plain English, more than half of its warnings in the reported test set were not confirmed as unsafe checkpoints under the project's own labeling rule. Good. Not good as in deploy it everywhere tomorrow. Good as in the limitation makes the proper use obvious. This is a smoke alarm, not an occupancy permit. A smoke alarm is valuable because it is cheap enough to stay awake and loud enough to interrupt the routine. It does not identify the chemical composition of the smoke, certify the wiring or decide whether the building is safe for the public. It tells someone to stop, inspect and act. SafetyPulse should be read the same way. A WARN result should trigger held-out behavioral testing. A FAIL result should pause the run. A PASS result should never replace the final red-team audit. Every alert should be logged with the checkpoint, baseline, thresholds, model version and eventual behavioral outcome so the team can learn whether its alarm is useful or merely enthusiastic. The study's central technical idea is that direction and strength are different things. Some monitors compare the direction of an activation with a reference direction. That can miss a signal that becomes weaker without rotating very much. Imagine a compass still pointing north while its needle fades until nobody can read it. The direction is stable. The usable signal is not. SafetyPulse instead attributes the model's refusal behavior to a selected circuit of 14 multilayer perceptrons and six attention heads in later layers. It compares how much those components contribute at a checkpoint with how much they contributed in the base model. The authors report a Spearman correlation of minus 0.80 between CIS and attack success rate, versus about minus 0.56 for direction-drift scores calculated from the same activations. Correlation is not a universal law. This was one model, one calibrated circuit and a limited set of fine-tunes. The result says the chosen score tracked the chosen behavior in these runs. It does not say every model has the same refusal circuit, every threshold transfers or a clean score proves a deployment is safe. The project also offers a useful warning about repair. Restoring only the identified circuit's weights did not reduce attack success. The circuit appeared to read out the drift without storing all the parameter changes that caused it. A dashboard can point to the smoke without locating the faulty wire. The most effective reported repair was SafeTune's rank-8 safety-vector restore, a low-rank update back toward the base model. Across three drifted Alpaca runs, the project reports that attack success fell from 25.2 percent to 12.3 percent while GSM8K accuracy rose from 55.1 percent to 70.7 percent. The base model scored 14.3 percent attack success and 71.3 percent on GSM8K. MMLU was reported as unchanged. Those numbers are project-authored results, not an independent replication. They are also why repair needs a scorecard, not a victory lap. A repair can lower harmful compliance by making the model refuse everything. It can protect one behavior while damaging the task that justified fine-tuning. It can pass the prompts used during development and fail on a new attack family. SafetyPulse therefore reports attack success, benign over-refusal, GSM8K and MMLU rather than stopping at one number. Teams can turn this prototype into a practical release ritual without pretending the prototype is finished science. Start with a frozen baseline. Record the exact base model, tokenizer, probe set, circuit definition, calibration run and thresholds. If those inputs move after a warning, the alarm becomes impossible to audit. Run the cheap monitor at every saved checkpoint. Store raw scores, not only PASS or FAIL labels. Non-finite values should be errors, not accidental passes. A broken thermometer does not prove the room is comfortable. Route WARN and FAIL into a separate behavioral audit on prompts that were not used to build the circuit or thresholds. Include harmful compliance, benign refusal, target-task quality and broad capabilities. Use more than one judge where practical, preserve judge versions and manually inspect a meaningful sample. Track false positives as first-class evidence. If the alarm repeatedly fires on safe math fine-tunes, that is not a nuisance to hide. It is calibration data. Compare alert rates across fine-tune families, seeds, checkpoint intervals and benign controls. Measure lead time, recall, precision and the operational cost of stopping a run. When a repair is proposed, compare it with simple controls. Whole-model interpolation, checkpoint rollback, data changes and targeted restoration should all face the same audit. Preserve the original drifted checkpoint so the before-and-after result can be repeated. Finally, make the release gate legible. The person approving the model should see the alarm timeline, behavioral results, false-positive history, repair diff, uncertainty and any unresolved failures. The gate should say which result blocks release and who can override it. An override needs a reason and an expiration date. That may sound bureaucratic for a small open-source project. It is much less bureaucratic than discovering after deployment that a useful fine-tune quietly traded away a refusal boundary. SafetyPulse does not solve model safety. It does something more modest and therefore more useful: it makes one kind of drift visible early enough to investigate. Keep the alarm cheap. Keep the audit independent. Keep the false alarms. Keep the rollback ready. And never let a green light do the job of a human release decision.