California wants a kill switch that someone can actually test

01

WHAT ACTUALLY CHANGED

Executive Order N-9-26 became effective on September 18. The signed three-page order directs the California Government Operations Agency to accelerate work already required by state law and to develop recommendations for possible amendments. The operative text is narrower and more useful than the political language surrounding it because it identifies agencies, subjects and dates.

By November 16, 2026, the Government Operations Agency must submit recommendations to the governor in consultation with the Governor's Office of Emergency Services and national experts. The recommendations must address technical feasibility and potential efficacy. That wording leaves room for the experts to conclude that some proposed controls work only for particular architectures, deployment models or threat scenarios.

One required topic is whether large frontier developers should embed designated independent verification organizations onsite in their laboratories for periodic audits and evaluations. Onsite access could expose evidence unavailable in a public model card, but it also creates hard questions about technical competence, trade secrets, evaluator capture, cybersecurity and whether a verifier can publish an uncomfortable result.

A second topic is independent verification of the safety frameworks, transparency reports and risk assessments frontier companies already have to file. The order asks whether standards deemed adequate by an independent verification organization should govern that work. It does not name the standards, the evidence package, the testing frequency or the consequence of a failed verification.

A third topic is the proposed kill switch itself. The order asks for recommendations on requiring one for frontier models and having its efficacy verified on an ongoing basis. It does not define frontier model, kill switch, ongoing, efficacy or the system boundary that must stop. Those definitions will decide whether the control reaches a model endpoint, a tool-using agent, a distributed service, training infrastructure, copied weights or some narrower target.

The fourth topic is incident reporting. The order asks experts to consider expanding the definition of critical safety incidents to include a range of loss-of-control events covering recently reported incidents from large frontier developers. The signed text does not create the expanded definition. It asks for a proposal that could later become an amendment to state law.

Two implementation deadlines sit outside the November review. By May 1, 2027, the Government Operations Agency must publish application requirements, procedures and criteria for independent verification organizations under Government Code section 8898.1. By December 1, 2027, it must complete specified auditor-registry requirements under section 11549.82 and begin the actions required by the next subdivision.

The order builds on three separate pieces of California law. SB 53, signed in September 2025 and effective in 2026, established public frontier-safety framework disclosures, reporting of specified critical safety incidents and whistleblower protections. SB 813, signed September 9, 2026, created a framework for certifying independent verification organizations. AB 1405 created a state registry and independence, transparency and integrity standards for AI auditors.

The governor's press release describes the work as accelerating independent oversight and advancing a kill switch. The signed order is the controlling primary document. It requires state work and recommendations, not immediate installation by private companies. It also says the order creates no enforceable right or benefit against California, its agencies, officers, employees or any other person.

No public technical specification accompanied the order. California has not published a reference architecture, threat model, test suite, performance threshold, false-activation tolerance, activation authority, recovery procedure or model-weight containment plan. Those are not clerical details. They are the substance of whether the phrase kill switch describes a safety system or a memorable prop.

02

WHY THIS MATTERS

Software shutdown is not one action. A developer can disable an API while internal copies keep running, autonomous tasks continue through delegated tools or downloaded weights remain outside the operator's control. Policy needs to identify the exact system boundary before anyone can verify that a stop occurred.

A switch can fail open or fail closed. Failure to stop may leave an incident running. An unnecessary stop may interrupt hospitals, emergency services, security monitoring, accessibility tools or ordinary businesses that depend on a model. A serious design measures both kinds of failure and defines when degraded operation is safer than complete shutdown.

Authority matters as much as machinery. A company executive, state official, independent verifier, court, incident commander and automated monitor should not all hold an identical trigger without a clear chain of command. The rule needs thresholds, authentication, multi-person approval where time permits and an emergency path when time does not.

Independent testing can turn a company promise into evidence. A verifier should be able to trigger a controlled exercise, observe every relevant layer, confirm that prohibited actions stop, preserve logs and test recovery. A demonstration arranged entirely by the developer proves only that the prepared demonstration worked.

Continuous verification should not mean a permanent auditor badge with no test frequency. The system can change after a model update, tool integration, deployment migration, security patch or acquisition. California will need event-based retesting, surprise exercises and a public record of whether required checks occurred, even when sensitive technical results stay protected.

Onsite access can improve scrutiny and create a new attack surface. A verifier may encounter unreleased weights, vulnerability reports, customer data and security credentials. Certification therefore has to evaluate the evaluator: personnel vetting, access controls, conflicts, funding, incident response, data retention and resistance to influence by the company being inspected.

Incident definitions drive learning. If companies report only spectacular harm after it occurs, regulators miss near misses, unexpected autonomy, failed containment, deceptive behavior, unauthorized access and control failures. A broader definition can reveal patterns, but vague reporting duties can bury agencies in noise and expose sensitive exploit details.

California hosts many leading AI companies, so its rules can travel through contracts, product design and national policy. That reach makes precision more important. A workable verification regime could become infrastructure for other jurisdictions. A theatrical switch could become a compliance cost that creates confidence without control.

The order connects law to an unresolved engineering question instead of pretending the engineering is settled. That is a good starting posture. The test is whether the November recommendations publish concrete control objectives, evidence requirements and dissent rather than converting the phrase kill switch into a yes-or-no checkbox.

03

WHERE IT COULD HELP

• Define the shutdown target by layer, including public endpoint, privileged tools, autonomous task queue, inference cluster, training run, copied weights, regional replica and third-party deployment
• Write a threat model for accidental activation, malicious insiders, stolen credentials, compromised verifiers, network partition, cloud-provider failure and an operator that refuses to comply
• Use tiered controls such as rate limiting, tool isolation, capability reduction, network containment, account suspension and full shutdown so every incident does not force the same response
• Require strongly authenticated activation with named authority, two-person approval for non-immediate cases, an emergency override and tamper-evident records of every attempt
• Run scheduled and surprise exercises in a controlled environment, then retest after material model, tool, infrastructure or access-policy changes
• Measure stop latency, systems reached, systems missed, task termination, data preservation, false activation, bypass resistance and recovery time rather than recording only whether a button changed color
• Preserve an evidence package containing alerts, authorization, system state, actions taken, affected replicas, logs, exceptions and recovery decisions for independent review
• Separate the verifier's inspection access from operational control so no outside organization gains an unrestricted path into weights, customer data or production systems merely by being certified
• Publish aggregate test frequency, pass criteria, material failures, remediation status and verifier conflicts while protecting details that would create a practical attack manual
• Design recovery before activation, including who can restore service, what must be revalidated, how downstream users are notified and how the state determines that the incident is actually contained