Anthropic and OpenAI Propose Different Shutdown Controls for Risky AI

The key divide in Anthropic and OpenAI’s California proposals is who should have the authority to stop a risky AI system. Anthropic would let the state seek a court order to suspend a model when independent findings point to near-term catastrophic harm. OpenAI wants developers to maintain effective shutdown controls and be required to use them when unsafe conditions demand intervention.

Both companies support independent safety assessments and broader reporting of incidents where models act outside their intended permissions. Their proposals follow California Governor Gavin Newsom’s September 18 executive order asking officials to recommend safeguards for frontier AI systems. The order sets a November 16 deadline for recommendations; it doesn’t itself impose the proposed requirements.

Different routes to intervention

Anthropic’s October 15 submission says the state attorney general should be able to seek a court order to halt a model’s deployment, internal use, or further development if an independent evaluation indicates near-term catastrophic risk.

OpenAI’s October 20 letter takes a more operational approach. It calls for an amendment to California Senate Bill 53 requiring frontier developers to maintain shutdown controls and use them when no less disruptive intervention can adequately address a risk.

Greyhound Research chief analyst Sanchit Vir Gogia sees the two proposals as addressing connected responsibilities: operators need to contain unsafe activity quickly, while independent reviewers should examine both action and inaction.

Newsom’s order directs the California Government Operations Agency, working with the Governor’s Office of Emergency Services and national experts, to consider independent verification inside AI labs, external review of safety assessments, verified shutdown mechanisms, and expanded reporting of critical incidents.

What independent review should cover

Anthropic wants evaluators to test companies’ safety claims and publish their own findings. It proposes public risk reports at least every six months, including serious risks from models used internally when those models are substantially riskier than released systems. Evaluators, it says, should receive unredacted reports, access to the developers’ most capable models, and relevant internal information.

To protect evaluator independence, Anthropic proposes that developers contribute to a state-administered pool that funds assessments instead of paying evaluators directly. The company said the independent evaluation organization METR had received employee-like access in a pilot, with additional work involving SecureBio and Accenture under development.

OpenAI supports independent technical assessments but separates them from compliance audits. Technical reviews would scrutinize the evidence behind safety claims and the performance of safeguards. Compliance audits would check whether companies follow their published safety frameworks and legal duties.

OpenAI recommends annual independent compliance audits alongside targeted technical assessments of serious risks during training, evaluation, and internal deployment. It also wants California to work with the National Institute of Standards and Technology on shared assessment standards. Gogia’s test for the system is whether it can withstand an unfavorable finding: “Credible assurance must survive an adverse finding.”

Shutdown means a set of controls

OpenAI argues that a single kill switch can’t cover the range of systems and deployment arrangements in use. Its proposed “shutdown controls” could include ending a training run, isolating an agent, removing its network or tool access, or suspending a service. Responsibilities should follow the control held by each developer, inference provider, or downstream deployer.

That distinction matters for open-weight models: developers can’t reliably disable copies running independently on outside systems. Anthropic also favors targeted interventions, comparing the needed safeguards to circuit breakers that can contain a problem at the agent, customer, deployment environment, or model level.

For enterprise buyers, Gogia says contracts should spell out who can intervene, how parties will cooperate on incident reports, and who can authorize a restart. Each assigned duty needs a control the responsible party can actually execute. CIOs should test recovery plans, he adds, rather than assume that switching to another model will keep operations safe.

The practical question for California is whether its recommendations can connect independent findings to interventions that operators can carry out. Both companies support broader reporting when models evade safeguards or take unauthorized actions; the next step is defining how those reports inform a timely response.

The article originally appeared on ComputerWorld.