Anthropic's Dario Amodei wants governments to have the power to block 'dangerous' AI systems

Overview

Anthropic CEO Dario Amodei called on the U.S. government to be able to block the deployment of dangerous AI models, comparing the approach to how unsafe airplanes are kept from flying. In a post on his personal blog, he proposed mandatory third-party audits of frontier AI systems and government power to shut them down if unacceptable risks are found. His proposal goes further than President Trump's June 2 executive order, which encouraged voluntary model sharing. Some analysts were skeptical of his motives.

Key Takeaways

Dario Amodei wants the U.S. government to be able to block deployment of dangerous AI models.
He proposed mandatory third-party audits of frontier AI systems and government authority to shut them down.
He suggested a compute threshold that would automatically trigger an independent investigation before launch.
His proposal goes further than Trump's June 2 executive order, which only encouraged voluntary sharing.
Analyst Holger Mueller was skeptical, suggesting the call could aim to freeze the market for a leader.

Stats & Key Facts

#Four specific risk categories identified
#Trump's executive order dated June 2
#Responsible Scaling Policy published September 2023

Anthropic's Dario Amodei wants governments to have the power to block 'dangerous' AI systems

The proposal

Amodei laid out an aggressive AI safety framework.

›He called on the U.S. government to block deployment of dangerous AI models like it prevents unsafe airplanes from taking off.
›He said there need to be mandatory third-party audits of frontier AI systems.
›He suggested governments be able to shut systems down if unacceptable risks are discovered.

He also called on the government to provide more economic support to people financially hurt by the rise of AI.

The compute threshold

Amodei tied audits to training resources.

›He suggested auditing new models based on the amount of resources used in training.
›He proposed a compute threshold that, if exceeded, would automatically trigger an independent investigation.
›The investigation would assess capabilities before a model can be launched publicly.

Four risk categories

The essay named specific areas for evaluation.

›Cybersecurity vulnerabilities.
›Biological weapons capabilities.
›The ability to accelerate automated research in dangerous domains, and the potential for models to grow beyond human control.

If auditors decide a model poses an unacceptable risk in one of those areas, Amodei wants governments to be able to prevent its deployment. He said regulations for aircraft, automobiles, and drugs should serve as models.

Comparison with Trump's order

Amodei's framework goes further than current policy.

›His proposal exceeds Trump's June 2 executive order, which mandates intelligence agencies play a role in testing models for risks.
›The order encouraged AI firms to share models voluntarily with government auditors a month before public release.
›Amodei used stronger language, asking for compliance rather than cooperation and giving independent auditors effective veto power.

The article notes there is no mention of the word voluntary in Amodei's framing, and it calls for independent auditors rather than leaning mainly on intelligence agencies.

Anthropic's track record and skepticism

The proposal fits Anthropic's history but drew doubt.

›Anthropic published a Responsible Scaling Policy in September 2023, introducing concepts such as AI Safety Levels (ASLs).
›The company supported other legislation, such as California's SB 53 bill.
›Analyst Holger Mueller of Constellation Research suggested the personal blog post might insulate Anthropic from backlash.

Mueller said the call for regulation gives off the impression of a market leader wanting to freeze the market and preserve its position, while acknowledging it is difficult to prove.

Frequently Asked Questions

What is Dario Amodei proposing?

He proposes that the U.S. government be able to block deployment of dangerous AI models, require mandatory third-party audits of frontier systems, and shut systems down if unacceptable risks are found.

What is the compute threshold idea?

Amodei suggested that if a new model's training exceeds a specified compute threshold, it would automatically trigger an independent investigation into its capabilities before public launch.

Which risk categories did the essay identify?

Cybersecurity vulnerabilities, biological weapons capabilities, the ability to accelerate automated research in dangerous domains, and the potential for models to grow beyond human control.

How does the proposal compare to Trump's executive order?

It goes further than Trump's June 2 order, which encouraged voluntary model sharing and intelligence-agency testing but stopped short of mandatory testing or enforcement.

How did analysts react?

Holger Mueller of Constellation Research was skeptical, suggesting the call for regulation could reflect a market leader wanting to freeze the market, while noting that is hard to prove.

Amodei is pushing for binding government oversight of frontier AI, a stance that drew both support and skepticism.