Anthropic's warning: AI is on the cusp of getting smarter on its own

US-based AI firm Anthropic warns AI development is advancing at a pace that could soon see agents building, training and improving themselves without human input — recommending a slowdown in development.

In a blog post published Thursday, Marina Favaro, lead at the Anthropic Institute, and Anthropic co-founder Jack Clark said agents can already run code themselves, delegate hours of work to other agents and could be on the cusp of taking over completely.

“For most of AI’s history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work,” they said.

“Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor,” Favaro and Clark added.

AI development is advancing at a pace that could lead to agents improving without human input. Source: Anthropic

There are concerns over what could happen if AI is able to become smarter on its own. In December, OpenAI said it is researching how to safely develop and deploy increasingly capable AI, including AI capable of recursive self-improvement.

“We want these systems to consistently follow human intent in complex, real-world scenarios and adversarial conditions, avoid catastrophic behavior, and remain controllable, auditable, and aligned with human values,” it said.

The company is also hiring a researcher for recursive self-improvement preparedness, which forms part of its Safety Research team.

AI model improvement has been roughly doubling every four months, rather than every seven months, according to Favaro and Clark. The role of humans is narrowing at each step, with Anthropic’s Claude model authoring around 80% of the code merged into Anthropic’s codebase.

“We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” they said.

“Once human- and AI-authored code quality reach parity, humans will stop writing code entirely and shift to only reviewing it. But if they can’t review code as quickly as Claude can generate it, human review will become the bottleneck to AI development,” they added.

Favaro and Clark also said that slowing development to allow more time to address its “immense” implications would be ideal.

In April, Anthropic ruled out releasing its AI model, Claude Mythos, to the public over concerns about the threat to global cybersecurity.

Claude Mythos was able to easily create software exploits, leading Anthropic to rule out a public release for now. Source: Anthropic

At the same time, a group of tech leaders, including some from Anthropic and OpenAI, released an open letter on Thursday, urging lawmakers to enact stronger guardrails around the technology over concerns it could be used to overcome “knowledge barriers” that have historically prevented bad actors from creating biological weapons.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” Favaro and Clark said.

“But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.”

AI agents are becoming increasingly popular, including among crypto users. Some crypto executives have speculated that AI agents settling transactions could drive adoption and transaction volumes. Circle CEO Jeremy Allaire predicted in January that billions of AI agents would operate on users’ behalf within five years.

Crypto investment firm Keyrock reported last month that AI agents settling payments went from concept to reality in the past 12 months, with $73 million settled across 176 million transactions.

Magazine: Korea’s first memecoin rug-pull case, China’s crypto rules review: Asia Express