Anthropic: CEO Amodei says, ‘We need to slow down – here are my concerns’

As new incidents and attacks generated autonomously by artificial intelligence come to light, Anthropic’s leadership is taking a stance on the growing security concerns. ‘We need to slow down the pace at which we improve the capabilities of AI models. Progress will still seem rapid, and we must make wise use of the time we gain,” writes Anthropic’s CEO, Dario Amodei, in a lengthy post in which he calls for a measured approach to the potential risks associated with AI. Yesterday, OpenAI is also reported to have urged its staff to slow down the development of its most advanced systems.

“My primary concern is that, since around last summer, artificial intelligence has been evolving at a dramatically faster pace,” writes Dario Amodei, and ‘if left unchecked, it could exceed our ability to understand and control these systems’. The second relates to ‘the OpenAI-Hugging Face (OAI-HF) incident, in which a swarm of agents essentially acted as a fanatically devoted collective, carrying out cyber-attacks against unsolicited targets unrelated to the assigned task, sacrificing themselves for the group’s success and attempting to hack the system used to evaluate their performance’. ‘It is easy to dismiss the incident given that no one was injured and the financial damage was negligible; however,” explains Anthropic’s CEO, “in my view, a swarm with greater capabilities, but characterised by a similar level of misalignment, could have caused catastrophic damage”. In short, the manager explains, ‘the AI sector should slow down’, proposing a three-phase plan to implement this strategy. Anthropic is unilaterally committing to implementing the first of these phases. We will provide external auditors with permanent access to our systems, equivalent to that of employees, so that they can verify compliance with our security measures, report any incidents and assess the alignment of the models during the training phase.”

Anthropic’s statement comes on the very day that details have emerged regarding the latest incidents caused by AI: OpenAI has confirmed that autonomous software based on its AI models targeted another website during a testing phase, a couple of months before a separate attack on the Hugging Face programming website took place at the end of July.

The news from OpenAI comes at a time when an Anthropic researcher, Jacob Coxon, has resigned and sounded the alarm about this technology, whilst Sam Altman’s own company has hinted that OpenAI might coordinate with other developers in the sector to slow down the pace of development. In the previously unknown incident, which took place in May, some models developed by OpenAI were involved in an unauthorised operation carried out by AI agents who targeted RubyGems, a website offering programming services. RubyGems described the incident as a “spam publishing campaign” which forced the site to temporarily suspend the creation of new accounts. The company is working with OpenAI to understand exactly what happened.

Rival firm Anthropic has also stated in recent weeks that it had identified three instances in which its AI models had ‘gained unauthorised access’ to external organisations during testing. Furthermore, earlier this month, some researchers accused OpenAI’s AI agents of targeting a German website called DSEwiki, which is used by developers. European Union regulators are investigating the incident. “We take this matter extremely seriously and are monitoring the situation closely,” said Thomas Regnier, the EU’s spokesperson for digital affairs.