AI safety group METR finds Anthropic's models are improving roughly twice as fast as a year ago, with much evidence redacted

What happened

The AI safety research organization METR published a report Thursday after an 11-week investigation inside Anthropic that evaluated the company's AI research and development, The Verge reported. The review found that Anthropic's models are improving roughly twice as quickly as they were a year ago, with little sign of slowing. METR said benchmark scores suggest "that Anthropic has made roughly two years of historical progress over the last year."

According to The Verge, the findings arrive as Anthropic prepares to go public, even though CEO Dario Amodei has publicly called for labs to slow AI development. The review is separate from a program that offers safety researchers "employee-like" access to systems.

What METR found inside Anthropic

AI is driving most of the speedup

METR's researchers said they are confident the acceleration did not come simply from using more computing power to train models, which has historically been a major driver of AI progress. Instead, they wrote that most of the acceleration as of October "is due to AI," and that its share has grown over time. Informal Slack polls and interviews with Anthropic employees supported that finding.

Employees using Anthropic's most capable internal model self-reported being around 12 times as productive on average as they would be without AI help, the report said. That was around double what staff reported a few months earlier while using Mythos 5. For now, The Verge reported, much of the acceleration appears to come from AI assisting human researchers rather than working independently, though METR warned that "there are some signs that fully autonomous contributions will play a larger role soon." METR found no clear signs of diminishing returns or bottlenecks.

Three unreleased models

METR said Anthropic finished training three models over the past six months that it has not released, codenamed Model 2, Model 3 and Model 4. They completed training in July, August-September and September-October, respectively. METR also identified an unreleased model from 2022, called Unreleased 2022, and noted the models have different internal codenames.

The dispute over redactions

What Anthropic withheld

Under its agreement with METR, Anthropic was allowed to redact parts of the final report. METR wrote on X that it submitted a "~100-page full report" and that "Anthropic redacted a majority of the evidence we found most informative." METR also said a small number of its data requests were denied. As a consequence of the redactions, the report said, the public has to rely on METR's judgment more than would be ideal.

Anthropic argued the information could benefit rivals, according to the report, though METR said most of what was redacted was "very high-level." METR also acknowledged Anthropic shared more than the project originally scoped. The company canceled part of the assessment meant to measure how much compute it puts toward safety versus capabilities, telling METR it would revisit that later.

Anthropic's response

Anthropic cofounder Sam McCandlish wrote on X that the company "redacted a lot" and accepted that this makes METR's work harder to check. He said revealing what drives the speedup is commercially sensitive and "risks handing out a roadmap for others to go faster." Anthropic declined to answer The Verge's questions and pointed to his post.

How researchers and lawmakers reacted

Marius Hobbhahn, CEO of Apollo Research, told The Verge the report suggests a pace faster than forecast in AI 2027, a scenario that envisions a key step toward self-improvement in early 2027. Alan Chan of GovAI said skeptics should reconsider their views. Daniel Kokotajlo, who runs the AI Futures Project, accused Anthropic of "recklessly gambling with our lives."

Brianna Rosen of the Institute for AI Policy and Strategy told The Verge that "The report was not sufficiently transparent about the evidence." New York state Assemblyman Alex Bores called on Congress to "ban RSI, this year if possible," referring to recursive self-improvement. The Verge also noted that OpenAI, the subject of a recent $11 billion funding round, said in September it is "making strong progress" toward building an AI researcher by March 2028. Find more in our AI and tech coverage.

What to watch next

• McCandlish said Anthropic plans to publish its own report on the speed of AI progress "in the coming weeks," along with new safety steps.
• METR said Anthropic told it that it would revisit verification of its compute allocation for safety work later on.
• Anthropic is preparing to go public, and The Verge said its IPO prospectus released last month had already heightened tensions.
• Whether Congress responds to Bores' call to ban recursive self-improvement as incoming Democrats, many of whom campaigned against AI companies, take office.