‘We May Imminently Meet’ the Threshold: Anthropic Warns of Self-Accelerating AI as Models Start Editing Models and Reading Laws Like Code

‘We May Imminently Meet’ the Threshold: Anthropic Warns of Self-Accelerating AI as Models Start Editing Models and Reading Laws Like Code

“We may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.'” That sentence, added to Anthropic’s revised IPO filing this week, is the closest a frontier lab has come to telling regulators and investors that self-improving AI is near. Anthropic also says it has already used AI to build models whose leaps in capability it “did not identify” at the time. The same week, MIT researchers showed one AI model directly rewriting another model’s weights to make it six times better at maths, and legal scholars warned that AI can now find loopholes in laws the way it finds bugs in code. The thesis is that AI is starting to work on the systems that govern it, both its own training and our rulebooks, faster than humans can check the result. The response can’t wait for the threshold to be crossed.

Deepen your knowledge: Ethics & Law — our analysis of AI regulation, liability and the rights of machines and people.

Go deeper: what the first S-1 disclosed, in ‘Behavior Resembling Blackmail’, and why human oversight is weaker than it looks, in The Human in the Loop Is Being Steered.

What Anthropic now says

In the revised S-1/A, TechCrunch reports, Anthropic adds two statements to its risk factors. First, its “increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures”, and it has “in the past not identified” such advances in time. Second, its “internal estimates and our expectations regarding future models… indicate that we may imminently meet” its own definition of AI-driven acceleration. It stresses that no current model has crossed that threshold and that it does “not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models” without meaningful human involvement.

The same filing shows the scale of the bet. Revenue was $33.2 billion in the first nine months of 2026, up 1,206%, but Anthropic warns it does “not expect” that pace to continue, and growth slowed to 46% between Q2 and Q3, The Register reports. It has nearly $900 billion in compute and infrastructure commitments, potentially more than $1 trillion including estimated payments. About $437 billion of its deals with Microsoft, Google, Amazon and Broadcom are non-cancellable. A company that must grow fast enough to pay a trillion-dollar compute bill has every incentive to use AI to speed up its own research.

Skeptics note that an S-1 is part disclosure and part marketing. Saying you are close to self-improving AI also signals that you are at the frontier. But the language is specific, legally exposed and consistent with what the company’s models have shown this year in coding and security.

AI that edits AI

Separately, MIT CSAIL researchers asked OpenAI’s GPT-6 Astra to edit the weights of a small open model, Qwen3.5-0.8B, directly, with no training run, The Register reports. Astra changed just 0.016% of the weights and improved maths performance by up to six times on MATH-500 and instruction-following roughly threefold. It also wrote, from scratch, the weights of a network that plays Breakout. “Models may treat raw weights like readable, editable code: diagnosing failures and patching them directly,” said PhD student Rikiya Takehi. The authors call it “Weight-Space Programming” and expect it to help with interpretability and alignment, and also with recursive self-improvement.

AI that reads the law like code

The same capability applies to rules. Felix Ringe, a visiting fellow at Stanford Law School, and Linus Folkerts argue that “legal systems encode the foundations of social order in natural language” and, like software, contain vulnerabilities that cheap AI reasoning can find at scale, The Register reports. They cite possible gaps in German law surfaced with LLMs, such as a regulation amendment that appears to omit a required citation of its legal authority, and a transit-visa rule that may be invalid for the same reason. “We believe that with cheap legal reasoning proliferating, AI systems could jeopardize the functioning of today’s legal systems,” they write. Meanwhile, some Los Angeles courts are testing AI as law clerks.

New institutions are appearing to judge AI as well. “Internet Court”, built on the GenLayer network, resolves contract disputes between agents and humans with juries of AI validators running different models, from Claude to DeepSeek, Cointelegraph reports. It handles about 12,000 decisions a day in beta, with escalating appeals to larger juries. It is a crypto-native experiment, but it shows where things are heading: machines arbitrating disputes between machines.

Why this matters before the threshold

Put together, the stories describe a feedback loop. AI improves AI, AI finds weaknesses in human rules, and AI is used to judge AI. Each step is useful. Weight editing could make models more interpretable, legal analysis could harden legislation, and cheap arbitration could settle small disputes. The risk is speed: progress that outruns the checks, which is exactly the risk Anthropic’s filing describes.

That suggests some practical responses for governments and businesses now, not after a lab announces a milestone:

  • Ask labs for evidence, not just thresholds. If a company says acceleration is imminent, regulators and customers should ask what measurements trigger which safeguards, and who verifies them independently.
  • Use AI to defend rules as well as attack them. Legislatures and regulators should run the same loophole-finding analysis on draft laws before they pass, as software teams now scan code before release.
  • Keep humans accountable for automated judgements. AI clerks and AI juries need a human who signs off, and a path to appeal to a person.
  • For businesses: plan for faster model change. If capability can jump between releases, contracts, evaluations and safety reviews should be repeatable rather than one-off.

The read

For years recursive self-improvement was a thought experiment. This week a company preparing a public offering wrote, in a legally binding document, that it may be close to that line, and academics showed both parts of the loop working in miniature. That doesn’t mean a runaway is imminent. Anthropic says humans stay in the loop, and the skeptics have a point about marketing. It does mean the time to build oversight that works at machine speed is now. When the threshold is announced, it will be too late to start designing the checks.

Written for Red Robot with AI assistance and human editing. Based on reporting by TechCrunch, The Register, The Register, The Register and Cointelegraph.

Happy
Happy
0%
Sad
Sad
0%
Excited
Excited
0%
Angry
Angry
0%
Surprise
Surprise
0%
Sleepy
Sleepy
0%
‘Ghost Students’: DOJ Charges Five in AI-Assisted Scheme to Steal Student Aid

‘Ghost Students’: DOJ Charges Five in AI-Assisted Scheme to Steal Student Aid

‘Agents Are the New Interface’: Who Writes the Rules for AI Payments, From Mastercard and OpenAI to Klarna's Disputes

‘Agents Are the New Interface’: Who Writes the Rules for AI Payments, From Mastercard and OpenAI to Klarna’s Disputes

Leave a Reply

Your email address will not be published. Required fields are marked *

7 − 5 =