There’s a new topic of discussion within the global AI community which has ignited a fresh fear in users and founders – is AI developing too fast and will it make humans obsolete?
In recent days, both OpenAI and Claude’s parent company, Anthropic, have publicly identified recursive self-improvement as a credible frontier risk in AI development.
Recursive self-improvement refers to a scenario where AI systems become capable of meaningfully contributing to their own development cycles, reducing the intervals between capability upgrades and potentially outpacing human oversight and control.
This has alarmed many within the industry with questions being raised about adequate and effective guardrails against AI going rogue.
“Inside the frontier labs, models now help write training code, propose experiments, and speed up parts of their own development, and that is real,” said Kurt Muehmel, head of AI strategy at Dataiku. “It is also narrow and supervised, a long way from a system that rewrites and redeploys itself with nobody in the loop.”
A pattern of rapid shifts in development
Levent Ergin, chief AI strategist at Informatica, feels the pace of recent disruptions offers a lesson for founders and developers. The release of ChatGPT in late 2022, the emergence of DeepSeek, and the arrival of Anthropic’s Claude Mythos model each reshaped industry assumptions within weeks highlighting the momentum behind AI development.
“These weren’t slow burns,” Ergin said. “They were quantum leaps. Both Anthropic and OpenAI have publicly identified recursive self-improvement as a credible frontier risk, making it something that warrants preparation now. The mistake organisations and regulators make is waiting for a capability to be fully proven before building governance around it. By then, you’re already behind.”
Innovation driven by machines, not humans?
If self-improvement does materially accelerate AI development, the consequences for innovation timelines would be significant. Previous technological revolutions required decades of organizational adaptation before productivity gains showed up in economic data. AI has covered that journey at the speed of a neuron and recursive self-improvement is just going to speed up that transition.
Muehmel pointed to US Census data showing only 17.5 percent of American businesses use AI in any function as evidence that the binding constraint has already shifted. “Model capability stopped being the limiting factor,” he said. “The constraint that decides whether any of this reaches the economy is enterprise diffusion, whether an organisation can put these systems into real workflows, govern them, and trust them.”
His research with 600 chief information officers found 87 percent are already running AI agents in business workflows, while only a quarter can monitor all of them in real time.
The real risk might be elsewhere
Experts caution that the more immediate risk is proper regulation of existing automation systems, rather than AI models replicating or improving themselves.
“Recursive self-improvement deserves serious attention from the labs and from the people who regulate them,” Muehmel said. “For everyone else, the thing that will shape daily life is more mundane and more certain: capable autonomous systems getting embedded in lending, benefits, hiring, logistics, and care, faster than the oversight around them matures. The frontier is a few labs’ problem. Diffusion is everyone’s.”
Along with this, there is automation bias which is leading to human decision making to take a backseat to AI recommendations, without applying independent scrutiny and foresight.
“Picture a competent machine that people stop checking,” Muehmel said. “That is the real exposure, quieter than the hostile-machine story and far more likely. When a system is right often enough, human review turns into rubber-stamping, and the skill to catch the wrong answer erodes.”
Accountability and governance in AI
If decision making is going to be outsourced to AI systems then a framework of explicitly mapped and defined accountability is going to become essential. For Ergin, developers are answerable for the systems they build and test. Similarly operators are responsible for deployment and monitoring, institutions for governance and humans for the final point of approval.
But this can work only with proper, and effective, empowerment and delegation of authority.
“A junior employee rubber-stamping an AI recommendation that carries multimillion-dollar consequences is a failure of process design,” he said. “The accountability was nominally present and practically absent.”
Muehmel framed it similarly: “Treat a recommendation as an input to a decision. The institution that deploys the system and the person who approved the action own the outcome. Accountability for a public or commercial decision cannot be handed off to the thing that suggested it.”
Ergin offered a summary principle for the period ahead. “Governance, done right, is not a brake,” he said. “It is what separates acceleration you can sustain from acceleration that eventually fails publicly.”
