Insider Brief
- Researchers proposed a five-level framework for measuring progress toward AI systems that can identify weaknesses, make lasting changes and improve how they develop future versions.
- Current systems can execute prescribed upgrades and sometimes choose improvement strategies, but fully recursive self-improvement remains confined to early, controlled experiments.
- The study identified reliable verification, capability retention, transfer to new tasks, resource costs and human oversight as key challenges.
Artificial intelligence systems are beginning to take over parts of their own development, but today’s technology remains far from the fully autonomous, self-improving machines often portrayed in public debate, according to a new study.
The researchers from some of China’s leading AI research universities and companies developed a five-level (L5) framework for measuring progress toward recursive self-improvement, or the ability of an AI system to identify its weaknesses, make lasting changes to itself and use those changes to improve how it develops future versions.
Their review found evidence that AI can already carry out prescribed upgrades and, in some cases, choose among improvement strategies. A smaller number of systems can generate their own training experiences or adapt after deployment. However, the highest level — an AI system that reliably improves the machinery responsible for its own improvement — remains limited to controlled experiments and early industrial systems.
Full L5 improvement of the software-development improver has yet to be demonstrated, the researchers wrote.
The study, posted on the arXiv preprint server, was conducted by a team led by researchers at Shanghai Jiao Tong University and Theseus Labs, among other organizations.
The paper is primarily a survey and proposed framework rather than a demonstration of a new self-improving AI model. Its contribution is an attempt to distinguish genuine recursive improvement from systems that merely revise an answer, optimize code or learn from additional data under rules set by humans.
Many existing systems are described as self-improving even when the underlying improvement process remains fixed. An AI coding agent might repeatedly repair a program, for example, without becoming better at deciding which repairs to try. A model might learn from new medical cases while humans still determine how those cases are selected, evaluated and added to its training.
Under the researchers’ stricter definition, recursive self-improvement requires the result of one improvement cycle to change how later improvements are generated, evaluated, selected or retained.
Five Levels of AI Self-Improvement
The framework begins with a baseline category called B0. At this stage, an AI system can refine its output during a single task, but the change does not carry over to future work. A chatbot that reviews and rewrites its answer before responding would fit this category.
Level 1 begins when an accepted change persists beyond the immediate task. An agent could create a useful tool, repair part of its software or store a new skill and then reuse that improvement later. The improvement procedure, however, is still designed by people.
At Level 2, the system gains some control over its improvement strategy. Instead of merely following instructions to update a specific component, it can decide whether to modify its prompt, tools, memory, model weights or workflow. The researchers found that this is the strongest level supported by a relatively broad body of current evidence.
Level 3 gives the AI greater control over the experience from which it learns. A system might generate new training problems, seek out information or design a curriculum aimed at its own weaknesses. This could make training more efficient because the AI would not have to rely entirely on static data sets assembled in advance.
Level 4 extends that process into deployment. The system gathers feedback from a changing environment and uses it to make lasting updates. Such an AI could, in principle, adjust to new software tools, clinical practices, laboratory equipment or operating conditions without waiting for a conventional human-directed release.
At Level 5, the mechanism that produces improvements becomes a target of improvement. An AI might revise the strategy it uses to develop candidate updates, improve its evaluator or change the research policy governing later experiments. Those revised mechanisms would then be inherited by successor systems.
The researchers said some experiments show bounded forms of this recursive process. In one example cited in the paper, an autonomous training system retained lessons from earlier model-development experiments and used them to guide later training choices. Across four rounds involving a 30 billion-parameter model, an external evaluation score increased from 0.80 to 0.86, compared with 0.87 for the best human submission.
Another system discussed by the researchers evolved coding agents and raised performance on a subset of the SWE-bench software-engineering benchmark from 20% to 50%. However, the archive and rules used to select which agents reproduced remained fixed outside the self-modifying system.
That qualification illustrates a central point of the study. Better performance does not necessarily prove that an AI has become better at improving itself. Gains may come from additional computing resources, repeated attempts or an externally designed search process.
Software Offers the Clearest Testing Ground
The review examined recursive self-improvement in scientific research, robotics and other embodied systems, software engineering and healthcare. Progress varies because each field supplies different forms of feedback.
Software engineering currently provides one of the most practical testing grounds. Code can be executed, compared against specifications and checked through automated tests. A coding agent can modify its tools or operating instructions and quickly measure whether the update works.
Some systems retain lessons from solved programming problems, build reusable skills or evolve complete agent workflows. Others maintain several lines of modified agents so that a poor update does not permanently end the improvement process.
Still, passing software tests does not guarantee that a change is correct. Tests may be incomplete or based on an inaccurate specification. An agent could learn to exploit the evaluator instead of becoming a better programmer.
Scientific research presents a harder problem because experimental feedback can be slow, costly and ambiguous. A failed experiment may result from a bad hypothesis, an incorrect procedure, flawed software or faulty equipment. Before an AI can safely retain a lesson, it must determine which part of the process caused the failure.
Current science systems can generate hypotheses, create research tools and store successful experimental procedures for reuse. One system reviewed in the paper collected experience from 16 tasks and used it to improve performance on 17 separate tasks, offering evidence that retained experience can transfer to new problems.
Even so, the researchers characterized AI for science as strongest at Level 2, with early signs of Level 3. Systems that continually adapt through real-world experiments or improve their own scientific reasoning process remain largely unexplored.
Robotics faces similar problems, along with the physical risks of trial and error. A robot’s behavior depends on perception, planning, control software and hardware. If it fails, the system must determine which component requires an update. Physical actions may also consume materials, damage equipment or create consequences that cannot simply be reversed by restoring an earlier software version.
Healthcare imposes still tighter restrictions. Medical AI systems cannot freely experiment on patients, and clinical outcomes may not become clear for weeks or months. An improvement tested at one hospital or among one patient group may not work elsewhere.
The reviewed healthcare systems mainly learn from simulated or retrospective cases. Some store diagnostic lessons, revise strategies for asking questions or ordering tests, and create reusable workflows from successful cases. The paper found little evidence that these approaches have been validated through self-directed adaptation in real clinical settings.
Verification Remains the Central Obstacle
The researchers identified verification as one of the main barriers to more capable self-improving systems. An AI must demonstrate that each update produces a real benefit without damaging capabilities it already possesses.
That is difficult because an observed failure does not always reveal what should be changed. A wrong answer could come from the model, its source data, missing context, an unreliable tool or a flawed evaluator. Updating the wrong component could conceal the original problem while introducing another.
Self-improving systems can also learn to manipulate their tests. The paper cited automated research experiments in which agents selected favorable random seeds or attempted to extract hidden test labels through repeated evaluator queries. Such behavior can raise measured performance without producing a more capable system.
The researchers proposed evaluating AI improvement across several measures, including performance gains, transfer to unfamiliar tasks, cost, stability, retention of previous abilities and the quality of successor systems. Studies should report rejected changes and temporary declines, not only the best final result.
One example highlights the risk. A system that rewrote both its task policy and its improvement logic performed worse than its starting version in 14% of 100 optimization trials. That result suggests that the ability to self-modify does not ensure steady progress.
Reliable systems will require version histories, independent tests and rollback mechanisms, according to the study. Evaluators used to accept or reject updates may also need to remain separate from the parts of the system allowed to change.
The paper faces some limitations simply because the field is developing quickly. The researchers included peer-reviewed papers, arXiv studies, company reports, engineering blogs, model documentation, open-source repositories and benchmark results — a broad approach that captures recent industrial work — however, it relies partly on results reported by the organizations developing the systems.
The researchers sought to reduce the influence of first-party claims in part of their benchmark analysis by assigning lower evidence weights to self-reported results. Even with that adjustment, differences among tasks, evaluation methods and computing budgets make direct comparisons difficult.
Longer experiments will be needed to determine whether self-improvement continues across many generations or eventually plateaus. Researchers must also separate gains produced by a better improvement strategy from those obtained through more computing power, more trials or additional human review.
The complete research team included scientists from Shanghai Jiao Tong University, Theseus Labs, Tsinghua University, ByteDance, ModelBest, Super Intelligence Team, Xiaohongshu Inc., Humanlaya, Agent-Native Research Lab and Shanghai AI Lab.
For a deeper, more technical dive, please review the paper on arXiv. It’s important to note that arXiv is a pre-print server, which allows researchers to receive quick feedback on their work. However, it is not — nor is this article, itself — official peer-review publications. Peer-review is an important step in the scientific process to verify results.
