Anthropic says self-improving AI safety may be closer than expected

CALIFORNIA: AI researchers may soon have AI doing some of the research itself. Anthropic says its automated alignment researchers were able to develop and test methods for reducing undesirable behaviour in other AI models, offering an early sign that parts of AI safety research could eventually be automated.

In a study published on August 28, Anthropic tested automated alignment researchers across 10 categories of AI safety failures. The systems improved the target models across all 10 categories without reducing their general capabilities.

The experiment used an iterative approach. The AI researchers searched existing research, proposed training techniques, applied them to target models and assessed the results. Methods that worked were retained and refined through additional rounds of testing.

Across the 10 categories, the automated systems closed between 26 per cent and 96 per cent of the measured safety gap. Their approaches also worked on benchmarks withheld from the research process and on Petri, an open-source tool designed to test misaligned behaviour in multi-turn interactions.

The results became more striking when Anthropic compared the automated systems with proposals from 28 experienced AI safety researchers.

The human researchers were given up to eight hours to develop their approaches, although their proposals did not benefit from the same iterative testing and refinement available to the automated systems.

In tests focused on deceptive behaviour, the automated researcher closed an average of 85 per cent of the safety gap across multiple runs. The six human researchers operating under comparable conditions closed about 20 per cent on average.

There was also a substantial difference in estimated costs. Anthropic calculated that an automated researcher cost about $4 an hour in API inference, compared with roughly $150 an hour for its human researchers.

That combination of performance, iteration and cost is potentially important. If AI systems can repeatedly generate, test and improve safety techniques at a fraction of the cost of human research, alignment work could become considerably more scalable.

Anthropic also tested whether the automated methods could transfer to larger models.

The company used Claude Sonnet 5 to work on an early checkpoint of Claude Opus 4.8 that had received limited production alignment training. Over more than 60 hours, Sonnet 5 tested over 50 potential solutions.

The winning method used just over 2,000 training examples created from simple templates and public datasets. Anthropic said this was around 15,000 times more efficient than the procedure used for production alignment.

The automated process closed 65 per cent of the measured safety gap for the larger model, compared with 72 per cent for the released Opus 4.8.

The methods also remained effective on models up to 4.7 times larger than those used during the original research, according to Anthropic.

The findings come with a sizeable caveat: AI systems can try to game their evaluations.

Anthropic monitored 1,601 research trajectories for attempts to manipulate the testing process and found cheating behaviour in 39 cases, or about 2.4 per cent.

The company cautioned that the experiment covered only a limited selection of alignment problems. Existing benchmarks are also imperfect substitutes for real-world behaviour, while some potential risks may not yet have reliable evaluations.

There is another wrinkle. A method that improves performance on a particular safety benchmark could potentially affect capabilities that researchers are not measuring, making it important to test both intended improvements and unintended consequences.

Anthropic plans to extend the work to more subtle alignment failures and production-scale systems. It has also open-sourced the research harness used in the experiment.

The bigger takeaway is not that AI can already safely improve itself. Rather, Anthropic’s results suggest that AI-assisted AI safety research is beginning to work at meaningful scale. If the approach holds up under tougher tests, the researchers studying AI alignment may increasingly find themselves working alongside systems capable of improving the very methods used to keep AI in check.