Leading AI researchers warn automated AI research poses extreme risks

September 28, 2026 at 08:54 PM2 min

This digest was compiled by AI from multiple sources — links to the originals are below.

More than 20 leading AI researchers, including Geoffrey Hinton, Yoshua Bengio, and OpenAI research lead Jakub Pachocki, warn that automated AI research could trigger an 'intelligence explosion' with extreme risks. In a new paper, they say capabilities growth could accelerate beyond society's ability to keep up, humanity could lose control over superhuman AI systems, and checks on power could erode. The researchers urge policymakers to develop ways to steer and constrain such an explosion and prepare society for its impacts.

Key Facts

More than 20 AI researchers, including Geoffrey Hinton, Yoshua Bengio, and OpenAI research lead Jakub Pachocki, signed the paper warning of extreme risks from automated AI research.
The paper states that AI systems already write most of the code at the companies building them and could automate the entire AI R&D pipeline within years.
The researchers warn that progress that normally takes years could then happen in months, and society might not keep up.
The paper calls on policymakers to gain far more visibility into how AI research is being automated and to develop ways to steer and constrain an intelligence explosion.
The paper joins a growing list of warnings, including a recent call from 42 leading mathematicians for more attention to existential AI risks.

The Warning

More than 20 AI researchers, including Geoffrey Hinton, Yoshua Bengio, and OpenAI research lead Jakub Pachocki, warn of a possible 'intelligence explosion' driven by self-improving AI in a new research paper. The paper states that 'although there remains much uncertainty, AI R&D automation might soon trigger one.' According to the study, AI systems already write most of the code at the companies building them and could automate the entire AI R&D pipeline within years. Progress that normally takes years could then happen in months.

Societal Risks

The authors warn society might not keep up, control over superhuman AI could slip away, and power balances between nations, companies, and governments could erode. The paper says that 'capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded.' They urge policymakers to gain far more visibility into how AI research is being automated.

Policy Recommendations

The paper says policymakers should develop ways to steer and constrain an intelligence explosion and prepare society to adapt to its impacts. The paper joins a growing list of warnings, including a recent call from 42 leading mathematicians for more attention to existential AI risks. Several AI lab employees have voiced fears that AI could destroy humanity, and Pachocki himself said no lab has solved alignment well enough 'to continue responsibly scaling at maximum speed for much longer.' Some Anthropic employees are reportedly already scouting safe havens in case AI goes sideways.