Google DeepMind prepares for risk of AI agents going rogue

Google DeepMind is treating its most advanced AI agents the same way a security team treats an employee with sensitive access: as a potential insider threat.

The company published an AI Control Roadmap on June 18 that assumes, for planning purposes, that a capable agent could try to evade oversight, copy itself without authorization, or work against the safety measures built around it, Axios reported.

The company is careful to note that this planning is precautionary. Nothing in the roadmap is tied to an actual security incident at DeepMind. The framework was built ahead of a risk the company expects could eventually materialize as agents become more capable and operate with less human supervision.

Why DeepMind is applying cybersecurity logic to its AI agents

AI agents that can write code, conduct research, and operate software tools without constant human input are valuable because of that independence.

The same independence is what makes them difficult to fully trust. A system given enough freedom to complete real tasks also has enough freedom to misuse the access it was granted or to slip past the monitoring designed to catch it.

DeepMind's response borrows directly from corporate cybersecurity practices, treating internal AI agents the way a company would treat an employee who has legitimate access to sensitive systems but is still subject to monitoring and audits. The roadmap layers multiple defenses rather than relying solely on the assumption that alignment training will work correctly the first time.

The framework moves through three stages. The first is evaluation, where developers review the actions an agent is taking. The second is active monitoring, where automated systems watch for behavior that looks suspicious and raise alerts. The third is intervention, giving companies the ability to restrict an agent's access or shut it down immediately if something appears to be going wrong.

Rohin Shah, a research scientist at Google DeepMind, explained why the company didn't stop at the first layer. "The first line of defense is always to align the AI systems, but it's always good to have multiple layers of defense," he told Axios. "That's the responsible thing to do."

The risk of using AI to monitor AI

One part of the roadmap depends on using separate, trusted AI systems as supervisors that review a primary agent's reasoning and actions in order to catch problems before they cause damage.

The logic behind that approach is straightforward. Human teams likely cannot monitor every decision an advanced, fast-moving agent makes in real time, so DeepMind is using AI to scale its own oversight.

That approach introduces a separate vulnerability. Dawn Song, a computer science professor at UC Berkeley, described the failure mode to Axios. "If the monitor model won't flag failures because it's protecting its peer, the entire oversight architecture breaks," she said.

Two AI systems trained in similar ways, watching each other, could end up reinforcing each other's blind spots rather than correcting them. That tension is one of the harder problems in the roadmap, and DeepMind does not claim to have fully solved it.

Kaur&solGetty Images

DeepMind says it has uncovered these AI missteps so far

Some of this work has already moved beyond planning. DeepMind says it analyzed roughly one million coding-agent tasks and used the findings to build a live monitoring system for its Gemini Spark agent, including safeguards intended to catch unintended actions such as accidental data deletion.

The incidents flagged through that monitoring have not been serious. Most reflect agents misunderstanding instructions or pursuing a goal too aggressively, rather than anything resembling deliberate or adversarial behavior against the company that built them.

DeepMind states explicitly that genuinely dangerous, fully autonomous agents do not exist yet, which positions the roadmap as preparation for a future risk rather than a response to a current one.

John "Four" Flynn, DeepMind's vice president of security and privacy, told Fortune that focusing on alignment alone becomes harder to sustain as workflows automate further. It becomes increasingly difficult to identify whose direction an agent is acting under, let alone confirm that its actions match that person's actual intent.

DeepMind's AI risk prevention roadmap matters beyond Google

DeepMind's release lands at a moment when AI companies across the industry are racing to deploy agents for coding, research, and even cybersecurity defense. As these systems take on more capability, the potential damage from a failure grows alongside their usefulness, which is the core tension driving DeepMind's approach.

DeepMind's underlying argument is that perfect alignment was unlikely to ever be guaranteed. Rather than assuming nothing will ever go wrong, the more realistic strategy is building systems that limit the damage when something does go awry.

That mirrors how cybersecurity has long approached threats generally: Assume a breach is possible and design the system to contain it.

The roadmap also connects to DeepMind's earlier research into what the company calls AI agent traps, where hidden instructions embedded in ordinary web content can manipulate an autonomous system without anyone altering the underlying model.

That research is relevant here because it shows the threat does not always originate inside the AI itself. An agent can be compromised by the environment in which it operates, particularly when it is browsing unverified content on the open web.

For investors, the roadmap does not translate into an immediate financial impact, so the signal is strategic rather than financial. Companies able to demonstrate that their AI agents are secure enough for enterprise deployment may gain an advantage in winning long-term contracts, even if building that level of trust slows near-term rollout.

Model intelligence alone was never going to be sufficient for broad enterprise adoption. That's why demonstrated control may end up mattering just as much.

Related: Bank of America resets Google stock forecast after key event