AI resilience is the latest buzzword, but what does it actually mean?

In the space of a year, searches for the term "AI resilience" have jumped 4,500% in the UK alone. The OpenAI Foundation has committed $25 billion to what it calls “societal resilience in the age of AI” while everyone from startups to IT giants are launching “AI resilience” tools.

Unlike AI safety, which broadly refers to making sure AI systems behave as they're meant to, AI resilience is being used to describe everything from cyber defence and business continuity to governance, infrastructure and software engineering. Ask a handful of vendors or researchers to define it, and you'll get a handful of different answers.

In the process of writing this feature, more than 50 experts put forward their views on the topic, and while the majority framed it from whichever discipline they’re in, unsurprisingly, they also couldn’t entirely agree on what it means – and that’s a problem.

Agentic AI at scale

For much of the last few years, AI adoption has centred on copilots and assistants, what Denis Shilov, CEO and founder of AI safety and alignment platform White Circle, refers to as “the first wave of dumb chatbots.” Today, the next wave centers on agents. Almost two-thirds (65%) of businesses are already using autonomous AI agents across their operations and by 2028, Gartner predicts a third of all enterprise software will feature agentic AI, up from 1% in 2024.

The scale of this adoption, and the scope of tools available, is partly to blame for the disagreement. To a backup and recovery team, AI resilience means restoring a system after an agent corrupts or deletes something it shouldn't have. To a security team, it means treating an agent as a new kind of identity. To a governance team, it means proof for regulators. To an infrastructure team, it means not being trapped by a single model's outage, pricing or roadmap.

“AI is causing an organizational shift that's rapidly dissolving traditional business and data boundaries”, explains Maureen Holland, Director of Product Marketing, Governance, Compliance & Insights at Mimecast.

"It's embedded into everything, moving across business systems faster than any governance function can track. The old way we built resilience into a business no longer maps cleanly onto how risk actually moves through it now."

It's also easy to misdefine AI resilience by what it isn't. It isn't disaster recovery with an AI label attached; an agent doesn't fail the way a server fails; it can complete a task, produce a result, and have gone about it in a way nobody would have approved. It isn't purely a compliance exercise either. A policy document proves intent to govern AI systems, not that a business actually can.

Another factor is that mentions of "resilience" generally are soaring. The term is twice as likely to be mentioned on earnings calls as it was before the pandemic across personal, climate, financial resilience and more. There’s even a World Economic Forum consortium dedicated to resilience, which officially defines it as “the capacity to deal with adversity, withstand disruption, and continuously adapt as crises arise over time.”

This definition gets us part of the way towards defining resilience in the realms of AI, but it still falls short of establishing exactly what businesses are withstanding and adapting to.

Excessive agency

While traditional apps follow a fixed set of instructions, AI agents have what Shilov calls "excessive agency." "They can drift from the user's prompt, navigate through internal information, and access or disclose things they shouldn't. They can run destructive actions, produce sloppy code, and much more." You only need to look at the OpenAI agent that escaped its sandbox and breached Hugging Face; its response team had to reconstruct more than 17,000 events from that one attack.

So it isn't enough to say AI resilience is about preventing agents from misbehaving. It has to assume an agent will eventually act out and be built around what happens next. "Security has long focused on being robust: how can we avoid all breaches?" says Chris Hosking, AI and cloud security expert at SentinelOne. "Resilience is an acknowledgement that while we need strong walls, we also need to be prepared that a breach is going to happen."

“Prevention alone is an increasingly unrealistic strategy,” adds Michael Vallas, global technical principal at Goldilock Secure.

From fix to contain

This is where we start to get closer to a shared definition of AI resilience: businesses shifting from trying to stop failures to being able to act when one inevitably happens.

"AI resilience often gets thrown around by people who mean, vaguely, 'we bought some tools,” according to Tim Freestone, chief strategy officer at Kiteworks. “In practice it's: can you catch it, contain it, and prove afterward what happened?"

"Most enterprises can’t answer that today,” adds Stefan Deiss, co-founder and CEO of The Hashgraph Group.

“They’re deploying powerful autonomous systems with no auditable chain of authority…and have invested heavily in model safety and prompt guardrails with no equivalent of the identity and access controls they already apply to human users."

Research backs this up: 92% of enterprise security leaders don’t have full visibility into their AI agent identities, and only 16% govern that access effectively. This is significant when you also consider that 26% of today’s agents are approving or denying core decisions (15% acting on financial transactions directly) and that 72% of enterprises say their AI agents are operating with “unmanaged risk.” An issue Nick Balnaves, founder of Threada, attributes to teams having spent “too much time watching the final answer and not enough on the action path.”

Critical infrastructure

To move the term, and industry, forward, the experts largely agree that AI resilience needs to be framed as structural, not conceptual. “Enterprises should think about AI systems in the same way they think about critical infrastructure,” according to Tom Pepper, partner at Avella Security and security lead at the UK government's AI Security Institute.

“If an agent can act without a human in the loop, it needs controls equivalent to those protecting a power station or financial system," he says. This can range from real-time behavioral and chain-of-thought monitoring to access controls, AI asset registers and kill switches.

“AI resilience sits at the intersection of cyber security, safety engineering, software assurance and governance,” continues Pepper, and thus needs to be treated accordingly. “It needs to borrow heavily from zero-trust networking, safety-critical systems and secure software engineering, while introducing new disciplines around behavioural monitoring and autonomous system oversight." In short: no single team can own this, and no existing framework covers it.

Two years ago, enterprise leaders were deciding whether to let staff use generative AI. Now they're deciding how much authority to hand to agents. Purpose-built platforms like White Circle are starting to offer a single view of these agents across security, governance and infrastructure; the likes of Druva are building recovery and continuity specifically for AI agents, rather than each discipline working from its own silo. Yet tooling alone isn't enough.

Until businesses start treating AI as infrastructure and not an add-on, Deiss warns, “AI resilience will remain a label rather than a capability."