NCSC issues urgent warning over growing AI prompt injection risks – here’s what you need to know
Many organizations see prompt injection as just another version of SQL injection - but this is a mistake
The National Cyber Security Centre (NCSC) is warning security teams to be on the lookout for AI prompt injection attacks.
These involve an attacker creating apparently innocent inputs to large language models (LLMs) which take advantage of the model's inability to distinguish between developer-defined prompts and user inputs to cause unintended behaviour.
Prompt injection attacks are often seen as just another version of SQL injection attacks, said NCSC technical director for platforms research David C, with data and instructions being handled incorrectly - but this is a mistake.
In SQL, instructions are something the database engine does and data is something that is stored or used in a query; much the same is true in cross-site scripting and buffer overflows.
Mitigations for these issues enforce this separation between data and instructions. For example, the use of parameterized queries in SQL means the database engine can never interpret it as an instruction, regardless of the input. The right mitigation solves the data/instruction conflation at its root, David C pointed out.
"Under the hood of an LLM, there’s no distinction made between ‘data' or ‘instructions'; there is only ever ‘next token’. When you provide an LLM prompt, it doesn’t understand the text it in the way a person does. It is simply predicting the most likely next token from the text so far," he said.
"As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be."
Sign up today and you will receive a free copy of our Future Focus 2026 report - the leading resource for IT decision-maker insight on priorities and investment areas in AI, security and more.
Security teams should stop treating prompt injection as a form of code injection, but instead view it as an exploitation of an ‘inherently confusable deputy’.
This is where a system can be coerced to perform a function that benefits the attacker, typically where a privileged component is coerced into making a request on behalf of a less-privileged attacker.
"Crucially, a classical confused deputy vulnerability can be mitigated, whilst I’d argue LLMs are ‘inherently confusable’ as the risk can’t be mitigated," said David C.
"Rather than hoping we can apply a mitigation that fixes prompt injection, we instead need to approach it by seeking to reduce the risk and the impact. If the system’s security cannot tolerate the remaining risk, it may not be a good use case for LLMs."
AI prompt injection attacks are rising
Prompt injection attacks have become a recurring talking point over the last three years, with security experts warning about the potential for threat actors to manipulate AI models into producing malicious outputs.
Pete Luban, field CISO at AttackIQ, said the NCSC advice should be taken seriously by enterprises using the technology.
However, just because AI prompt injection attacks can't be mitigated in the same way as SQL injection attacks, Luban said this doesn't mean that lessons can't be learned from SQL injection defense.
"Developers need to build systems around LLMs with the awareness that prompt injection attacks are a threatening class of vulnerability. Since these attacks cannot be handled with a single product or appliance, careful design and operation is paramount to preventing exploitation," he said.
"Security teams should understand that all known methods of prompt injection prevention can only reduce chances of an attack or breach. Updating and strengthening their overall security posture, including continuously monitoring systems for irregularities and testing against common adversarial tactics, can help systems identify earlier stages of an attack and quickly move to mitigate it."
Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.
MORE FROM ITPRO
- AI-enabled cyber attacks exacerbated by digital divide in UK
- Cyber experts have been warning about AI-powered DDoS attacks – now they’re becoming a reality
- Generative AI attacks are accelerating at an alarming rate
Emma Woollacott is a freelance journalist writing for publications including the BBC, Private Eye, Forbes, Raconteur and specialist technology titles.
-
UK tech leaders are implementing AI faster than related governanceNews IBM research shows that chief AI officers are being given strategy roles, but not full control
-
11:11 Systems appoints new EVP in EMEA growth driveNews The experienced executive will oversee the provider’s operations and strategic execution across the region as it targets further growth
-
Looking for workarounds: Why some employees turn to shadow AI when outages hit – and how IT leaders can stop itNews IT leaders need to accept that staff will try to find workarounds. It’s up to them to set clear ground rules
-
OpenAI says its new cyber-focused AI model found thousands of Linux kernel vulnerabilitiesNews The new cyber-focused AI model boasts improved bug hunting capabilities compared to its predecessor
-
‘This is a massive change’: PagerDuty wants to supercharge incident response with a new SRE agent and command centerNews The new Paige agent from PagerDuty will act as a digital assistant for teams battling incidents
-
‘AI’s price tag is starting to shape its path through the enterprise’: 89% of enterprises have scaled back AI deployments over token costs – and they’re still struggling to track spendingNews Volatile token costs across 2026 have caused major headaches for IT leaders
-
88% of UK firms have had an “agentic security incident” in the last year – ServiceNow warns more will comeNews Concerns are growing over the scale of adoption and an inability to match “machine-speed” capabilities
-
Microsoft wants to tackle the biggest issue for UK AI adoption: confidenceNews UK and Ireland CEO Darren Hardman says it's our responsibility to get AI into every corner of the UK
-
Microsoft says AI is causing a workforce rupture among British businesses – leaders are backing the technology, but workers still aren’t soldNews New Microsoft research shows a huge disconnect between leadership and employees when it comes to confidence in the UK leading in AI
-
‘Rising threats and under-resourcing for cybersecurity is taking a toll on the people tasked with managing it’: Cyber teams are being pushed to breaking point – and AI is doing little to alleviate strainNews An increasingly perilous threat landscape and lack of funding means cyber teams are being stretched too thin