AI tools such as GitHub Copilot, Gemini Code Assist, and Claude can be used for powerful programming assistance – but security leaders are concerned about the risks they pose. Security leaders in the US, UK, Germany, and France are worried that the use of AI-generated code within their organization could lead to a security incident, according to recent research.
With the ability to boost efficiency and productivity, it’s no surprise that the use of AI tools for coding is an increasingly common practice. According to a recent report by cybersecurity firm Venafi, 83% of firms are already using AI to generate code, with 57% using AI coding tools as standard practice.
Against this backdrop, chief information security officers (CISOs) are concerned about the questionable integrity of code produced with generative AI, as well as a lack of oversight into when and where the technology is being used. According to the report, 92% of security leaders are concerned about the extent to which developers are already using AI code within their companies.
Without human oversight, AI-generated code can contain bugs or introduce security vulnerabilities. As the issue threatens to get out of control, what can CISOs do about it?
AI-generated code is becoming common
There’s no doubt AI-generated code is on the rise. StackOverflow’s 2024 Developer Survey found 76% of respondents use or plan to use AI for development in the next year and Amazon CEO Andy Jassy has publicly stated that its own AI assistant has saved $260 million and 4,500 developer years of work.
Anton Osika, CEO and co-founder of AI stack developer Lovable, says that AI use is already widespread throughout development cycles. “Managers might not know it, but their employees are pasting in code from ChatGPT, even if the company policy tells them not to.”
This shadow AI use is being driven by the efficiency gains provided by AI tools. Many developers find AI coding assistants such as GitHub Copilot useful. “They can be used for quickly scaffolding projects or knocking out routine bits of code and they can be great time savers,” says Camden Woollven, group head of AI at GRC International Group.
However, AI-generated code poses multiple security risks. One of the main issues with AI-generated code is that it can contain bugs or introduce security vulnerabilities, especially when the AI lacks the proper context, says Loris Degioanni, chief technology officer and co-founder at Sysdig .“While this is expected to improve as large language models (LLMs) become more advanced, there is still some risk in the short term.”
The chance of vulnerabilities and malware being introduced is “significantly higher” than when code is written by humans says Shobhit Gautam, security solutions architect at HackerOne, who stresses that AI-generated code is not guaranteed to follow security guidelines and best practices. “As the code is generated from LLMs, there is a possibility that third-party components may be used and go unnoticed,” he explains.
Another, perhaps bigger concern, is that developers might become less engaged with the underlying codebase, says Degioanni. “By relying heavily on AI tools, developers may lose a deep understanding of the code, which could result in long-term maintainability challenges. When developers don't fully understand the code they are working on, there is a greater chance of introducing defects, including security vulnerabilities, especially when changes are required in the future.”
At the same time, AI models can introduce common vulnerabilities such as SQL injection, cross-site scripting (XSS), and more, says Vitaly Simonovich, threat intelligence researcher at Cato Networks. One serious type of vulnerability that can be introduced by AI is business logic vulnerabilities, Simonovich explains, in which an attacker manipulates business rules or processes to their advantage, leading to losses or damages. “These are not straightforward and require deep knowledge of the code base,” says Simonovich.
Another risk involves privacy and intellectual property. In 2023, Samsung engineers made headlines when they unwittingly sent sensitive data to ChatGPT. “ChatGPT could have been trained on this data and exposed Samsung's IP to other users when generating code,” says Simonovich.
Securing AI-generated code
There are certainly risks involved, but it’s difficult to prevent employees from using AI-generated tools altogether. Taking this into account, oversight is important.
First, security leaders need to understand how AI tools are being used in their organization, says Woollven. “This means talking to developers, finding out which tools are popular and how often they're being used. It's only with this insight that they can start crafting sensible policies.”
The tools themselves are important, too. “Really vet the tools you use to ensure they have the ability to use the most up-to-date libraries and coding techniques,” advises Yang Li, co-founder at Cosine.
Woollven concurs. “AI tools are not all equal when it comes to security features or transparency, and understanding these differences can help organizations make informed decisions about which ones to approve for use,” she says.
Another important factor is code reviews, which need to evolve with the increasing use of AI. “Beyond catching the usual bugs, reviewers should keep an eye out for issues specific to AI-generated code,” says Woollven. “It might be worth flagging these snippets for extra scrutiny.”
Training developers is also key. “They need to understand both the benefits and pitfalls of AI coding tools,” according to Woollven. “The goal should be to foster a healthy skepticism – use the tools, but don't rely on them blindly.”
It is undeniable that AI is a highly effective tool. However, it must remain just that: a tool, not a replacement, says Gautam. “Security leaders must ensure their organizations do not become over-reliant on programs purely driven by machine learning and AI. These tools are not infallible, and errors in the generated code can be harder to detect.”
Security leaders must select the right tools for their teams and define clear usage guidelines, adds Degioanni. “Not all AI tools are created equal, and understanding which tools align best with the team's needs and policies is crucial. Firms should put processes in place to monitor how these tools are being used, ensuring compliance with established guidelines and identifying areas for improvement.”
Knowing whether to use private or open source models can help, says Simonovich. “Many companies, such as Microsoft, offer private instances of LLM models that can be used without IP and privacy concerns for organizations. Open-source models can be deployed locally and provide the functionality that developers need in their workflow.”
