This automatically-generated transcript is taken from the IT Pro Podcast episode 'Can generative AI change security?’. We apologise for any errors.
Jane McCallion
Hello ITPro podcast listeners. This is Jane, one of your co-hosts. We wanted to let you know that this episode was recorded one day before OpenAI announced it would be releasing an API for business for ChatGPT. As you'll hear this is something we discussed as a theoretical during the episode, but it turns out that OpenAI had exactly that idea in mind. Now on with the show.
Rory Bathgate
Hi, I'm Rory Bathgate.
Jane
And I'm Jane McCallion.
Rory
And you're listening to the ITPro podcast where today we're discussing the defence applications of generative AI.
Jane
If you’ve been keeping up with the news since November, you’ve definitely heard the name ChatGPT. The newest chatbot by AI firm OpenAI, it’s capable of powerful text generation and interpretation based on its vast training model, and is just one of a host of AI models being made that can create content, known as ‘generative AI’.
Rory
But as a publicly-available resource, ChatGPT also carries the risk of empowering threat actors. As companies seek to fund development into more advanced large language models and the public grapples with the full potential of chatbots like ChatGPT, Bing Chat, and Bard there are concerns about how to tackle malicious use of generative AI.
Jane
Today we’re speaking to Hanah Darley, head of threat research at cybersecurity firm Darktrace, about the dangers of generative AI, and how it can also play a role in keeping businesses safe from this new wave of threats.
Hanah Darley
Thank you so much for having me, it's lovely to be here.
Jane
So what is the potential for threat actors to use ChatGPT or any other sort of large language model AI, this is only one of them?
Hanah
There's quite a lot of potential. If you think about pretty much any application of AI, it's a resource multiplier, it's a way to increase efficiency. And so in the same way that kind of the good guys and defenders can increase efficiency for security teams by using different AI applications. You can think of it again as kind of helping to lower the bar for threat actors as well. So it's a way to increase efficiency for humans, and of course that can be leveraged by threat actors and applications like phishing or sometimes with code, although there's a lot more management that has to happen with code writing. And so you can see the potential for it to be misused and exploited especially as it's publicly available.
Rory
So when you're talking about this use of models like ChatGPT for phishing, are you talking about creating an entire phishing email or spear phishing campaign through generative AI?
Hanah
Yes, exactly. You could do a myriad of things, you could say "can you please craft for me a very, very well written email targeted towards a banker who regularly reads the Wall Street Journal, and can you include specific details" because ultimately, language modelling is designed off of training data. So as long as you have kind of the right training data involved, you should get an output of sophisticated language, and mostly English language being kind of a key point to make as well. But you should get kind of an output of what looks like a very well-written email. And so you can see, again, the applications for education as well as for corporate writing, helping to improve general email sending, and in the same way that could be exploited to craft a more targeted phishing email. And it's not necessarily the case that that will happen flawlessly because, again, it's based on the kind of training data and the amount of specificity in your request. But yes, you can definitely use it to craft a very well-written phishing email.
Jane
I suppose one way that I could see that playing out, we've spoken before and whenever there's some threat research done, it shows that a lot of campaigns whether that is malware, ransomware, or whatever or phishing campaigns which obviously could be the front end for one of those, come from non-English speaking countries, or countries where English is not their main language or official language. Could it give bad actors for whom English is not their first language a bit of a leg-up in that respect, to use a language model that English is its first language, if you like, and get something that's a bit more as if the person who's supposed to have written it is the person who wrote it?
Hanah
Yeah, I think you could definitely use it for that application. I mean, some of it is going to come across with difficulty in the request itself. So how clearly can you communicate in English on a basic level? Because that will alter the amount of input that you have into the software. But yes, absolutely you could use it to improve your language skills, you could use it to improve your English. I think just thinking out loud, though, one of the biggest things for my role at Darktrace is head of threat research, so it's my job to look across the cyber threat landscape as a whole and to understand not only what are kind of the emerging and evolving threats, but specifically how those threats are impacting customers within Darktrace's fleet. And one thing I would caution is the idea that threat actors only live in a certain region of the world, threat actors can come from anywhere. And some of them speak English very well, perfectly fine. And so I think in terms of a capability builder, it is definitely a capability builder to have this kind of generative AI technology for threat actors. But I don't think that you're going to see a cataclysmic shift in the total language aptitude of threat actors across the board to the point that suddenly there's a massive amount of phishing emails, where before that wasn't the case. Or that suddenly, there's a massive trend in beautifully written phishing emails that weren't possible before. It's definitely a capability multiplier, but I wouldn't say it's a complete world changer in terms of the capability itself.
Rory
When it comes to things like ChatGPT, I know this is the example we've been using but the same is true of a number of popular chatbots and similar AI models. There are supposed to be guidelines in place, there are supposed to be measures that will stop people from generating malicious content, illegal content using them. Is this currently a case of people finding loopholes with chatbots? Or is this them using it for seemingly mundane tasks that add up to a malicious purpose?
Hanah
I mean, without falling into complete conjecture, I think the answer is yes to both, I think you can see examples of both. I watched a really cool example today on LinkedIn of a guy who was able to navigate around ChatGPT's ethical filters, to do exactly what you're asking, he wanted to harvest credentials and gather cookies. I can't remember exactly the request. But effectively, he blurred out the input so that his proof of concept was ethical, but then he showed how he could navigate around it. But I think you can also find much more innocuous applications where you don't have to do all that legwork, and you can still use it as as a capability gain. I think the key thing to remind people of, and one of the things that I think is kind of lost in the ChatGPT conversation in general, is that AI isn't necessarily kind of a one size fits all stamp. And a lot of the benefit of ChatGPT is actually getting it into the public conscience, this is an application of AI and this is an example of it. But actually the AI kind of applications and algorithms and uses are such a spectrum. And language modelling has limitations. So as much as absolutely you can see people navigating around ethical filters, you can see people kind of reverse-engineering ChatGPT and using it that way, in a really more technologically-sophisticated way. Comparatively, you can also see it for much more mundane, simple things like "give me an example list of what an invoice should say if I ordered 50,000 chips", you know that that might also be an application. So but I think just coming back to kind of the broader conversation it has limitations, just like any other AI application. So language modelling, you have to have training data available. So you can't craft out any sort of request out of the air, you have to have the available training data. And then once you have that, yes, you can definitely use it for malicious gain. But it's an application of AI as part of a much broader spectrum. And hopefully the the kind of positive side of ChatGPT is that public consciousness around AI kind of raises because ultimately, sadly, a lot of the kind of general consensus around what AI is kind of stops at Terminator. And people know the Skynet references, that's the limitation for a lot of people in terms of how the public interact with AI. And so I hope that as much as this is a force multiplier for, unfortunately, threat actors as well as Blue Team defenders, I hope that this is also an educational tool for a lot of the general public.
Jane
When it comes to its usefulness for threat actors, when it comes to you potentially crafting a phishing email or code if we're kind of gonna go down that route a little bit. It strikes me that whether or not it's actually going to be that helpful when it comes to getting a correct answer and something that's actually useful. So Stack Overflow, for example, has stopped people from being allowed to submit answers that have used ChatGPT because they come off as authoritative, but are often wrong. And I found this myself. I stuck something and asked it to tell me the difference between British Sign Language and American Sign Language, it sounded very authoritative but the answers that gave were wrong. And so is there a danger, I guess, for the threat actors which is an upside for us, that what this is going to check out for them is going to look like it's what they're looking for. But in reality, it is garbage?
Hanah
That would be a great outcome. I don't know the differential likelihood of that happening. I would love that to be the outcome, it would make my job and much of the Blue Team's job a lot easier. I would say, I think in general threat actors are understanding the importance of trust in a way that heretofore hasn't been at the centre of focus. I think you've seen a lot more focus on the "hack" kind of technological advancements and ways to kind of navigate around security controls. And I think, especially with the multi factor authentication hacks that you saw last year, that there is a lot more awareness in the cyber threat ecosystem of the importance of trust and how if you can just compromise a few trusted services, or even gain someone's trust through email or through known correspondence, you might be able to leverage that without having to do anything as technologically fancy and wonderful as you previously had to, to be competitive in the threat system. So I think one thing that we kind of looked at is the importance of trust. And so when ChatGPT came out, of course a lot of people were asking Darktrace because we do AI, what are you seeing in your own customer base? What impacts has ChatGPT had? And my colleague Toby Lewis actually made a really good point, which was even if on the receiving end of a phishing email it had been generated by ChatGPT, chances are you'd have no idea. Whether you're a security team, or whether you are a user who's receiving that email, there's very few indicators unless they kind of left in some sort of a ChatGPT 'TM' branding somewhere in it, that there's there's been AI involved in this transaction. And so I think what you do see though, is that if a threat actor is able to generate out a higher quality phishing email, then you might see the impact of that. Our statistics internally found that actually, relying on malicious links had decreased from about 22% of the phishing emails we saw it to about 14%. But the average linguistic complexity of phishing emails jumped or increased by about 17% since the outset of ChatGPT. Now, it would be absolutely statistically crazy of me to say that that is a 1:1 correlation. But what I will say is, the importance of making that point is that actually, language does matter in phishing emails in a way that previously just bulk sending might have gotten the job done. But we're seeing a lot more interest in, and focus on, that trust factor. And so by compromising something from either a known correspondent, or even using services through very basic social engineering that the user expects to receive mail from, you get a lot of payout from that. And so any sort of reduction in the barrier of complexity is obviously going to be a massive win as a threat actor. But I think that's where you kind of plug in the importance of anomaly detection, and you plug in the importance of going beyond identified, bad, or watch-listed either IPs or links, or kind of services that are attempting to sift through pattern anomalies. Because what you miss is the human factor, what you miss is the trust, and the services that you do expect to see correspondence from that might be spoofed or compromised. And that's really the crux of the issue when it comes to ChatGPT or services like that with either some sort of language modelling AI application, or large language modelling application being able to duplicate or impersonate a human. It's much less about kind of the overall production of a phishing email and much more about how authoritative can it sound, as you were saying Jane, how much does it sound like the original service comparative to something that might just use a similar domain spoof but sounds absolutely bonkers? So you're kind of like okay, well, this isn't correspondence I receive from this.
Jane
No, I love it when my DHL is like "Hello, dear." This is definitely a legitimate email.
Rory
Nothing quite like receiving an email that says it's from Bill Gates to spur your trust at 9am.
Hanah
One that made the rounds not too long ago were from Winnie Mandela. And they had her passport scanned in it, and obviously she's passed away a number of years ago, and so Winnie resurrected from the grave to send you an email urgently requesting something from you, and included her passport as verification. Which I thought was lovely, because personally if I want to know about a new correspondent my minimum expectation is send me a state ID. That's kind of what I think. I think, you know, as much as as kind of defenders can get surly and negative, we do have to appreciate the absolute creativity that can go into phishing emails and that can go into spoofs. Some of the Microsoft login page spoofs I've seen, if I weren't upset about them, are beautiful. They're really great work, they look very convincing and you really have to hunt really hard, and so I think as much as ChatGPT and services like it are obviously a newer phenomenon to be talking about, it is important to kind of highlight that this technology isn't new. Generative AI isn't isn't novel, it's been the subject of research for years, it's been applied in various ways. It's just this new open source kind of exposure of it that is kind of the novel bit, but we've done everything at Darktrace from applying supervised learning styles like this and large language modelling all the way to the looking at what an AI-augmented attack would look like. So supporting almost a Red Teaming side of it, and looking at that more 'Terminator Skynet' side of things where you look at what AI-augmented attacks would look like. And that research has been kind of ongoing for years, we're turning ten this year. So I think it is also worth bearing in mind that as much as this is newer to kind of the media cycle, and newer to kind of speak about, it's not a new technology it's just a newer offering of that technology. Hopefully it will kind of ease that public conscience a little bit. But this isn't the first time this has ever been done, and actually there's a lot of comparable services, Rory, I think you listed some earlier that are doing the same thing, that are applying language modelling in a similar way. One that just came out, I think recently by Nebulae or Nebula is Llama. And I am obsessed with it, because they have like a little llama robot that looks very like a mix of Star Wars and a cartoon as their logo. So it has my full support, regardless of the output. I'm very interested in the large language modelling called LlamaGPT or something.
Rory
So you've touched on it a bit already. And you've talked around the best practice approach to tackling output from generative AI. But I was wondering if you could go into, in a bit more detail, how firms could use generative AI models of their own or subscribe to a generative AI model in the future to directly tackle these kinds of threats.
Hanah
I would hesitate to say that your solution to a generative AI problem has to be a generative AI solution. I think going back to that spectrum of AI, there's probably an AI solution to the problem that you're tackling but it may not be in through the lens of generative AI. It may not be we see ChatGPT and services like it as a force multiplier for threat actors, and so we want to employ ChatGPT or services like it within our own offerings. I think, mostly, I would go back to the application of unsupervised learning and anomaly detection as the first port of call if you're looking to detect something like this within your own digital estate. In terms of how you can apply AI and services like this, there are so many different applications. And actually, my colleague, Max Heinemeyer, made a really interesting point recently that was essentially: we haven't seen a huge adoption of AI across many industries, and the technology has been around. We haven't seen tonnes and tonnes of people rushing to implement it at their own business level. So I think there would be some work for any organisation that wanted to implement this right away to not only identify their own use cases, but to see how it worked within their workflow, what kind of training data they could give it, if you're looking at a supervised learning application. And understanding how to make sure that your models don't have biases, because obviously if your models or your training data don't have kind of quality assurance and control, then anything you get out of it is a little bit poison fruit as it were. So I think there are a lot of barriers to immediate tete-a-tete implementation of something like this. But I think there are actually a lot less barriers to starting to apply AI technologies throughout your security stack, or to looking at ways to kind of integrate this more into your organisational consciousness. I think that is a very low bar, I think in terms of responding directly to generative AI with generative AI, you might have mixed results.
Jane
So yeah, it's interesting actually, you say that ChatGPT has kind of brought the idea of of AI and generative AI to the fore because we can go out and we can play around with it ourselves. Now, I was listening to The WAN Show, watching it, from Linus Tech Tips. And Luke Lafreniere, who's the co-host, pointed out that people are developing tools based on ChatGPT, people are getting very excited and stuff, but it is a testing model. It's been very clear that this is it in testing. And when it comes out of testing and into production, they're presumably going to try to monetise it somehow. For people who have been relying on it to create, say, their spam emails or their malicious code or anything like that, is that gonna put the brakes on them as well? Like, if you've seen an increase with ChatGPT becoming available, assuming that there is a real causation there, are we likely to see a dip in the linguistic quality for example, again when that goes away? Or is the cat out of the bag now?
Hanah
I suppose you could look at it a couple of different ways. If someone's gone through the trouble of reverse engineering, then yes, the cat's out of the bag. And also, once you've released something onto the internet, many celebrities would tell you, it's very hard to put the cat back in the box, as it were. And it's very hard to get that back inside, once you've exposed it. And once it lives on the internet, does it ever actually go away? And yes, ostensibly, you would see some sort of closure of it. But I think you'll see a version of this open source available, because there's now almost a public demand for it. It's part of the general cyber landscape. And so I think it would be harder to say, "All right, now that you've had a taste of it, pay £6.99 a month for it", and completely rule out any kind of free to try applications. Although maybe we'll see a subscription service, who knows? But what I would say is, I don't think that that will necessarily impact the direct correlation in terms of the improvement of phishing emails. I think if you see net gains because of ChatGPT, then you could look at it a number of ways. I mean if you look at it from a psychological standpoint, threat actors are people, they have the ability to learn. So perhaps they're taking English notes, if as you suggested, they don't speak English very well, and they've saved a template of this phishing email. It would be a very short-sighted threat actor who would just kind of pop it out and use nothing from it again, you've gone to all that effort so why not recycle it? I certainly would, if I were sending out 50,000 emails a month, or whatever the figure is, I would take something from that. So I think you probably won't be able to track high spikes and dips in terms of exactly what a closure of services like this would look like in the threat landscape. But I think what you can see is, generally speaking, hopefully we move towards not only that public conscious understanding of this application of AI, but also from a defence perspective understanding the possibilities and limitations of technology like this, that you can accurately predict and defend against it. Because ultimately, if you're kind of expecting it to be able to make autonomous decisions, and you're expecting to kind of have an input source where someone's giving an AI bot instructions on how to kind of hack into your network that's not realistic. And that's not what the technology does. So also there's a tracking of the limitations of language modelling technology from Blue Team defenders, and from organisations who are trying to learn lessons from this. And they can see that and accurately try to defend against it.
Rory
So when we're looking at the rise of AI models, and there's a lot of regulatory activity around AI right now, as you say this has been in development for years, but obviously the public scrutiny is now very much on AI, and generative AI specifically. We've got the UK and the EU, in particular, going through various drafts of of AI legislation. And the key of the EU's legislation is it's looking at transparency on how models work, the data that they use, how they operate, and also to put pretty strict controls on how people can use them. Do you think that this is likely to make the landscape better? Will it have a measurable impact, government action of this kind?
Hanah
I think cyber centric legislation is always a good thing for the cyber threat landscape as much as legislation can keep up with emerging technology, which is always going to be a bit of an uphill battle because a lot of legislation and governance around any sort of technological advancements are going to be very wet ink, they're going to be hard to kind of keep pace with this pace of the development of the technology itself as well as the applications of it. I think in general, it's a great thing that legislators are not only thinking about but actually applying ethical concerns as well as controls on to technologies. Because ultimately, that's how you keep social accountability. I think there is a danger in placing all of your hopes and dreams in legislation, and kind of regulation as the way to go. I think you can see the story of that in even compliance, you look across the past years of compliance regulation and legislation, and there's still quite sizable gaps between meeting compliance standards and being fully protected as an organisation from compliance breaches. And so I think you'll still see those gaps existing, whether we keep pace with regulating AI technology and how it's implemented, and I think it's a positive step but again I don't think it's a silver bullet any more than applying AI to your own organisation in one form is a silver bullet of protection. You have to have kind of that layer of defence in depth, and I think it's kind of the same with legislation. It's a good step, but it's not necessarily a one stop shop to keep AI in a governable box.
Jane
Well, unfortunately, that's all we've got time for this week. But Hanah, thank you very much for joining us.
Hanah
Thank you so much for having me. It's been an absolute pleasure.
Rory
As always, you can find links to all of the topics we've spoken about today in the show notes and even more on our website at itpro.co.uk.
Jane
You can also follow us on social media, as well as subscribe to our daily newsletter. Don't forget to subscribe to the IT Pro Podcast wherever you find podcasts. And if you're enjoying the show, why not tell a friend or colleague about us?
Rory
We'll be back next week with more from the world of it. But until then, goodbye.
Jane
Goodbye.
