This automatically-generated transcript is taken from the IT Pro Podcast episode ‘Inside the DDoS arms race'. To listen to the full episode, click here. We apologise for any errors.
Adam Shepherd
Hello, and welcome to the IT Pro Podcast. I'm Adam Shepherd.
Connor Jones
And I'm Connor Jones.
Adam
And today we're going to be discussing one of the most persistent threats to organisations' online security.
Connor
In some respects DDoS attacks are one of the simplest tools in a hacker's arsenal. They work by flooding a target's network or infrastructure with junk traffic in order to overload it and bring it down, and they can be accomplished with nothing more sophisticated than a forum's-worth of bored teenagers with an axe to grind.
Adam
The potential disruption they can cause to organization's operations still represents a very real danger, however. What's more, while basic DDoS attacks can be simple to launch, professional cyber criminals are using more and more sophisticated tools to pull them off more rapidly, cheaply, and in greater numbers than ever before, with DDoS for hire services putting them in reach of a worrying number of potential threat actors.
Connor
Were joined this week by John Graham-Cumming, CTO of infrastructure and security vendor Cloudflare, to discuss the technical sophistication of modern DDoS attacks and how the security industry is working to keep pace with them. John, thanks for being with us.
John Graham-Cumming
Thank you very much for inviting me.
Adam
So let's start off by talking about the scale of the problem. How common are DDoS attacks in general?
John
Well, so from our perspective, because we mitigate DDoS attacks, these are things that we deal with literally all the time, there are actually DDoS attacks going on against a Cloudflare customer every minute of the day and night. In fact, there's usually multiple of them. And it used to be, there's an internal dashboard, where you can look at the DDoS attacks that are ongoing, and I used to look at it because I would be a little bit, you know, blown away that there were so many going on. And it's become such the norm that there are multiple going on, I never look at that dashboard anymore. It was just, well, that's what happens.
Connor
What kind of impacts can DDoS attacks have on an organisation?
John
Well, I think if you think back a few years, when you think about DDoS, it was often, you know, the website was not online for a business. And at a time when websites were primarily sort of marketing tools, almost a brochure, right? It mattered because it mattered from a reputation perspective and some loss of business. But of course, we've switched to using the internet for pretty much everything. I mean, look at us talking like this over the internet and schooling and working from home and ordering lunch. And you know, you imagine the the things, the myriad of things we do on the internet, all of those things are vulnerable to DDoS attacks. And there's a direct financial cost, right, you get knocked offline, you know, if you knock offline a major food delivery company, probably at around this time of day, we're recording this around lunchtime where I am. And there's a lot of business there, right? And if you knock off an ecommerce, you know, site, or if you knock off the endpoint a company uses where its, you know, employees get access to internal apps or their email or something, well, that has a real cost. And unfortunately, DDoSes are relatively easy to do, and they can have quite a big impact.
Adam
So for an organisation that doesn't have the protection of service like CloudFlare, how much traffic would it take to knock them offline, do you think, on average?
John
It's a very hard question to answer because it depends how they are hosting their website, for example. It can be the case that it's a very small amount of traffic. And you know, for example, if you have a small blog, and you're hosting it with, you know, a VPS, for example, you know, just a server that you're renting yourself, which itself doesn't have any sort of DDoS protection, it may be that hundreds to thousands of requests per second to the server - so a bit like lots of people have turned up at your server and therefore it falls over, right? That could knock it offline. Or if the connectivity for the server is, you know, relatively small in terms of gigabits per second, right? If you, if you've got a single server somewhere, I have one, I have a private server somewhere and I think it's got, you know, a gigabit per second connection. Well, if you can send me more than a gigabit per second of traffic, that server is not going to be accessible. So actually, depending on how you have actually set this up, it can be a relatively small amount of traffic and relatively easy to do.
Connor
And then what's what's the likelihood of us seeing another sort of Saudi Aramco style scenario where we see like, massive real world impact on equipment being destroyed and things like that?
John
We see professional attacks by hackers now, and what I mean by professional is that they are seeking out targets where they can make money. So if you think back at one point, DDoS attacks was a favourite of Anonymous right? We're gonna knock off this website cause we don't like, I don't know, we don't like Rupert Murdoch, or something, we'll knock off this website offline. And then it was used for political reasons and things like that. And then also, just for, you know, kids would just do it for fun, right? I will knock off this thing offline and actually one very famous case, somebody who had not studied for an exam, used a DDoS attack to knock off the exam provider on the day of the exam. So no one could take the exam, to buy themselves some time; unfortunately, they bought themselves a criminal record. But you know, there sorts of things happen. But I think what's notable in the last few years, particularly the last two years, essentially how this has become a business. So what will happen is attackers will scope out a company. And they will, they will do a DDoS often as a demonstration of like, well, we knocked offline your email server or your VoIP server, so you can't make any phone calls. Or your blog, because we don't like your blog, or whatever, or your blog is popular. And then they'll send you a ransom note. And they'll say, you know, it's, it'll happen again, unless you pay us some money.
Adam
It's like the 21st century version of a protection racket, essentially, it's basically extortion.
John
I mean, if you think about it, it's it's not rocket science to figure out this is going to happen, which is that all business has gone online. So obviously, criminals are going to follow, figure out how to make money out of that. And, you know, we've, we've moved everything online, essentially. And the pandemic has only accelerated that trend. So what a surprise that attackers are figuring out, they can make money this way. And in fact, we've heard of cases where attackers have figured out the size of the ransom based on the size of the company. So actually, you know, I've heard of an attack where some hackers broke into a company's financial records and used that to size the size of the ransom. Well, you know, this company can pay us, you know, half a million, and that one will only pay us a few thousand. So that, you know, they'll, they'll, they'll do that sort of thing. So that I just for me, it's become very professional. And so you know, what happened to Saudi Arabia? Well, I don't know. But it will not be surprising if more things happen, where companies get knocked offline, because there's money to be made out of it.
Connor
Yeah, absolutely.
Adam
So how are these attacks being enabled by other technologies like cloud and like AI?
John
Well, I don't know about AI, whether AI is really involved; now unfortunately, don't have to be that smart to to do a DDoS attack, what you need is a bunch of servers around the world. And there are really two ways to get the servers with enough capacity. One is you go to some provider, and you literally rent them, maybe you rent them with stolen credit cards or something. Or you go to your provider that's slightly dubious. Or you steal someone else's servers, right, you break into somebody else's servers and use them. And so you'll see that happen where you know, a hosting provider will get used as the source of the DDoS attack. The second way is you build a botnet, like in the old days, if you send out a virus or a worm of some sort, you break into a bunch of computers in people's homes, could be Internet of Things devices, like cameras or DVRs. It's funny, there's an interesting anecdote, which is that years ago, we used to see that DDoS attacks and other types of attacks dropped on Earth Day. And the reason they dropped on Earth Day was people switched off their computers. And those computers were no longer available to be part of a botnet, which had, you know, they'd previously been hacked. That trend does not seem to have continued. And I think the reason it hasn't, is we're leaving a lot of stuff on even on Earth Day. And if we switch off our computer, we don't necessarily switch off our internet connected camera, which may have been hacked and participate. So they will hack whatever they can to build these botnets. And then they use the botnet to send out a lot of traffic. So that's, that's what they do. And the third way is find some vulnerable protocol on the internet, which will do your bidding for you. So the classic one was you find an open DNS resolver, something which we all use to get DNS responses. And you forge a request that appears to come from your victim for something very large. And so you send a little message out to this resolver and say, Hey, what's the DNS, you know, give me all the DNS records for example.com. And because you forged the source of that, the DNS resolver says, Okay, here you go, and he sends back a load of data to the victim. And that was called reflection and that remains incredibly popular. The protocols being used vary from from day to day in terms of what you know what is latest, but anything that can reflect and amplify, i.e. send back something bigger than what was sent, is very handy for attackers, disguises where the attack is coming and gives them more firepower.
Connor
Interesting. And so how would you say botnets have evolved over time? If they're sort of so sort of intrinsically linked with with DDoS attacks? Have there been any sort of major changes in the way in they're sort of behaving or the way attackers are utilising them?
John
Well, I think the big change is this move away from it being necessarily, you know, the sort of vulnerable Windows machine that's in someone's home or office to, to devices. And we saw that with the Mirai botnet, which was used to attack Dyn, which had a big knock on effect. And then, you know, from there, you you know, you see, that sort of the world of Internet of Things devices, there's another one which has been, which is using home routers, because, of course, you know, when you attach a computer something like a camera or a doorbell, or you know, your home router is a computer, right? You tend to bring along with you all the problems of computers. So they're hackable, they have bad defaults, people don't change the password. And in particular, I think on devices, it's pretty common for people not to change defaults, you know, you install your smart toaster, and you never change the password to something else.
Adam
If it even has the functionality to allow you to change the kind of default access credentials, I know that's something that a lot of kind of governments and legislators are looking at is kind of essentially regulating the construction of IoT devices, to give them kind of better security and to give users more control and visibility over that security.
John
Absolutely. And also, you know, a lot of them have not just default passwords, but I mean, with Mirai, it was, you know, it was very simple to go off and be like, you know, Telnet, to them, one of the oldest protocols out there. And then they just are exposed to the internet. And, you know, most people are buying an Internet of Things device are not thinking about the security of it, and nor are they thinking about, do I have a firewall in my home in order to protect all this kind of stuff. They're just thinking, well, I want my camera to, I want to watch my baby at night, or I want to, you know, have a smart doorbell or something. And so yeah, I think the, the regulatory approach is actually a good idea to go out and say that there ought to be some standards, you know, we have standards for like car seat belts, and we have standards for you know, I can pretty much be assured that if I buy a microwave, it won't set fire to my house, because over time, we've, you know, we've had standards, we've tested things. So, you know, I hope, I think that in some respects, it's a good idea, if the legislation is well written, to do that.
Adam
So we've discussed the kind of DDoS for hire services that can be used to launch DDoS attacks on on targets without necessarily having to get access to botnets or to rent the servers and launch the attack yourself. But how much does it cost to launch an attack using these services? How much capital do you have to front up in order, as a kind of lay person, if you like, to launch this kind of attack?
John
Well, I'm pretty sure - I haven't done this for a long time now - but if you knew what to Google, you could go out and Google and find a service that would do this DDoS for you. And it would, in many cases, you know, you'd pay with your credit card, and you'd be talking, you know, tens to hundreds of dollars to perform a DDoS attack against someone now; it won't be the greatest DDoS attack. But you know, if you've got a rival business, maybe you're a florist, and it's February the 12th, and you'd like your rival florists to be offline. You know, that's the sort of thing people do and that, and by the way, that's a real example of things that actually happen, which is that businesses will attack each other. I mean, the more - I'm not sure illegitimate is the right word - but the more interestingly regulated a business is, so gambling, for example, the more likely it is you see this kind of, this kind of behaviour happening. So yeah, I mean, DDoS attacks, one of the reasons why we see DDoS attacks all the time, is they happen for all sorts of reasons, you know, political reasons. I, you know, I disagree with somebody's view about something, but also just pure financial reasons. And that could be on a small scale or it could be on a large, you know, the DDoS for ransom style, which is very popular.
Connor
Yeah. So if say, say an attack is successful, what kind of business cost are we looking at compared to, is it sort of comparatively much larger than the cost it is to launch such a such an attack?
John
Oh, yes. I mean, I mean, you think about the the cost of launching the attack is is quite inexpensive, right, for the people who are doing the work, because they're not using their own, you know, they're not paying for all that servers, right? They've, they've hacked other people's smart cameras or something, build that into a botnet. And then they're using that for the attack. So it's relatively inexpensive for them to launch the attack. And for a business, of course, it can be crippling, in terms of, you know, if it's an ecommerce business, they can't actually do ecommerce. If somebody is not an ecommerce business, you know, some people go after your email server, or your VPN concentrator, or, you know, the links that everyone has to use in the, in an office, for example. There are lots of, there's lots of ways in which a business can get attacked. In fact, one thing we saw at one point was some attackers who, rather than do a DDoS attack at all, what they do is they write to the business and say, we've done a survey of your network, here's a network diagram, right? We've done it for you, this is your weak spot. Tomorrow at noon, it'll go down, unless you pay us a certain amount of money. And sometimes those threats would be real, especially if they've done a real good survey of the network, and then you have copycats, who just send a message like, tomorrow, this is going to happen. So now pay us. So you have this this world of like, sometimes you're not even ever doing a DDoS attack, there's no cost, it's just you send an email, essentially, as a spam, threatening some. So yeah, it's a lot cheaper to send this stuff than it is to deal with it.
Connor
Interesting. So and DDoS protection is obviously one of the main services that Cloudflare offers. But how does it actually work? You know, what's the what's the process of stopping a DDoS attack?
John
So. Well, first of all, important thing is in the architecture of our network so that the first D in DDoS is distributed, right? So very few DDoS attacks of any significant size come from a single source, right, they tend to come from a botnet and the botnets will inevitably be spread over a geographical area, they may be all over the world, typically, right? If you ever look at one of those 'pew-pew' maps, you know, those maps they have online, like war games with the missiles coming in, you'll notice that the attacks come from a lot of places simultaneously, because of course, the botnets are in lots of places. So by having servers in 250 cities around the world, what happens is if somebody DDoSes a client of CloudFlare, inevitably, the DDoS gets sort of spread out across our network, right? So it doesn't, it never gets to one place. And actually one of the difficulties with DDoS mitigation, the way it used to be done was it used to be that you would, if you were under DDoS attack, you would send all your traffic to a scrubbing centre, and that scrubbing centre would deal with it. And, you know, that was a, that was actually a big problem for the scrubbing centre, because they had to have the capacity to take the whole attack. So the first thing we do is we spread it out across our network. And then, within our data centres, we spread out the attack across every machine. So actually, the traffic gets, all of our traffic gets sort of smeared across every machine in the data centre. And then within our systems, we have built software over the last 10 years that will look for patterns. So repetitive patterns, the fingerprints of known DDoS tools; we'll also look for signals such such as you know, if a customer's server is under stress, and we're receiving a lot of traffic, then we'll use that and say, wait a minute, this is clearly a problem, the traffic getting through is hurting the customer. So we can actually then throttle it. But yeah, it's a it's a combination of a bunch of architectural stuff around how we lay stuff out across the world, across the servers, and then the fingerprinting and the analysis in real time looking for repetitive, interesting, anomalous behaviour. And then we thought, essentially, we drop it on the floor.
Adam
So in terms of the the tools that you use for that kind of pattern recognition and analysis and whatnot, what role does AI play in helping you to, to detect these trends?
John
So we particularly use AI, for when we're trying to determine if something is being done by a human or not. So alongside the DDoS problem, there is a massive problem of bots. And these bots are often they're pretending to be human in order to try and beat humans at something. So the classic example is a new sneaker gets released, limited edition. And there's 1,000 pairs. And what will happen is there is an absolute war between bots to buy the sneakers from retailers. And the reason is the resale value is so high, maybe you can suck up the whole market and you can sell it someone else for more. And that problem of detecting what is is non-human behaviour in a human? What is something that's intended to be human, right? It's something where we extensively use AI in the DDoS mitigation side of stuff, we do use some machine learning techniques to put stuff together. But over the years, we have built up a system, which is able to, what I would really describe as anomaly detection, which is to look at traffic that has anomalous features, because one of the things about DDoS is it's repetitive. So unlike the bots, which are coming in, and they're trying to do something, they might click on the front page and then put something in the basket, there's sort of there's quite, it's quite a rich interaction, right there. DDoS tends to be, I'm going to send you the same thing over and over again, you know, million times a second, yeah, or it's going to vary very slightly. And we can distinguish that from genuine traffic.
Adam
So if someone's sat on a page, and is refreshing that same page, once every kind of half a second, that's probably not a real person.
John
It might be; there might be somebody who's desperate for something to happen, because they, you know, in the case of the sneaker, right, if you really wanted to try and get like, you're trying to get tickets to a concert, you might actually hit Refresh quite a lot. Right? So we would want to know, does that, is that actually a real person doing that? Or is that a bot that's doing that?
Adam
Yeah. And we've we've run across a couple of instances over the last few years, where kind of for for stuff like that, for kind of limited edition product releases, or for hotly demanded tickets, vendors have essentially DDoS themselves by just creating this enormous demand. I seem to remember when the musical Hamilton first came to London. Basically, every time the the new batch of tickets got released, the website ended up going down, because the demands were just so insane.
John
Yeah, I mean, that happens a lot. I mean, we saw this in the pandemic, when the vaccines became available. For a sign up, you know, depending on the country, there was often maybe like, for example, here in Portugal, there was one website where you could go and sign up, because it was run by the National Health Service here. And we saw in different places around the world, depending on how well that stuff was architected, it would fall over because everyone wanted the vaccine, right? That's clearly not a DDoS. And you don't want to be blocking those people. And in fact, what we ended up doing was creating this free product called Fair Shot, which anybody who was distributing vaccines could use, and it essentially creates a waiting room where you get in a queue, and we control the traffic getting getting through to the real server. So you know, our job is to try and figure out what's real and what's not, and get rid of the not real and let the real through.
Connor
Yeah, absolutely. So we've we've sort of discussed throughout the session that sort of botnets are not helping things in terms of attack volumes, and things are getting getting worse year by year, can you give us sort of a flavour of what kind of attack volumes Cloudflare is dealing with maybe on a daily, monthly yearly basis?
John
Well, the interesting thing is, so the majority of attacks are quite small, actually. So what happens is, you'll see that every, every month, you know, there'll be a lot of attacks, which are under half a gigabit per second, things like that, right? So what happens is, the news is always about the really big attacks. Okay? So if we look to, you know, q4, we, in 2021, we saw an attack, that was almost two terabits per second.
Adam
Oof.
John
Right? And that gets in the news. And people are like ooh, it's the biggest ever seen, you know, all that kind of exciting stuff. And you know, we also saw one, which was about 17 million requests per second. And then just to make sure we understand the distinction, the terabits per second, you're just talking about flooding the network to try and overwhelm the network, right? The request per second, well, you're not trying to flood the network, you're trying to cause the server to die, because it's trying to do processing. So that's like 17 million hits of f5 per second. So if you have different reasons, you know, there are different ways to attack. So those are, you know, those, those get the headlines. The reality, though, is that we just get continuous, you know, DDoS attacks, month after month after month. And often, you know, relatively small in terms of, so in terms of on the network side, you tend to be you know, under a gigabit per second, under 10 gigabit per second, those things are quite common, because those values will knock offline an ordinary server right? I mean, those are still big enough. And on the, you know, we see packet rates in the 1000s of packets per second. Packets, the distinction there is that some attackers will go after the the network infrastructure, not just the the capacity, but also the actual switches and routers by sending a lot of packets and using small ones. And the typical DDoS lasts well under an hour. So it's like you sort of attack you and knock you offline. And sometimes that's to demonstrate for a ransom. And sometimes that's just because that's what someone paid for. And, you know, I don't like this, this, you know, this attack this person, I'll attack them. So, you know, we see just continuous attacks of all sizes, and it seems to be growing, unfortunately. I had kind of hoped this problem would slow down, but it doesn't seem to be doing so. Sadly, DDoS is just part of what we deal with. It's almost like it's the background noise of the internet, there's always some nonsense going on.
Adam
Hmm. So the the attacks that are less than a gigabit per second and the fairly small scale stuff, I would imagine that that's something that that Cloudflare's infrastructure can just absorb pretty much without really thinking about it. But is there a scale of attack that, you know, beyond a certain threshold, you actually go, oh, this is a serious threat, this is something that we need to pay attention to?
John
Basically, no. So we go, for example, the two terabit per second attack, we didn't do anything about that. Nobody was running around doing anything, all the automated systems dealt with it.
Adam
Really?
John
We built, we built our systems up so that they can detect and mitigate this stuff, automatically. And so I think that from from a, from a sort of a capacity perspective, because we built the network to be very, very large. We, you know, we don't worry about the size of attacks. Obviously, we worry in the sense of planning for bigger and bigger attacks, and making sure we have the infrastructure in place, making sure our systems are working and when something happens, like the two terabit per second attack, we'll, we'll review it and say, what was, you know, how did this work? What didn't work well, or what did work well, but in both the case of the two terabit per second attack, and in the case of the 17 million requests per second attack, those are just automatically mitigated.
Adam
Wow.
John
So in fact, it's the only way we could possibly do it, because there are so many attacks going on, you know, right at the beginning of Cloudflare, we used to manually edit things to stop attacks.
Adam
Oh, wow.
John
It just doesn't work, so...
Adam
It's not scalable.
Connor
So you mentioned the the attacks there. So they're sort of growing, and from an outsider with no sort of DDoS or little DDoS knowledge, to what end can this sort of increase reach?
John
Do you mean, you want me to predict how big the next DDoS is going to be? I mean, I, it's hard for me to answer. The reality is, you know, if you, if you went back a few years, I think Cloudflare reported on a 400 megabit per second attack, and it was like, Wow. I remember now, it's like, that was one and then if you go, if that was right at the beginning, and if you go a bit forward, it starts getting to the gigabits per second. And so and the news story is always about, you know, megabits and gigabits, and then terabytes, you know, so on. It will inevitably get bigger, because the internet is getting bigger, and we're getting better connections, and we're getting faster internet all round. So sort of all boats rise, including the DDoSer's boats. So, you know, we shouldn't be surprised that, you know, when we see a three terabit per second attack, or maybe one day a five terabit per second attack, this will just inevitably happen just as the internet gets faster overall. I think the real message around DDoS is not necessarily about the headline numbers. It's around the extent to which people get attacked. And, you know, one of the reasons why Cloudflare made DDoS mitigation free for everybody is it was just so common, it seemed crazy. We just, it's a bit like spam filtering in your email. You know, that's just included. And we just felt like, you know what, this should just be included.
Connor
Interesting.
Adam
So, on that note, are you concerned at all about the security industry's ability to keep up with the development of criminal DDoS capabilities?
John
I'm not concerned about it, I, that's what we do, it's our job, right? We spend time thinking about this stuff and we have to come up with new things and we, and you know, we write new software or our systems detect and and mitigate these things, we learn from that. So it's not something that that worries me. I think that it's just, it's an unfortunate reality that this will continue and companies like Cloudflare which protect people will continue to be needed. A bit like we, for some reason, we still need spam filters because obviously people are still answering spams, so there's obviously a business there, and there's a business in DDoS. And so, there you go, people are gonna keep doing it, and we will keep defending against it.
Adam
Well, unfortunately, that's all we've got time for on this week's episode. But thanks once again to Cloudflare CTO, John Graham-Cumming for joining us.
John
Thank you very much for having me.
Connor
You can find links to everything we've spoken about today in the show notes and even more on our website itpro.co.uk.
Adam
Don't forget to subscribe to our social media and YouTube channel for more great content and leave the podcast a rating and a review.
Connor
We'll be back next week with more analysis from the world of IT but until then, goodbye.
Adam
Bye.
