ZX Security attacked a big piece of Government technology—and we were proud to help
In one way, a denial of service (DoS) attack is pretty simple. All you need to do is keep sending traffic to a server until it overloads, then watch the fallout. But when the cyber security consultants at ZX Security are the ones doing the DoSing, there’s a bit more to it.
That’s why we spent some time talking with ZX’s founder and CTO, Simon Howard, and Security Consultant Dave Robinson. ZX Security is a Wellington company that helps clients understand their own security posture. They perform penetration testing of systems and cloud infrastructure, consult on security strategy and incident response, and also train people whose jobs involve cybersecurity.
For this particular project, a Government agency (who we can’t name) wanted to be sure that an important, new online tool couldn’t be overwhelmed by an excess of traffic after it’s been released to the public.
A carefully planned overload
As Simon explains, “Our client is going to be delivering a service that’s public facing and quite important, and they wish to determine if someone could disrupt that service through a Denial of Service attack.” It’s ZX’s job to simulate that attack from the outside, and monitor its impact from the inside. How much traffic does it take to cause an issue? Which infrastructure buckles first? How far does the impact spread, and how well can the organisation respond?
Start with the amount of traffic. ZX has plenty of experience sending excessive traffic to customers - 1Gbps is often more than enough to cause a problem - but this large Government client was operating on a different scale. “This was the first time we’d come across a request for 10Gbps,” Simon says.
Dave ran the project. “What we do with DoS is we try to replicate the full end-to-end path,” he explains. “We want to be on the internet and go through the entire path, like an actual user would, so we’re hitting all the infrastructure along the way.”
One early question was how, exactly, to produce 10Gbps of external traffic. Your average ISP isn’t going to let you monopolise that much bandwidth just so you can see what happens next. Looking internationally wasn’t a realistic option either. “If we needed overseas traffic, we’d need to buy so many VPSs and do 10 megabit here, 10 megabit there, because otherwise they’ve got rules in place that would see the amount of traffic and isolate the VPS.”
Enter SiteHost
“We called on our good friends at SiteHost who, at short notice, were able to provide us with servers that could generate a volume of traffic that we thought would be appropriate,” Simon says. As a long-standing SiteHost customer, he knew he could rely on us.
I don’t know of many other companies that could have provided that. You have the networking knowledge and skills, and expertise in the team to facilitate it. You were interested in understanding what we were trying to achieve, too. There are only a few providers in New Zealand that I’d trust to do that.
Dave adds, “Few companies in New Zealand could do that scale, or would be happy for us to throw that much traffic down a pipe. Or would understand their own network well enough that they’d know we weren’t going to affect anything of theirs.”
We were literally well-placed to help, too. ZX’s client uses a CDN with a point of presence in our home town, Auckland. Even though the DoS test needed to start on the internet - somewhere external to the client - proximity still matters. We set up 10Gbps servers in two different, equally handy, local data centres.
It was a novel set-up for an unusual test - exactly the sort of bespoke infrastructure request that we enjoy. Our CTO, Quintin, worked on it himself with ZX always in the loop. “SiteHost is small enough that we can have the right sort of contact with you,” says Simon.
Two test runs in a day
In one day, Dave ran morning and afternoon DoS tests, with some quick infrastructure improvements in between. That was all the time it took to increase the maximum load that ZX’s client could handle by around five times, and to see where they could make up even more ground.
The morning test didn’t take long. “From SiteHost, across Auckland into the exchange - the CDN was also on Auckland internet exchange - that part was all fine. The problem was that from the CDN’s point of presence there was only a 50 megabit link back to our customer’s data centre. We’d proved our point by the time we had about 400 megabit going through to that 50 megabit pipe. Response times were up in the 10s or 20s of seconds. And that was the first run,” Dave says.
ZX’s client knew about the 50 megabit pipe and thought that it wouldn’t be a problem since they had a CDN. The failure was unexpected, but quickly fixed. “They reconfigured the CDN to perform some protections, rather than just caching.”
A second discovery wasn’t so easy to handle.
Our customer had multiple web presences. We were just testing one particular site. Their other sites, which we weren’t sending any traffic to, also went down because they were using the same pipe. So now we’ve identified other collateral damage.
This gave the client an architectural question: Should they introduce a dedicated pipe for the critical site? It’s the sort of change that you want time to plan, but without ZX’s test they wouldn’t have known to even think about it.
With that decision left for another day, it was time for the second round of testing to see how the reconfigured CDN performed.
“We replicated the tests in the afternoon, increased the amount that we were doing, and we found that the protections that they’d turned on had worked. By the time we were throwing 1 or 2Gbps at them we knew that the CDN and the rules that were now in place were doing what they were supposed to.”
Any time you become five times more resilient to attack, it’s been a good day.
The larger lessons
Every failed test teaches you something. As Simon says, “It’s very good for the client to have tested this prior to going live. Maybe normal user traffic could have saturated that 50 megabit link. It really highlighted a weakness that wouldn’t have been apparent otherwise.”
Seeing separate systems fail together is a good eye-opener as well, as the ZX team knows well. Dave says:
We’ve done DoS testing where we go for a company’s self-hosted corporate site, then the test takes out their VPN and their Outlook web access, and things like that. That can surprise customers who haven’t thought, ‘wait, our VPN is behind the same internet connection and firewall.’ So not only have they knocked out our email, they'll also knock out our VPN, and now no-one in the office can access the internet, because the internet pipe is filled up.
In situations like this, your incident response process decides how well you cope and recover. When your website’s offline, and your email is cut off, and your internet is effectively blocked, what are you going to do? To ask the same question in ZX’s words, how’s your Denial of Service Maturity? If you’re a 1, you’ll find it hard to even know what you’re up against. If you’re a 5, you’ll know exactly what to do.
ZX often tests incident response processes without actually triggering an incident first. “Tabletop” exercises like these can reveal a lot. “They’ll have the wrong phone number for their ISP, or the wrong person to talk to to mitigate or start dealing with the incident,” Simon says.
Dave adds, “We had a case where the ISP had our customer’s previous help desk provider. So the ISP is saying, ‘hey, you’re under attack,’ but the help desk says, ‘nope, that’s not our customer,’ and it all gets lost in phone-tree land.“
“But it’s always fun when we actually get to do a real life scenario and run the two together,” Simon says. That’s because a properly-run DoS test does more than just make some infrastructure fall over.
For ZX’s Government client, Dave uncovered a lot of lessons about their processes and about collateral damage. “We’re helping them understand how their responses work, understand how their ISP or their DoS scrubbing works, and how the comms channel and phone trees work.”
How major is a 10Gbps DoS attack?
Since a 10Gbps DoS test was new ground for ZX, and for us, you might think that it’s at the large end of the scale. But it’s a very realistic scenario.
“In New Zealand, a standard home connection offers 1Gbps down, 0.5Gbps up. So get a few mates together and you could create the low end of that 10Gbps volume. A handful of NZ home connections could actually knock out businesses,” Dave says. “We’ll have DoS discussions with our customers and find out that a lot of corporate pipes are quite small - the pipes into offices, or where VPNs terminate.”
Those home connections keep growing - 10Gbps is already possible for residential internet - which makes attacks of this size more and more plausible . That’s the main reason to run tests, and to act on the results. “This isn’t testing for the massive botnets that run ransom campaigns,” Simon says.
In some ways, this sense of scale makes it more surprising that SiteHost was ZX’s only choice when it came to setting the test up. Not that we were stretching too far, yet.
If a client wanted us to simulate a massive botnet, we could spin up the resources to do that.
The unhealthy risk appetite of SMEs
Thanks to their range of clients, Simon and Dave have a good idea of how different New Zealand organisations view cybersecurity.
Simon: “The New Zealand Government invests a lot of money in security. It’s high on their radar, and this customer requested DoS testing off their own bat. Whichever New Zealand Government department you’re working for, you’re very aware that they don’t want to be on the Stuff home page. There’s a mandate that comes from quite high up to keep everything shipshape.”
What about the private sector?
“For companies at the growth stage where they’ve got people sitting across governance, things like the NZX DDoS news in 2020 put security up at the board level. At the board level, cybersecurity is the new health and safety. A few years ago health and safety risk was a number one concern for boards, now cybersecurity risks are 3 of the top 10.”
So far, so good. But at smaller companies, the story changes. “Things need to be quite tangible to small or medium businesses,” Dave says.
Simon agrees. “Until something happens to your business - or to your owners’ mates - you probably won’t do anything. Your average NZ business wouldn’t really think about it until they have a problem.”
If you, or one of your customers, might be the sort of organisation that needs a security wake-up call, and if you want it delivered by a heap of traffic turning up all at once, you know who to call.