August 9 2016, census night… Families all over the country are preparing to complete their census online, but no, they can not access the website. People are venting, the ABS is copping it left, right and center, and our PM is being hammered for the government’s incompetence or running a proper census. And then we hear the news… The site was hacked, but it wasn’t hacked, and people started complaining. So what really happened? Let me explain.Let me start by saying that I have no idea what really happened at the ABS that night. This is merely my opinion, based on what I read in the news, and also my own experience of working in IT, and also in Security.
Australia has about 25 million people. Not all of them would actually complete a form. Take my own family. We’re 4 in the house, but only 1 filled in a form. So assuming an average family of (shall we say 3?), that leaves us with around 8 million forms. The ABS estimated that around 60% would complete it online, while the rest would use manual forms. That leaves us with around 4.9 million forms. While it sounds like a lot, 4.9 million forms is really not a lot of data.
Some users have shared the tender documentation on Facebook and Twitter, showing the contacts from IBM and other vendors that worked on the project (citing how we spent $500k on a project that doesn’t work). One particular vendor was contracted to do a VST (volume stress test). What this means, is that the solution is placed under increased stress, submitting more forms per hour than would normally be expected. The purpose of this test is really to see if the solution will hold up under severe load. I managed to complete my census form around 6:40PM, before the problems started, and overall, my experience has been quite pleasant. Performance was good, and the site behaved well. This leads me to the conclusion that the site was adequately sized for the required capacity.
The site was taken down, and the news reported the site was hacked, but what really happened, was what we call a DDOS attack (or a distributed denial of service attack). This is not a hack. A hack would imply that data got lost. At least according to the news articles, the ABS claims that no data was lost. But what does make sense, is the DDOS attack. For the layman, what is a DDOS attack?
A DOS (denial of service) attack is when you overload the system intentionally with the aim of either crashing it, but really to stop it from working, thus causing an impact to the people trying to use it. A single PC for example, could initiate a DOS attack, by making thousands of connections to the web site, with the hope of starving it of memory or CPU cycles, thus causing the thing to run slow. With a DOS attack, it is usually easy to see where the attack comes from, and you can stop it. For this reason, hackers do not usually use a DOS attack.
A distributed DOS attack, as the name suggests, is targeted from multiple computers. So while the ABS is trying to thwart off one attacker, another one is also pounding on their network. Now increase that by a few thousand computers, each making a few thousand connections every second, and before you know it, the system crashes.
It’s easy to say to just increase the bandwidth, buy more servers, put up more firewalls, but the reality is, it costs money. As IT people, we always have to weigh up cost vs benefit. A system like this would have been sized to handle a number of thousand forms per minute, which would have been more than enough to complete the census. But when a bad guy (or bad guys) start hammering the system with unnecessary connections, it takes up memory, bandwidth, and CPU time, and before you know it, the bad guys are making more connections to the system than the normal Aussies trying to complete the census, and then the thing crashes. A DDOS attack would not normally result in the loss of data, but it is likely.
Do not blame the NBN… The NBN has absolutely no impact here. This was a problem on their data center, where the backend was overloaded.
So before you go on Facebook ranting about the government’s incompetence, maybe take a step back, and try to understand first what happened. Then you’ll realize it’s not the government’s incompetence, but it is a group of bad people who has nothing better to do with their time, than to cause us all grief.