Census 2016: The $10 million online census – What went wrong?

Well, hate to say I told you so, but – this was always going to happen.  It’s as if Click Frenzy didn’t happen in 2012 and the ABS just assumed they had it all under control.  Nope.  Tonight, millions of Australians were ready to follow instructions and complete the census online – but the website failed.

So here’s the thing.  A website is just a bunch of files sitting on a computer.  Any one computer can only really handle so many people accessing it at one time.

As the internet has grown, some great things have been put in place to account for this.  Tonight, we saw how even the best laid plans can come unstuck.

The real issue for Australians, and the Australian Bureau of Statistics (ABS) is the $469,367.50 spent on “Load Testing” to ensure this very thing didn’t happen.

Through open tenders, the Australian Bureau of Statistics awarded the contract to “Design, development and implementation of eCensus Solution 2016” to IBM.  They built a pretty darn slick solution that created a platform by which all Australians could fill out the 2016 Census – at a cost of $9,606,725.00

When someone raised the prospect of issues with people connecting, the ABS wisely undertook to tender for services to “Load Test” the site and system.

Load Testing is a process by which a system is tested to see how much traffic it can cope with.

The ABS paid an Aussie company “Revolution IT Pty Ltd” $469,367.50 across three tenders to undertake “Load Testing Services for Census 2016”.  $325,000 of that was for software licences to get the equipment required to do the testing, the rest we can assume was for the teams to actually undertake the testing.

Here’s the issue.  We can’t yet “blame” Revolution IT for tonight’s outage.  In fact, they may well have tested it exactly as requested.

The ABS themselves have been quick to respond to concerns of the website’s capacity in recent days:

So, let’s assume for the moment that in fact, 1,000,000 form submissions per hour was exactly what the site was tested for.  The above message from the ABS assumes they expected 500,000 submissions per hour.

Aren’t there 10,000,000 homes in Australia, so probably many more than that when you consider other institutions like hospitals etc?

Yep, 1,000,000 or more completed the census before Tuesday August 9.  Yep, lets say another 2,000,000 completed it during the day on Tuesday.

Let’s also assume another 2,000,000 are doing paper forms.  Heck, lets put another 2,000,000 into the “in the days after Tuesday” category.

That leaves 3,000,000 to complete it on Census night.  Did the ABS really think that number of submissions would be evenly spread over 6 hours?

What a joke.  Of course 7-9pm was going to be peak time.  Of course they were going to get millions of people each hour.  They simply didn’t expect it as the rest of us did.

This is an utter failure of planning and process.

And it reflects poorly on anything being “online” in the future.  Australian’s will be scarred by this experience, forced to be sceptical of any such concept in the future.

What could they have done though?

Spread the load.  Sure there is probably a server based solution that would handle the millions in the short space of time.  God knows what that would have cost.

But if the ABS had actually encouraged people to complete the forms early, or late, then perhaps the load would have been more evenly spread with reasonable and manageable traffic spikes.

Not to be.

An investigation will be required, and likely an assessment of just how accurate and complete the data really is after it’s all said and done.