The Australian Bureau of Statistics threw 400 censuses of load at the online portion of last year’s same sex marriage survey to ensure it would not suffer the same fate as the 2016 online census.
The lead solutions architect for the survey, David Crisp, said this week that the ABS treated the marriage survey project “first and foremost [as] an exercise in rebuilding trust”.
The “daunting task” of running the survey was foisted upon the agency by Prime Minister Malcolm Turnbull at a time - Crisp said - when trust in government generally was low and there was significant public doubt in the ABS’ internal capabilities.
The day after Turnbull set a date for the survey, Crisp said the initial internal team of eight people awoke to articles that the survey “could break the ABS”.
“The marriage survey was a lot of things for a lot of different people, but for us in the ABS it was first and foremost an exercise in rebuilding trust,” he said.
“After our 2016 census outage, also known as censusfail, we needed to build trust in ABS’ capability and we also wanted to build trust in the Australian public service overall.
“So to do this we needed to provide a positive user experience for all of Australia through the survey, in addition to integrity of the result and security of the data.”
Life after Censusfail
The ABS’ eCensus suffered a massive outage on Census night in August 2016. It was initially taken offline by a series of DDoS attacks and then kept offline due to behind-the-scenes failures by outsourcing partner IBM.
In addition to running the marriage survey on a completely new architecture - ABS chose an AWS stack - the agency also threw massive traffic at the online form to ensure it could withstand pressures it might face.
“Any outage of the online form would have drawn comparisons back to that 2016 online census so we put the AWS form through a rigorous load testing and DDoS battery of tests,” Crisp said.
“To give a feel for the scale of the testing, if you were to ask everyone in Australia to hop online and fill out a form on a single night like we did with the 2016 census, you’re looking at a bell curve of usage that will peak at about 250 submits per second.
“In our largest test with AWS, we threw 100,000 submits per second at the [marriage] form so that’s 400 Australia’s worth of load.
“The lab technology behind the scenes handled that extremely well and had response times of less than a second despite all of that load.
“So if you want to run 400 concurrent censuses all at once, this is the type of infrastructure to run it on.”
Knowing your options
Crisp said that one of the key learnings from the use of AWS to underpin the survey was “that cloud is the only option for Australia-wide events.”
“There’s no doubt of this in our mind after seeing what the AWS form could do,” he said.
“I think our census for 2021 will be heavily cloud-based and of course the online form will be cloud-based”, he said, without firmly committing to what kind of cloud might be favoured.
Crisp also said that AWS had been important in allowing the ABS to accept responses from Australians in a wide variety of situations.
“Malcolm Turnbull described the survey as a chance for everyone to have their say so this meant inclusive strategies for people overseas, with disabilities, remote and indigenous communities, and people who spoke different languages,” he said.
“It was clear that a paper-only survey was not going to cut it. We needed multiple ways for people to be able to respond, including online and on the phone.”
Under the hood
The phone-based mechanism for the survey was built on AWS Connect - a cloud-based contact centre service - and AWS Polly, a text-to-speech engine.
“The backend was powered by serverless Lambda and DynamoDB technology,” Crisp said.
He noted the system was able to be stood up rapidly: the agency only began working with AWS on day 14 of the 99-day project.
It “had a first cut of the solution by day 20. We then iterated on the solution, added security features and integrations with our in-house systems, and went live as scheduled on day 50,” Crisp said.
Crisp said the agency was “happy with the result” of the AWS-powered system and with “the outcome of the survey as a whole”, with the project continuing to receive external recognition.
He also lauded the project as a key step to restoring public trust in the ABS’ IT credentials and ability to handle Australia-wide data collection.
“As an organisation we are on a journey to build trust with the public,” he said.
“This was an intense, high stakes activity, and in this activity AWS showed it could be trusted to deliver solutions at a really high velocity and the solutions they delivered were extremely robust and reliable.
“The bottom line is: we’re on a journey to build trust, and on this journey we placed our trust in AWS and they delivered.”
