Telstra outage was maintenance error, more Triple Zero numbers revealed in Government submission

Telstra has today provided a comprehensive update and timeline of the events of last week’s outage ahead of a Senate Committee appearance looking into the Triple Zero systems.

While much of the timeline is detailed and the overall gist of the outage remains the same, there’s a couple of interesting nuggets worth unpacking.

Firstly, we now know one and for all that it was in essence Human Error that caused the outage.

The Telstra Time Failure

Telstra confirming that what they’ve referred to up until now as a “Software Defect” was explained in more detail. Telstra says that “We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault had not been properly documented. “

This is a change management issue, processes, procedures and systems must be documented to within an inch of their capacity to ensure – well, to ensure things like this don’t happen.

So when routine maintenance was performed, including a restart to the time-sync system Telstra say “This meant the maintenance team was not aware of the way the device would behave when restarted. “

Additionally “A software update had also not been applied to the device.”

In essence “Had that software update been completed or had the design change been properly reviewed and documented post the earlier incident, and reflected in the maintenance procedure, the outage may not have occurred.”

Completely in line with Telstra’s messaging on this, they say “That is clearly unacceptable. If maintenance work can trigger this kind of outage, it suggests our controls were not good enough“

An internal investigation as well as an external expert investigation will be ongoing.

Triple Zero Woes

Given the media reporting, you’d think Triple Zero was broken during this outage. But it wasn’t.

This latest submission confirmed a few more details.

58,835 calls to Triple Zero successfully connected on the first attempt.

1% did not. That’s the 604 calls we’ve heard about before.

Of them 335 told Telstra on checking, that they did not need any assistance.

For 102 of them, Emergency Services were in attendance.

For 23 of the 604, They still needed help and Emergency Services were referred.

Another 144 did not respond and were referred to police for in-person checks.

Critically, 172 calls DID successfully use the Optus or TPG networks to connect to Triple Zero (Camping on as it’s called).

Any failure to call Triple Zero is a bad one, these numbers highlight the scale far better.

Telstra will front the Senate today, and no doubt Sarah Hansen-Young will yell and scream at Telstra and their Failures to get some good social media fodder.

What actual change or impact will come from this appearance we don’t know.