Telstra network back to 100% – almost 12 hours after outage today caused by Software Update

Telstra’s CFO Michael Ackland has confirmed that their network is now fully restored and that all services should now be back on the network and we can now reveal the outage was caused by a “software defect” which came from an update to a core part of their mobile network’s computer systems.

Mr Ackland was adament about the cause not being malicious, saying “I want to be very clear – this was not the result of a Cyber Incident”

Triple Zero Woes

With a lot of media focus on Triple Zero after the Optus outage late last year you might see a whole lot of reports about Triple Zero and being let down today.

Let’s be clear, this issue was not the same as Optus. Optus, by their own admission, had an issue that did affect people reaching Triple Zero.

In the case of today’s outage, Telstra today confirmed that they have bene in contact with 333 people who attempted to call Triple Zero during their outage.

That’s a huge number. But, Telstra reached out to them all by SMS, 138 replied that they were AOK. Another 189 of them were contacted by Phone. 110 confirmed they were ok.

The 79 who could not be contacted are now being visited by Police in person.

Critically, of those 333, it’s clear that many DID successfully get to Triple Zero by “camping on” to the Optus or TPG networks.

I believe, perhaps naively, that this is an issue of education.

Firstly, it’s undoubtable that some of those 333 people simply called 000 to see if it would work.

Secondly, those that did, while their phone was in SOS state, likely did not think it was working – when in fact it would have connected had they just waited

People just don’t know about that, so we need to educate on this, plus work out once and for all – why? Is it the device, is it the network? Why does it take so long?

What is a Network Timecode issue?

The root cause here was an “Update to the NTP service” that’s the Network Time Protocol, and that update “impacted the GPS node” – meaning a bunch of Telstra’s network computers were out of sync.

When important network infrastructure is out of sync, it either fails or causes the devices to go into a fail-safe mode and that’s essentially what would have brought down the network.

Here’s the full statement from TELSTRA CFO MICHAEL ACKLAND:

I also want to say thank you for your patience and understanding as we’ve worked to put things right today. We don’t take that for granted.

When I briefed media at 9:45am this morning, we had just under 90% of calls and data back up and running.

Since then, we’ve worked to restore all remaining services and as of 4:00pm this afternoon, the issue in our network is fully resolved.

It may take some time for all devices to reconnect correctly, and we encourage customers still facing issues to restart their phone or device.

We are still conducting our investigation into the root cause, but we are confident we have identified a software defect and have been able to isolate it. We will provide further technical details on this once we complete our investigation.

I want to be clear that this was not the result of a cyber incident.

We have seen some reports of customers receiving calls from fraudsters trying to take advantage of this moment. Our advice to our customers is if you get a call from someone claiming to be from Telstra asking you for details in light of today’s outage, please hang up and call us directly. 

We are deeply sorry for the impact this issue has had on so many today. We know how much our customers rely on us to get it right – to do their jobs, run their businesses, stay safe and keep in touch.