The Optus CEO, Stephen Rue, along with Senior Optus Staff and the Chairman of Optus, attended a Senate Committee hearing today investigating the Triple Zero outage which occurred mid-last month. Having had experience watching Senate Committee hearings in the past, rather than requiring my worst enemy to watch it— which would be punishment enough— I sat through it today on your behalf.

As a couple of hours’ access to the senior leadership team at Optus, I have to say, this committee of Australian Senators failed to hit the mark of what really matters about the outage— what went wrong.

Senators— by my count— spend almost an hour and a half talking about who Optus told and when, basically skirting around why it took Optus so long to tell regulators and the government about the outage. While this is, of course, cause for some concern, it fails to even come close to looking into what actually went wrong.

Optus, as is common at these hearings, prepared an opening statement, and along with that, a timeline of the events related to the Triple Zero outage. That timeline is extremely detailed, and I’ll go through some of it down below because it is quite fascinating as a matter of fact rather than the hysteria of the politicians both then and now.

When you look through this timeline, you’ll perhaps notice what I notice, and that is a lot of activity by the Networks team, hours and hours before some senior leaders within the Optus organisation were even made aware of this issue.

In fact, literally one minute after the Optus Executive team were advised, a Senior Adviser at the Minister for Communications office was contacted about the issue.

Stephen Rue, under questioning by Senators, continually conceded today that failures were made and multiple times escalation processes were wrong.

But today, with the Senior Leadership Team of Optus in front of them, no Senator asked questions about what went wrong.

Let me help those senators out with some questions:

  • Of the 455 calls that were unable to connect to Triple Zero, are you able to confirm what devices those calls were being made from?
  • Were any of the 455 calls made from Samsung devices which have since been determined to have issues with the Triple Zero network?
  • Based on investigations to date, is there any reason to be concerned about other specific devices operating on the Australian Triple Zero network?
  • How do these devices qualify to be sold in Australia if they don’t meet the standards of our Triple Zero network?
  • Are these failures in any way as a result of the 3G shutdown last year?

Frankly, the failings of individuals inside Optus, at low levels and management levels, are clear, and an internal investigation by a third party into the incident will clearly expose those things. Why we need an Australian Senator yelling at corporate bigwigs I just don’t know.

Senator Dean Smith was the closest on the committee to getting to questioning that mattered today. Hopefully, he gets his head around the technical issues we need to uncover before and if the Optus executives appear before the committee again.

What the Australian people need to know is how it happened, and what, if any, vulnerabilities exist in the community today should any one of our Telcos have an outage or network issue again.

Here’s the top line timeline of what happened during and after the Optus Network outage.

September 18

  • Midnight – Firewall upgrade commenced by Nokia
  • 18 minutes past midnight – First unsuccessful Triple Zero call
  • 20 minutes past midnight – Nokia raised an incident ticket due to a spike in monitoring graphs.
  • 47 minutes past midnight – Optus raised an incident ticket with Nokia.
  • 53 minutes past midnight – Microsoft Teams chat between Optus Senior Network Engineer and Major Incident Manager at Nokia.
  • 2.10 am – Network change (firewall update) complete by Nokia.
  • 10.13 am – Offshore call centre receives a customer call about Triple Zero issue.
  • 10.47 am – Offshore call centre receives a customer call about Triple Zero – noting that three or four people on Optus calls couldn’t call Triple Zero, the person is reported as in hospital and not in immediate danger.
  • 11.37 am – Offshore call centre receives another call about Triple Zero – again noting several people were unable to call.
  • 12.57 pm – Offshore call centre receives another call about Triple Zero.
  • 1.15 pm – South Australian Ambulance advises Optus “Operational Architect” of a possible Triple Zero issue.
  • 1.17 pm – South Australian Ambulance confirms to the same person there is an issue.
  • 1.25 pm – South Australian Ambulance calls again asking for an update.
  • 1.38 pm – Optus Networks Major Incident Management had a technical call with Nokia Operations.
  • 1.51 pm – A major incident was declared.
  • 1.53 pm – South Australian Police report an issue with Triple Zero failures.
  • 2.11 pm – Offshore call centre receives another customer call about Triple Zero.
  • 2.16 pm – Optus’ director of Network and Service Operations advised the Corporate Affairs and Marketing Manager.
  • 2.20 pm – Optus Networking team requests the change to the Firewall be rolled back.
  • 2.31 pm – Optus Associate Director of Government Affairs attempted to call a Senior Adviser within the Minister for Communications.
  • 2.34 pm – The outage was resolved (14 minutes after the rollback was requested).
  • 2.37 pm – Optus Emergency Services Support emailed South Australian and Western Australian Police re the outage and its resolution.
  • 2.42 pm – Optus advised the ACMA of an issue, advising they would email when resolved.
  • 2.51 pm – Optus CEO advised of the Triple Zero Outage during an Executive Committee Meeting.
  • 2.52 pm – Optus Government Affairs Associate emailed the Senior Adviser for the Minister for Communications.
  • 2.54 pm – Optus Government Affairs Associate sent a text message to the Minister for Communications’ Senior Adviser.
  • 3.17 pm – ACMA was advised of a significant network outage.
  • 8.08 pm – Contact Centre team advised of the list of people (100 at the time) to contact for welfare checks.
  • 8.43 pm – Fatality Advised by Customer during a Welfare Check
  • 9.01 pm – Second Fatality advised during welfare check
  • 9.33 pm – First tranche of Welfare Checks complete
  • 10.57 pm-11.04 pm – Police in NSW, Victoria, South Australia and Western Australia supplied with referrals after Welfare Checks

September 19

  • 8.13 am – Optus CEO calls Corporate Affairs Office back after a 7.37 am missed call
  • 10.00 am – Additional list of 524 contacts requiring a Welfare Check provided to Call Centre
  • 12.04 pm – Second tranche of Welfare Checks commenced
  • 12.40 pm – Additional Fatality advised of during Welfare Check
  • 2.00 pm – Second Tranche of Welfare Checks complete
  • 3.00 pm – Optus Board Meeting
  • 3.49 pm-4.33 pm – Various calls to and from Department of Communications, Minister for Communications office.
  • 4.36 pm – Email sent to Media about a press conference at 5.30 pm
  • 5.08 pm-5.18 pm – Various calls to South Australian and Western Australian Police
  • 5.45 pm – Media Conference
  • 5.52 pm-5.58 pm – Various calls to Offices of WA and SA premiers

Quick thoughts on that timeline

This was a disaster, for the company, and perhaps many, many people within the company, both at call centres and management levels, failed to escalate in a way that might have mitigated some of the concerns about this whole event.

But what I see most is a failure by the Network teams at Optus to properly escalate this issue high up within Optus.

Stephen Rue is a hands-on leader. He will be roped in that no one thought they could or should simply tell him or his office about this issue. Likely, that will be the focus of the Optus-instigated internal investigation into the issue.

The unbelievable focus by a screaming Senator Sarah Hanson-Young on a timeline relating to Government advice is misplaced. The requirement of Optus to advise the Regulator (ACMA) is clear, but from the timeline we have, it appears that the first time someone in Government relations was aware was at 2.24 pm, the ACMA was advised at 2.42 pm – that’s a pretty quick turnaround for any business in an emergency situation.

What we, the public, need to know is – is there any current risk to any devices or users currently active on our telco networks that they may not be able to contact Triple Zero, and how were those devices allowed to fail?

TECHNICALLY – Are Australians at any further risk – across any or all the telcos?