Optus CEO Kelly Bayer Rosmarin and her head of networks Lambo Kanagaratnam appeared in front of the Australian Senate’s Environment and Communications References Committee to answer questions about last week’s entire network outage, given the chance to explain their position on issues and processes around the outage which affected every single Optus consumer and Small Business customer.
The learnings were fascinating, but in many cases a touch confusing too – but much of that is down to the need to bring highly technical terms into a mainstream framework.
Key to the learnings today, was clarification that a routing update at the SingTel internet exchange was passed through to the Optus Network, and all parts of the network accepted that change and due to the complexity of the change or errors within it, around 90 routers on the Optus Network went into a “fail safe” mode after a pre-set safety limit was reached. A Fail Safe is meant to protect a network and devices like this, but unfortunately this worked entirely against Optus.
Additionally, the outage itself made it hard for engineers to access the network to make any changes, so physical attendance at a vast number of sites to physically reboot the equipment and commence bringing the network back up.
Frankly, it seems a dramatic oversight of the core IP network to allow a routing change at an external third party to have such a dramatic effect on the Optus network.
Optus Network engineers were working on this from just after 4am. That they were unable to narrow down the issue until around 10am. That’s six hours after the outage began.
That duration will long be the subject of analysis within Optus, but it does show some faults within the Optus network operations centre. Optus themselves say they have rectified the issue so it cannot happen again, that they are able to do so in such a short space of time is itself an indication of how avoidable this outage should have been.
CEO Kelly Bayer Rosmarin faced strong criticism about the failure of the Triple-Zero calls on the day of the outage, telling the Senate committee “There were 228 triple-zero calls that were unable to go through, and we have done welfare checks on all of those 228 calls. And thankfully everybody is okay,”
Mrs Bayer Rosmarin was unable to articulate why this is, only to point to an upcoming government enquiry into that, though without seeing any evidence of another outage on the day, it can surely only be linked to the Optus outage, and thus, Optus has responsability.
There was much questioning of Optus’s CEO around the potential for compensation for losses to their customers. Interestingly, it was revealed that so far 8,500 customers had sought out compensation from Optus which totalled $430,000, with just $36,000 paid out so far.
However, Mrs Bayer Rosmarin made some valid points about the potential precedents that might come from Compensation demands if such a thing was to be legislated. “On the question of extending financial compensation for consequential losses, we understand that this would create a new precedent that would extend far beyond Optus and apply to all other telecommunications providers, as well as other providers of essential services, critical infrastructure and public services. This makes it a much broader policy question for government that would have far-reaching implications across many sectors of the economy and the cost of these services for Australian consumers.”
Consequential losses refers to the impact of an outage or failure that the consequences it has on a person or business.
Interestingly, if we look at that from a power company perspective, are we to assume businesses are compensated when their power goes out? Or their Water?
In many cases, the failure here is actually on the individuals and businesses who put their entire communications strategy in the hands of one telco. If your EFTPOS terminal only works on Optus, I’d suggest Urgently arranging a backup SIM or an alternate WiFI Internet provider.
How Optus themselves can be held liable is difficult to assert.
However, it should be noted, that from a customer care point of view, it would be in Optus’ best interests to work with their customers on compensation where possible – just from a pure Customer Service outcomes perspective.
The handling of the technical outage behind the scenes is difficult to fault, of course, the team would wish it never happened, but it was pointed out numerous times, and not ignored, but I think not well understood by the Optus CEO that their communication on the day was poor to their customers and the media.
It’s a broken record replayed from the Cyber Attack when the Company thought that written statements to the media were enough to get a blanket message out to their customers.
The facts are, The outage commenced at 4.05am. I personally called the Optus Media line at 5.01am and was told to send an email, and got nothing until 6.35, by which time I had done countless radio and TV crosses already. 20 other media calls were made in that same hour and a half window.
CEO Kelly Bayer Rosmarin didn’t directly answer the question as to when she was aware of the outage, but she did say she woke up, had no service and went straight to the office. It’s been reported she arrived at the office around 7.30, so basically the whole world had a two and a half hour head start on the Optus CEO.
Optus executives couldn’t communicate with each other as their network was down, but it was confirmed today that key executives do have alternate SIM cards from other telcos.
Why, in 2023 those were not installed as Dual SIM operating services on their phones so they could be contacted at any time – is beyond me, and itself a massive oversight by the company’s executive, risk management team.
Optus has a lot to do to win back customer support. Did they lose a vast customer base? It’s unlikely, and we won’t know for a little while.
One thing’s for sure, Complaints to Optus and to the Telecommunications Industry Ombudsman are key metrics for the CEO, so if you are an upset customer, don’t just rant at your mates, pick up the phone or send an email – otherwise the company will assume they did a top job.