Optus confirms outage source and offers Data to customers

The outage began around 4am on Wednesday, was rectified around 10-11 hours later and now, a full 36 hours since the outage began, the Optus CEO has come clean about what went wrong.

Confirming the various rumours in telecommunications and IT circles, it’s all down to the routers.

In language designed to be appropriately digested by the average Optus Customer, Optus says “In common with major global telecommunication networks, the Optus network is designed with multiple layers of fall back and redundancy. At the heart of this is a modern intelligent router network developed with the world’s leading vendors.

Despite this, a network event yesterday triggered a cascading failure which resulted in the shutdown of services to our customers.”

And it is complex. With millions of devices connected to the network, the routers act as a traffic manager, and a giver of directions to the many requests that pass through them.

But Optus go no further into the detail, only to say “Our engineers are investigating thoroughly and we will learn from this outage and continue to improve”

This router failure was suspected by many as early as half way through the outage, given there are not many pieces of the network puzzle that both mobile and home broadband services are dependent on.

It’s believed, though not confirmed by Optus, that one of the “Route Reflectors” used across the network, designed in a series to offer redundancy from failure was updated with new configurations or commands, and may not have been fully tested before additional devices were administered with any required changes. This caused all of the devices to accept as fact the information coming from that one device, and in turn cause chaos on the network, and disrupt the main gateway thus slowing or grinding traffic to a halt.

Additionally, EFTM understands that with the network down, Engineers were unable to remotely access the devices to install any fix, making the whole process slow and combersome.

To her customers, CEO of Optus Kelly Bayer Rosmarin says “We’re deeply sorry for yesterday’s outage. We know how important connectivity is to all our customers, and that we let you down.

“We truly appreciate our customers’ patience and understanding as we worked to restore our operations yesterday.

“We know that there is nothing we can do to make up for yesterday and what customers want most is for our network to work all the time – which is our number one priority – but we also want to acknowledge their patience and loyalty by giving them additional data to help during the holidays, when so many people consume more data with friends and family.”

As a “thank you” or “sorry”, Optus is offering 200GB of extra data to postpaid customers which can be activated from next week, and prepaid customers will have unlimited data on weekends until the end of the year.

Extra Data sounds nice, but in reality its almost free for Optus to dish out, and the vast and frankly overwhelming majority of Mobile users already have far more data than they need.

To those businesses affected by the outage, Optus says “We understand some businesses were uniquely impacted and encourage any with concerns to contact their Optus Business Centre or Business Care on 133 343.“

All a bit too little, too late really.