Telstra apologises to NBN and ADSL customers for Friday’s outage

Fronting the media for the first time since their fifth major outage this year Telstra Executives have today expressed their apology to customers who were affected by last Friday’s outage on some ADSL and NBN services.

Kate McKenzie, Telstra Chief Operations Officer told EFTM that “10% of Telstra broadband customers” (some 370,000 accounts) “were offline overnight on Thursday into Friday morning” with still 33,000 off-line for much of Friday with a residual 15,000 customers offline into the weekend.

This outage was caused by an update to a server (DNS Server) at Telstra.  The software update caused the server to fail, and the flow on effect was disastrous for Telstra’s engineers and of course the customers.

Each of the Telstra “Gateway Modems” at customers homes and businesses are programmed to “ping” the network regularly, and if they detect any failure, automatically reboot.  Because as the best IT people will tell you, the best test is to turn it off and turn it back on again – so it’s a little bit of genius.

However, little known to Telstra some of those devices simply had no ability to recover from the outage which was quite short-term, and as a result became trapped in a cycle of reboots.

While many were restored, many were left offline and the customers very unhappy.

Telstra had to scale up their call-centre and social media support teams to cope with the influx of rather unhappy customers.

Fortunately, a factory reset seems to fix most of the residually impacted devices.  However, that’s not an easy thing for many average Australians for whom Telstra has the bulk of the market share.

As it stands, any modems still affected are being replaced for free.

When it comes to compensation, Telstra reckon their customers aren’t interested in another free data day, so they are making individual apologies to customers as they get in touch.

That “apology” could be a $25 credit on your bill, so if you were impacted by the outage, Telstra will be crediting your account in the next billing cycle.

If only the customers affected into Friday claimed that rebate, it would cost Telstra less than $1,000,000, however if all 370,000 think they are entitled to the rebate (unlikely if your internet was down overnight while you slept) it would cost closer to $10,000,000!

Telstra are confident this issue is completely unrelated to the previous mobile outages earlier this year, however with the scale of their customer base Telstra are always going to be in the spotlight.