Crowdstrike Blue Screen of Death – How a global IT outage can happen

Optus Network outage affecting their entire network across Australia? Hold my beer says Crowdstrike, we can bring the entire world to it’s knees.

That’s how it feels, and with the benefit of some 15 hours to absorb what’s just gone down, we can now look at the issue with a bit of clarity – something we didn’t have late Friday Afternoon here in Australia as the Blue Screen of Death hit millions of computers around the world.

What is Crowdstrike?

Crowdstrike is a global cyber security company. Put simply, think of them as like Norton or Trend Micro, but for big businesses.

They have over 23,000 customers around the world, each likely with hundreds if not thousands of staff and therefore computers to protect.

Listed on the stock exchange they have a value of over $74 Billion and make $255million in revenue per month!

In simple terms, that means on average those businesses who Crowdstrike count as customers are paying something like $11,000 per month.

Their whole goal is to stop your company being a victim of a cyber attack.

Why did computers fail?

Firstly, this was not a cyber attack.

Because Crowdstrike is more than just a bit of software running on your work computer (or that checkin counter at the Airport), it’s deeply integrated into the Windows Operating System wherever an IT team has subscribed to the Crowdstrike Service.

Deep integration means that when they send out new updates to their “Falcon” defence systems, they happen at the Windows Operating System level, so a reboot is required to get them installed and working.

Yesterday afternoon, computers around Australia started getting these updates and there was a bug. A fault in the update, something that computers didn’t like, so the standard way for Windows to handle such an error is to display the “Blue Screen of Death” – essentially a very big, clear error message.

Again, because most of us don’t have the administrative privileges or knowledge, there was nothing a user could do, other than wait.

And at the same time, IT departments were unaware what caused it for a short while, and even when it was clear, had to work out work arounds to “roll back” the update, or bypass it.

Businesses without any on-site IT team might have struggled, and thus the outage lasted well into the night, if not still today.

How did Crowdstrike Respond?

Poorly.

Their Support and IT teams were communicating with customers behind a “paywall” – basically a login area for customers to get support.

There was no public statement until at least 5.5 hours after the incident, at around 7.45pm last night Sydney time, a statement communicated to media around 30 minutes later – from the CEO George Kurtz.

By 1am Sydney time, he had realised one epic failure of his own – he needed to apologise.

By this morning, he’s talking about transparency around what happened and keeping people informed:

In a situation like this, the world, the public and their customers, deserve communication, fast and frequent. It’s really quite simple.

Crowdstrike failed at that.

Will this happen again?

I doubt it will happen to Crowdstrike again, they will have learned their lessons.

At the same time similar companies and any IT software vendors with excessive reach will rethink their testing before deployment procedures no doubt.

Impossible to say we won’t be struck by another outage again – but, let’s hope it’s not as big.

More concerning is the red flag this waves to Cyber Criminals – letting them know how one company can impact so many machines and cause such havoc. Exactly what “bad actors” would be keen on.

That’s our next real threat.