OT: Potentially largest IT outage in history today

horshack.sixpack

Well-known member
Oct 30, 2012
9,065
5,068
113
It shut Delta Airlines down. Wild stuff.
Context:

Here's the impact of IT disruptions listed as bullet points for each country:​

- Australia: Media, airlines, supermarkets, banks, and hospitals impacted.

- Belgium: Train ticket sales, digital announcements, media, banks, airports, and government services disrupted.

- Canada: TD Canada Trust app and Vancouver International Airport affected.

- China: Widespread blue screens, some businesses allowed early dismissal.

- Croatia: Health information system and air traffic control issues.

- Czech Republic: Prague Airport affected.

- France: TV channels and Paris Olympics systems disrupted.

- Germany: Berlin Airport halted flights, Lufthansa affected, hospital operations canceled.

- Hungary: Budapest Airport issues.

- Hong Kong: Airport check-in delays, airline booking systems down.

- India: Major airlines and IT firms impacted.

- Israel: Emergency services, hospitals, and banks affected.

- Japan: Spring Japan airline experiencing issues.

- Malaysia: KTMB railway ticketing system issues.

- Netherlands: Schiphol airport, banks, and medical services disrupted.

- New Zealand: Banks, supermarkets, Auckland Transport, and Christchurch Airport were affected.

- Philippines: Major banks, telecommunications, airlines, and government websites down.

- South Africa: Banking issues.

- South Korea: Jeju Air is experiencing issues.

- Singapore: Changi Airport delays, various service disruptions.

- Spain: National airport traffic control IT outage.

- Switzerland: Zurich Airport halted landings.

- United Kingdom: News channels, airports, rail companies, NHS, and various services disrupted.

- United States: Airline ground stops, 911 service disruptions, Microsoft and CrowdStrike shares dropped.
 
  • Like
Reactions: Pars

Dawgg

Well-known member
Sep 9, 2012
7,584
6,171
113
Wow! I still say the worst virus I’ve ever had is Microsoft Update. Antivirus software is almost worse than the viruses. I know that’s a gross exaggeration & not really true. But it sure seems like it sometimes.
My work computer is forever updating.
 
  • Like
Reactions: patdog

TrueMaroonGrind

Well-known member
Jan 6, 2017
3,674
853
113
Starbucks was all screwed up this morning. I never get Starbucks but decided to stop by today. The app was receiving order but the stores couldn’t see them. It was a hot mess
 

pseudonym

Well-known member
Oct 6, 2022
2,489
3,556
113
Starbucks was all screwed up this morning. I never get Starbucks but decided to stop by today. The app was receiving order but the stores couldn’t see them. It was a hot mess
Funny, I randomly went to Starbucks today as well. After I paid at the register, people started coming in with mobile orders. They just told people to tell them what they ordered, and they prepared it for free. I felt stupid standing there after paying full price...
 

NTDawg

Well-known member
Mar 2, 2012
2,086
664
113
Pretty convenient how this got resolved in time for me come to work this morning
I received a call this morning at 6:30 and they thought at the time that it was all Microsoft systems. I preparing for the flood of calls and texts if we should open, like when a winter storm is coming I get constant calls that it is snowing in Canton so we need to close.
 

Dawgzilla2

Well-known member
Oct 9, 2022
863
1,006
93
Funny, I randomly went to Starbucks today as well. After I paid at the register, people started coming in with mobile orders. They just told people to tell them what they ordered, and they prepared it for free. I felt stupid standing there after paying full price...
Starbucks has technical issues on a regular basis, so not necessarily related to the Crowdstrike problem.

The baristas should have made the mobile order people show them their orders on their app. That's SOP.

Some people will try to take advantage of the situation by showing a screenshot of an old order, or claiming they are picking up someone else's order, but if someone wants free coffee that badly, just give it to them.
 
  • Like
Reactions: Trojanbulldog19

pseudonym

Well-known member
Oct 6, 2022
2,489
3,556
113
Starbucks has technical issues on a regular basis, so not necessarily related to the Crowdstrike problem.

The baristas should have made the mobile order people show them their orders on their app. That's SOP.

Some people will try to take advantage of the situation by showing a screenshot of an old order, or claiming they are picking up someone else's order, but if someone wants free coffee that badly, just give it to them.
This was related to the CrowdStrike issue. The barista said, "Microsoft got hacked, so just tell us what you ordered, and we'll make it."

They may have been deviating from the SOP under the pressure of the situation.
 

Dawgzilla2

Well-known member
Oct 9, 2022
863
1,006
93
They may have been deviating from the SOP under the pressure of the situation.
They were just making things harder for themselves that way. Like I said, Starbucks has these kind of problems a lot, so the baristas should know how to handle it.

On the upside, the cost to Starbucks for a drink is practically zero, so the only people affected are the baristas and the customers...and Starbucks doesn't seem to care about either of those
 

horshack.sixpack

Well-known member
Oct 30, 2012
9,065
5,068
113
"Yeah, who needs QA? Gotta squeeze out a little more stock price for our investors so I can get that sweet sweet bonus though." - CrowdStrike executives, probably.
I even wondered about supply chain attack. One compromised GitHub repository before you packaged the "tested" build. Should not have that kind of process/vulnerability, but the idea that a "content update" could cause a Windows BSOD loop was not on my risk bingo card.
 

horshack.sixpack

Well-known member
Oct 30, 2012
9,065
5,068
113
Fortunately we use a different antivirus provider. I don't envy anyone having to roll out frequent updates on such a large scale.
Current process. Login to the machine in safe mode-> delete a specific file. Apparently many of these machines can't take an update from Crowdstrike now to fix an update from Crowdstrike. Very manual solution that does not scale, so far.
 
  • Like
Reactions: Trojanbulldog19

QuadrupleOption

Well-known member
Aug 21, 2012
1,012
804
93
Honestly if you see a BSOD, your first thought is Microsoft.
I don't know how it is now but back in the day the actual issues were typically bad drivers written by third-party developers causing kernal failures. It's why Microsoft introduced their driver certification process, and provide tested generic drivers for most commonly used peripherals.
 

QuadrupleOption

Well-known member
Aug 21, 2012
1,012
804
93

Oddly enough I'd never even heard of CrowdStrike before this kerfluffle. I guess I should google it.

EDIT: Ooh, software designed to protect systems from cyberattacks.

Alanis Morissette Reaction GIF by MOODMAN
 
Get unlimited access today.

Pick the right plan for you.

Already a member? Login