for those of us who hold CrowdStrike, @Barkman Turner Overdrive and me at a minimum....
(credit to BTO as it is his joke)
(credit to BTO as it is his joke)
OT: Potentially largest IT outage in history today
- Thread starter horshack.sixpack
- Start date
Our company switched to Crowdstrike last year. I hate them. There's no easy way to limit their updates to certain computers as early adopters, so our security didn't. Four times they've randomly taken down some clients because of an update incorrectly shutting down services.
I'm now the preacher with a lot of converts.
Meanwhile, I have team members with likely-bricked laptops because they can't even get to the screen required for the workaround...
I'm now the preacher with a lot of converts.
Meanwhile, I have team members with likely-bricked laptops because they can't even get to the screen required for the workaround...
I thought it was because Jackson is so great, a Utopia really, that it did not need a 911 system at all.**
Probably like constant pressure to beat unrealistic schedules with current workload of staff and higher ups trying to cut corners to get it out
I don't know anything about security but I do know my computer has been much slower since we switched to CrowdStrike.
My sister in law manages the systems at Le Bonheur Hospital in Memphis and they didn’t have a total shut down but it’s pretty bad.
So far, the only post I've seen from someone at Crowdstrike was tone deaf. It started with "Today was not a security or cyber attack. Our customers remain fully protected."
Cybersecurity has 3 primary elements that need attention: Confidentiality, Integrity and Availability. Crowdstrike knocked the one of the legs off a three-legged stool and have decided that customers are "fully protected". They basically crafted a new DDOS attack and are cool with it.
ETA: that "someone" was the CEO...
Cybersecurity has 3 primary elements that need attention: Confidentiality, Integrity and Availability. Crowdstrike knocked the one of the legs off a three-legged stool and have decided that customers are "fully protected". They basically crafted a new DDOS attack and are cool with it.
ETA: that "someone" was the CEO...
Last edited:
Coworker of mine, has been in ATL since 4pm yesterday. Flight delayed by weather, then the CrowdStrike hit. I asked him why he didn’t rent a car. No one would rent one way, thus they end up with no cars.
I received guidance my IT for how to delete it and it worked. Took several steps, but essentially just deleted what CrowdStrike pushed last night.
IT folks: How in the world does this kind of bug get through UAT and staging to make it to production?
Also, why did CrowdStrike not have the ability to immediately revert back to previous version?
I am no IT person, but as a business owner for an IT system that has regular deployments, I have made the call to revert back to previous version in real time when we realized the updates pushed to prod had a bug. Surely they realized very quickly something was wrong.
IT folks: How in the world does this kind of bug get through UAT and staging to make it to production?
Also, why did CrowdStrike not have the ability to immediately revert back to previous version?
I am no IT person, but as a business owner for an IT system that has regular deployments, I have made the call to revert back to previous version in real time when we realized the updates pushed to prod had a bug. Surely they realized very quickly something was wrong.
About 3 weeks ago I tried to rent a car thru Enterprise to drive from Orlando to Memphis. The transfer fee alone for MCO->MEM was $1,200.
First thing I thought about this morning... bunch of those people stuck in airports are screwed even if they had short flights.
They probably finally turned off mobile ordering for everyone eventually. That will be configurable after the mess today if it wasn’t already.
The first store I tried to mobile order from had nothing available other than black coffee. I’m assuming they cleared their inventory so everyone would stop mobile ordering.
There are lots of reasons. Poor Quality assurance is probably the cause. A company that provides that type of product should have a very robust compliance process. If you get cheap that compliance process will get whittled down and mistakes can make it to your end users.
I would never work in a job that could have the impact of CrowdStrike. There are so many devices they support the complexity is unreal. It would be hard to sleep knowing if you made a mistake this could happen.
Have they tried turning it off and back on?
Yeah. This is worse than most of the “real” viruses. If they screw up the response, this is the kind of thing that can put a company out of business.
On my work pc, I can't even start Windows in safe mode without an authentication key. I tried installing an app on my phone to get the code but it's not working. I called our company tech support and was on hold 30 minutes.
Finally I decided to have a tech inspired vacation day and mess with this next week. What a cluster17.
Finally I decided to have a tech inspired vacation day and mess with this next week. What a cluster17.
I feel like they will be. Too many companies affected.
The damages have to be in the tens of billions and heading north. Not only did you impact your clients' sales and customer care, but hospitals were impacted, including scheduled surgeries.
Yeah that's the kind of crap you have to testify in front of Congress for. Not only the lawsuits coming their way.
Had some folks flying out n southwest today. Good thing southwest is still using DOS
That’s the issue. With my organization, they are having to manually fix each computer or give customers the key.
The fix is easy, took me 5 minutes to do it myself once I had access and could open the C prompt. It’s just a matter of IT working through all the calls for each computer at the company
That guy is wearing my shoes. However Crowd strike went down over 11 percent today, did not do MSFT any good either.
I have a friend who allegedly works at Intel, who allegedly said that the line to walk into the IT support center was across a few buildings. Multiply this by a few hundred OR thousand companies and you see a lot of happy trial lawyers
I believe it. I called our IT at 7:30 and I was 240th the que. By 8:30 there were 500. A lot of people didn’t call because they thought IT would fix access to the system, but then an email went out in the afternoon that people had to contact IT directly or go by in person to get their computer specific key for the fix. That flooded tech support and I think most folks just went home.
If you use Microsoft as a mission critical server, you deserve this. It's a personal computer OS that's embarrassingly easy to bring down and not fully thought out for situations like this.
Last edited:
Expect to see more of these type of events in the future. With there being a push to cloud for all SaaS apps and cloud hosting, the We are an interconnected global economy now. Everything relates to the other, and causes a domino sequence.
The only way to combat these type of failures is to have better planning, testing backup plans, etc. While not realistic, the best way to plan to to have a fleet of offline devices at all time, have a copy of all data offline (Airgapped), and have multiple vendors for different products. Have ATT and Verizon, have Mac and Windows OS. Have multiple EDR Solutions, etc.
I will admit this one is a new one no one has seen before. The main issue with this event was that it required boots on the ground for physical endpoints. This wasn't a situation that was isolated to a single organization like a typical Ransomware event where you could bring in an IR firm on reinforcements.
You can rest assure that our adversaries (China, Russia, Iran, and North Korea) has taken note. The best way to have the biggest impact is to infiltrate the "supply chain". An example of this was back when SolarWinds was compromised via updates a few years back. You hire a developer and gain trust in the software development process, you get the access you need and learn the ropes of the approval processes. You learn the culture and determine the checks and balances, then you slip in a little code over time and have it deployed.
While this wasn't a compromise, it was similar in that a single piece of software used global by all organizations was impacted.
Imagine having the ability to remotely "kill switch" all devices (Windows, Nest, iPhone, etc.)
One day this will occur, and when it does all hell will break loose.
The only way to combat these type of failures is to have better planning, testing backup plans, etc. While not realistic, the best way to plan to to have a fleet of offline devices at all time, have a copy of all data offline (Airgapped), and have multiple vendors for different products. Have ATT and Verizon, have Mac and Windows OS. Have multiple EDR Solutions, etc.
I will admit this one is a new one no one has seen before. The main issue with this event was that it required boots on the ground for physical endpoints. This wasn't a situation that was isolated to a single organization like a typical Ransomware event where you could bring in an IR firm on reinforcements.
You can rest assure that our adversaries (China, Russia, Iran, and North Korea) has taken note. The best way to have the biggest impact is to infiltrate the "supply chain". An example of this was back when SolarWinds was compromised via updates a few years back. You hire a developer and gain trust in the software development process, you get the access you need and learn the ropes of the approval processes. You learn the culture and determine the checks and balances, then you slip in a little code over time and have it deployed.
While this wasn't a compromise, it was similar in that a single piece of software used global by all organizations was impacted.
Imagine having the ability to remotely "kill switch" all devices (Windows, Nest, iPhone, etc.)
One day this will occur, and when it does all hell will break loose.
Don't be surprised if we learn it was intentionally done by an employee.
We may have worked with McIntosh, but we hesitated to install an update when it came out. We always waited at least a couple of weeks to ensure there weren't any problems with other users before we installed the updates, and having a backup of everything makes it easy if you had to roll back. We did have a couple of stations that worked on Windows, but I didn't update them very often.
I think it would be a lot harder for anyone to do this intentionally than for it to just be a 17 up & a failure in quality control.
Get unlimited access today.