Company behind global IT meltdown to address failures that led to outage

US cyber security business promises reform following ‘post-incident review’

The US cyber security business blamed for last week’s global IT outage has pledged to overhaul how it issues critical software updates.
In a “post-incident review” of the IT blackout, CrowdStrike said it would reform how it tests and sends out future upgrades.
The bug last Friday was found to have originated in a rogue file sent out to customers globally at 5.09am. CrowdStrike routinely sends out software updates for its Falcon product, which is designed to detect and protect against new hacking threats.
However, the “rapid response” change included a broken file which caused Microsoft’s Windows operating system to crash in a so-called blue screen of death.
CrowdStrike admitted it had since uncovered a fault with its “content validator” technology, a piece of software that is used to test code before it is sent out to customers, which failed to stop the bug.
It added that the file tweaked a previous upgrade, which had until that point run smoothly.
The faulty software was deployed globally to CrowdStrike’s tens of thousands of customers, impacting any machines that were turned on as the changes came online.
The company spotted the bug and sent out a fixed version 78 minutes later, but by that point millions of PCs had crashed.
The company said it now planned to “implement a staggered deployment strategy” for similar changes, rolling them out to only a handful of machines before launching globally – known in the industry as a “canary deployment”.
It also said it planned to provide customers with greater control over how quickly they received these updates.
Friday’s CrowdStrike outage took airlines, train companies, TV broadcasters and NHS systems offline, leading to thousands of flight cancellations and widespread disruption.
Cyber security experts have estimated the cost of the bug could run into the billions of pounds, with some describing it as the biggest outage of all time. 
CrowdStike, which is listed in the US, has lost more than a fifth of its value since the fault emerged.
The bug hit 8.5m computers running on Microsoft’s Windows software. On Monday, the US technology giant blamed European Union rules for forcing it to open up its operating system.
CrowdStrike had so-called “privileged” access to Windows, meaning it could install deep-lying updates into customers’ computers.
Microsoft, which offers its own cyber security technology called Windows Defender, said it had been forced to provide this access in a deal with the EU in 2009 amid a competition investigation.
However, a European Commission spokesman said: “Microsoft is free to decide on its business model to adapt its security infrastructure to respond to threats, provided this is done in line with EU competition law.
“Microsoft has never raised any concerns about security with the commission, either before the recent incident or since.”