Last week I participated in a panel discussion on the Executive Order’s Impact On Embedded Device Security hosted by ISSSource.com. I signed off with a comment about my biggest worry: someone will combine professional ransomware with a software supply chain attack to create a truly massive ransomware attack.
Then, being a holiday here in Canada, I set off on a good long bike ride, unaware how prescient my remarks were. For that’s exactly what happened over the weekend, courtesy of the Russia-based hacking group REvil, which attacked the Florida-based software company Kaseya Ltd.
If you are not familiar with Kaseya, they provide network and security management services for small to medium-sized businesses (SMBs), not unlike what SolarWinds offers for large businesses. So this is yet another attack taking advantage of poor software security at companies that provide security management products and services. “Quis custodiet ipsos custodes?”*
I don't expect many major OT operators, like the big oil & gas companies, will be impacted. Similarly, US government agencies will probably be okay, unlike in the December SolarWinds supply chain attack. However, this could be a real mess for the industries with lots of smaller operations, such as water utilities, smaller power utilities (like Munis), or the food and beverage industry.
In my experience, industrial SMBs often have a very decentralized security management strategy; that is, it’s every plant for itself when it comes to security. For example, just before the pandemic struck, I met with an OT security manager at a Fortune 500 food and beverage company and asked him what the software approval process was for OT systems at their company. The answer:
"Each engineer or technician downloads the software they need for the PLCs they manage directly from the PLC vendors' websites. They then make their own decisions on whether they should install that software. There is no company-wide strategy to validate the safety or security of that software."
This approach is going to make Kaseya an issue for industrial SMBs for two reasons:
As a result, the fan-out from this attack is going to be nasty and the true impact will trickle out in the weeks and months ahead. I saw the following statement from the cybersecurity firm Sophos:
“At this time, our evidence shows that more than 70 managed service providers were impacted, resulting in more than 350 further impacted organizations,” CSO Ross McKerchar said in a statement. “We expect the full scope of victim organizations to be higher than what’s being reported by any individual security company.”
This “attack once, affect many” result is exactly why software supply chain attacks are on the rise. The ROI is just too attractive. And tracking down all the victims is difficult because it’s hard to tell how many branches there are in the tree.
Another complicating factor is that the victims were using Kaseya products as part of their cybersecurity toolkit. I read a great quote from a Washington Post article on Kaseya:
“From a criminal standpoint it’s a brilliant supply-chain target to take away the tool that’s needed to recover from the threat,” Murray added. “They’re not only encrypting the systems but they’re also taking the recovery tool out of the equation.”
This is going to doubly affect industrial SMBs as many operators are unlikely to have the well-defined, well-tested recovery systems common in more high risk industries.
So what is a small or medium-sized enterprise to do to protect themselves against these supply chain attacks? The answer is simple: Don’t blindly have staff install patches and updates just because they are available.
Industry has become obsessed with deploying security patches rapidly and widely, but a well managed rollout strategy is a better idea.
As far back as the mid-2000s, pharmaceutical companies published papers on how to use a staged rollout strategy to reduce the risk of patches on operational reliability. The figure below is an example from the pharmaceutical company AstraZeneca and shows the patch cycle for their systems. Note that none of the patching is performed in a rush – there is always a process to collect feedback from one stage before embarking on the next stage. You’ll also note that patches are initially trialed in isolated test environments before they are widely deployed across the facility or company.
Source: Joakim Moby, AstraZeneca, ISA Expo 2006
Of course, back in 2006 (or even 2016) the threat of supply chain attacks wasn’t on the security radar screen, so a critical stage is missing from many patch processes: the need to validate that the patch is both legitimate and current. Tools like FACT were designed specifically to make that initial step easy for all companies.
If you’d like to arrange a call for an overview of how FACT can help protect your company from a supply chain attack, click below and we'll set it up.
* “Quis custodiet ipsos custodes?” is the question attributed to first century Roman satirist and poet Juvenal. “Who will guard the guards themselves” is the rough translation of the Latin phrase.