Vulnerability Tracking

The Wretched State of OT Firmware Patching

By Eric Byres on October, 11 2023

Back

This blog is a follow-up to our first post on the 2023 Microsoft Digital Defense Report where I described our collaboration with Microsoft on identified exploitable OT vulnerabilities.

There is a saying in the IT world: patch early, patch often. It refers to the best practice of ensuring software is always running the latest version with the least number of bugs and vulnerabilities.

It’s good advice in the IT world, but for OT it is often impossible. PLCs cannot just be turned off and patched when tons of logs are racing through a sawmill, dependent on those PLCs safely guiding their journey. Nor can the PLCs controlling the painting process be stopped while an automobile assembly line is still running. They’re not like executive laptops that can be rebooted at midnight when they need the latest Windows update.

The need to ensure that an update will have no accidental impact on safety or production inevitably delays patching. Most guidelines for OT patch management suggest that companies push down patches to machines on a priority basis. This takes time: the pharmaceutical company AstraZeneca estimated back in 2006 that safe patching of OT systems requires 34 working days from when the patch is first released to when the most mission-critical OT device is patched.

Is that Patch Safety-Certified?

Another consideration that can limit timely updates are certifications. For example, a SIL Safety Certification is often required in situations where a PLC is controlling a hazardous process. Obtaining certifications is expensive so equipment manufacturers often don’t pursue certifications for every firmware version. Asset owners may thus need to skip some patches until the next certified version is available.

In reality, the patch cycle for OT devices can be years long for processes where high availability is critical. The costs involved in stopping are significant, so typically these patches are scheduled on an annual or biannual basis as part of a normal maintenance cycle.

Or at least they should be.

That Excuse Expired

What we discovered while digging into the distribution of firmware versions across Microsoft customers’ PLCs shocked me. Across various models of PLCs, 60% were running ancient firmware versions with 8 or more exploitable CVEs. And even more astonishing, updates have been available for over 10 years!

Yes, there are more patching constraints in OT than in IT. But that excuse eventually expires.

The Risk of Exploitable Vulnerabilities

Old firmware invariably means more CVEs, and the data bears that out. For example, firmware versions prior to 2012 had roughly 11 CVEs. But in 2012 a new version was released reducing the number of CVEs to around 2. Clearly the manufacturer invested the time and energy to eliminate vulnerabilities and improve the quality of their product. But that significantly lower-risk firmware was often never installed by the asset owners, even though it has been available since we all started dancing Gangnam Style.

Running firmware with 11 exploitable vulnerabilities introduces an unacceptable (and unnecessary) degree of risk. Yes, many old PLCs use inherently weak communications protocols (as Dale Peterson says, “Insecure By Design” protocols), but that has improved significantly over the past decade. Now most PLCs offer a secure communication option, but that is available only if you install the current firmware version.

The Reasons for Delinquent Patch Management

So why are control engineers not updating their firmware? It seems to be more of a people problem than a technical problem given the availability of the patches. I think these are the most likely culprits:

An “if it ain't broke, don’t fix it” philosophy
Lack of awareness
Procrastination

The Low-Hanging Fruit

The upside to our findings is that many asset owners can realize a significant reduction in risk by patching their devices.

Through our research with Microsoft, we determined that if operators updated their firmware to the latest version, the number of devices free of exploitable CVEs would increase from 4% to 40%. This is a significant improvement, but there’s still plenty of risk in the remaining vulnerable devices. To quote the sage philosopher Jon Bon Jovi, “Whoa, we're halfway there, Whoa oh, livin' on a prayer.”

Here’s another way to look at the impact of updating those PLCs to the latest version: asset owners could reduce the percentage of the devices with more than 8 exploitable CVEs down to only 18%. Again, that’s still a lot of vulnerable PLCs on the OT network, but the goal should be improvement rather than perfection.

If, for whatever reason, OT asset owners choose to remain using ancient firmware, they can at least look at hardening configuration settings or using compensating controls (I wrote about this a decade ago - check out Solving the ICS Security Patch Problem). Of course, perhaps the owners of those unpatched PLC had applied compensating controls to protect their systems from all those vulnerabilities, but I doubt it. In most cases deploying and maintaining compensating controls takes far more effort than installing patches.

The research we did with Microsoft shows that far too many asset owners have leaned into the “we can’t patch this OT asset” excuse for far too long.

Microsoft’s key takeaway from the research was that having visibility into OT assets, patch levels, vulnerabilities, and availability of updates is an important part of a comprehensive OT risk management program.

Eric Byres

Eric is widely recognized as one of the world’s leading experts in the field of OT, IT and IoT software supply chain security. He is the inventor of the Tofino Security technology – the most widely deployed OT-specific firewall in the world. When not setting the product vision, or speaking at a conference, Eric can be found cranking away on his gravel bike.