S4x23 SBOM Challenge — Part 1

The aDolus Team has just returned from participating in the SBOM Challenge at the S4x23 cybersecurity conference in Miami, Florida. This blog is the first of a series reporting on what we did during the challenge and what we learned from the process.

The challenge was run by Idaho National Labs (INL) who noted “the S4x23 conference has organized a challenge event to explore the functionality of tools available in the SBOM market and share information about SBOMs and tools with S4x23 attendees.”

INL specifically looked at Software Bill of Materials (SBOM) creation and ingestion, vulnerability analysis and management, and Vulnerability Exploitability eXchange (VEX) document ingestion. The challenge wasn’t framed as a head-to-head bake-off but rather a more thoughtful survey and comparison of the different approaches available in the SBOM market. INL advised that “No scoring or comparative evaluation of the tool’s performance will be provided. However, where informative, functionality descriptions or screenshots may be used to illustrate differences in approach or show different ways specific tools performed analysis and presented results.”

Our team reported back that this was, for the most part, a really collegial affair with lots of SBOM-vendor cooperation and a genuine desire to drive more education about SBOMs. We'd like to thank the other participants for bringing different perspectives and approaches to the challenge.

The Targets

For the challenge, INL presented the challenge participants with three software/firmware targets for analysis:

The Tasks

INL then asked the participants to do three tasks for each target:

Day 1: Generating SBOMs and Identifying Vulnerabilities

Using our FACT platform, we broke down two of the three targets in less than an hour. Here is a summary of the extraction results:

Analysis Results by Artifact

Windows Installer

This Windows-based artifact was the installer for OSIsoft’s PI ProcessBook 2015 R3 Patch 1. It contained 30 unique products from 3rd-party software suppliers like Microsoft, LEAD Technologies Inc, and Rogue Wave Software. FACT also reported that the file was signed with a certificate chain that did not match a qualified certificate chain for OSIsoft. This was not unexpected, as the original installer had clearly been modified for the challenge.

FACT’s AI vulnerability detection initially suggested some potential vulnerability matches, in particular for the WIX Toolset (CVE-2019-16511). However, analysis showed that although the WIX toolset used in the binary was a vulnerable version (3.9), the software did not include the Microsoft.Deployment.Compression.Cab.dll needed for this vulnerability to be exploited. Thus, this vulnerability was considered non-exploitable and not associated with the product. 

Similarly, the installer contained 12 Lead Tools DLLs. While some versions of Lead Tools do have known vulnerabilities, FACT quickly determined that the version used in this artifact wasn’t one of the vulnerable versions. 

In the end, we were confident that there were no known exploitable vulnerabilities in this artifact.

Bare-Metal Binary

This “bare-metal” binary came from an ABB/Hitachi TPU 2000R Transformer Protection Unit that was developed in the early 1990s. The symbol tables had been stripped out and it wasn’t based on a known operating system, such as Embedded Linux, QNX, or Windows. As a result, we weren’t able to decompose this “bare-metal” binary into a detailed SBOM in the 10 hours provided, and frankly, no one else could either.

Binaries that aren’t based on a known operating system but are custom developed for an ancient chipset like Motorola 68000 are rare in 2023. They can be analyzed, but it requires extensive custom engineering that wasn’t feasible in the time provided. Because of their rarity, “bare-metal” binary analysis is an edge case that really isn’t suited for SBOM analysis. That said, if someone really wants us to look at microcontroller firmware, we’d be happy to (for a price 😉). 

Embedded Linux

The embedded Linux artifact had an abundance of open-source software with numerous vulnerabilities. FACT was able to confidently identify 18 common applications, such as Linux kernel, OpenSSL, Postfix, BusyBox, Sqlite, Python, and Broadcom firmware — and many of these had documented vulnerability disclosures. FACT also alerted us to possible malware in a text file that contained “suspicious Linux command line commands,” but upon further analysis, this was found to be a false positive.

Note that when analyzing software there can be a lot of additional analysis beyond CVE vulnerability counts. For example, FACT found 178 high risk executables, 886 buffer overflows, and 66 command injection flaws. While interesting as an indicator of software quality, this information can create a lot of false positive issues for the end user if compensating measures such as stack protectors are present. Since testing SBOM and VEX vulnerability capabilities was the focus of the challenge, we omitted this additional depth from our submissions.

1-Click SBOMs

After completing this analysis, we generated — with just the click of a button — NTIA-compliant SPDX SBOMs for each of the three targets (albeit the bare-metal SBOM was pretty “bare”). Along with these SBOMs, we submitted three separate Vulnerability Disclosure Report (VDR) JSON documents. Why were the SBOMs separate from the VDRs? Because NTIA’s best practice guidelines recommend “that vulnerability data be tracked in separate data structures from the SBOM” because of the “dynamic” nature of vulnerability data compared to a static SBOM. 

We strongly agree with NTIA and CISA on this guideline because our vulnerability data is continuously updated for every binary in our database. Just wait a few months and all of these targets could have new vulnerabilities popping up. We don’t want stale vulnerability data being passed around in a static SBOM. Instead we allow our partners to monitor their SBOMs continuously through an API where they can access the most up-to-date vulnerability data on demand.

The Threat of False Positives

You probably noticed we mentioned detecting and managing false positives several times in this blog. False positives are the curse of both the vulnerability and malware detection worlds. Like the boy crying “Wolf” in Aesop's famous fable, a technology that cries “Vulnerability” where none exists (False Positive), soon becomes ignored even when real (True Positive) vulnerabilities are present. This is especially true for SBOMs. Our own Eric Byres presented last year at S4x22 on the risk that SBOMs could generate a tsunami of false-positive vulnerabilities that would overwhelm security teams. Just because a platform claims to find dozens or hundreds of vulnerabilities in an SBOM doesn’t mean it is a better solution. 

At aDolus, we’ve taken the position that fewer, more precise results are better than a lot of low confidence reports. A key feature of FACT is its ability to reduce false positives and provide less noisy, more actionable intelligence. We take great care to minimize false positives for our users by only presenting high confidence vulnerabilities to our partners based on calculated probabilities of association. This means our AI system vulnerability analysis coverage is significantly beyond “embedded Linux firmware” and designed to maintain precision across a broad spectrum. For any of those data science junkies out there, we’ve basically favored precision over recall. 

Stay tuned for Part 2 where I’ll go into more details on SBOM and VEX ingestion — the primary tasks during Day 2 of the challenge.

And if you would like to see how FACT generates SBOMs, we'd be happy to show you!