malware

Harnessing FACT for Swift Cyberthreat Response

By Marcello Delcaro on May, 1 2024

Back

In the cybersecurity world, anticipation and rapid response are crucial in safeguarding against emerging threats. Recent events, such as the discovery of a vulnerability in the XZ Utils library used in many software applications, underscore this need.

For background, a weakness was deliberately created in the open-source compression library liblzma (XZ). This library is a downstream dependency of sshd — a remote connection library typically used for administration and is widely deployed across the software industry.

Affected versions can include 5.6.0 (released February 24, 2024) and 5.6.1 (released March 9, 2024). Because these two versions are relatively new, it’s probable that the spread is minimal.

This overview from Akamai Security sums up the level of covertness applied to this attack: “In what seems like an attempt to avoid detection, instead of pushing parts of the backdoor to the public git repository, the malicious maintainer only included it in source code tarball releases. This caused parts of the backdoor to remain relatively hidden, while still being used during the build process of dependent projects.”

The following graphic demonstrates the execution chain:

Akamai Security graphic showing compromise of liblzma

Original graphic by Akamai Security, redesigned for use in this blog post

But while this particular threat was mitigated quickly after a random developer stumbled across it, the event serves as a reminder of the continuous risks in software dependencies. In this post, I’ll share how a platform like FACT can be instrumental in preparing for and responding to similar cybersecurity challenges. In a digital landscape — be it IT, OT, or IoT (or most often a combination of all three) — we face increasingly sophisticated threats to the software supply chain.

Understanding the Cyberthreat Landscape

The backdoor found in liblzma versions 5.6.0 and 5.6.1 not only underscores the necessity for rapid detection and response mechanisms but also casts light on broader issues within the open-source community. These include the mental health of open source developers (often unpaid) who face increasing pressures to secure and maintain critical libraries, and the extensive measures that malicious actors are willing to undertake to exploit these people and systems. A relentless campaign to get a compromised version of the library merged into the GitHub repository eventually bore fruit once the bad actors secured the trust they needed.

Such vulnerabilities highlight the complex interplay between technological diligence and human factors in cybersecurity.

A really good technical timeline was written by Evan Boehs: Everything I know about the XZ backdoor. It is incredible to see the lengths the threat actor went through to introduce this vulnerability first into the XZ library and then into a number of common Linux distributions like Fedora and Debian.

The Response Investigation from aDolus

On March 29th, Microsoft developer Andres Freund uncovered the liblzma backdoor; aDolus quickly mobilized our resources to assess and communicate the potential impact on our customers. Our data lake contains software frequently deployed within critical infrastructure and operational technology (OT) environments, where security breaches can have particularly severe consequences. We used FACT to help triage the incident for our customers (and their suppliers) by rapidly identifying the presence (or absence) of the XZ vulnerability and advising on mitigation opportunities.

Here's a detailed look at the steps we took during this incident…

What aDolus did first was perform a comprehensive scan through FACT’s database for any metadata related to the affected libraries. This incisive search was done to identify all instances of the affected versions across our entire database, ensuring comprehensive threat coverage with no blind spots.

In our examination, we discovered approximately 250 instances of the library with versions ranging from 5.0.4 to 5.2.9. A review of historical commits indicated that Jia Tan, the malicious maintainer, began contributing fixes around version 5.2.10. However, malicious code was not introduced until the versions 5.6.0 and 5.6.1. Notably, the most prevalent versions found in FACT were 5.2.0 to 5.2.5, suggesting that the perpetrator's nefarious activities had not impacted the products we manage or monitor.

FACT screenshot with unaffected XZ versions highlighted

Manual Investigation of Anomalies: For files that did not match the known affected versions but were added to our data lake within the relevant release date window, we conducted detailed manual investigations to uncover any discrepancies or hidden versions that might have escaped the initial automated screening. Happily, no anomalies were uncovered.

Leveraging our Internal File Similarity Model: Finally, we used FACT’s File Similarity Model to look up extraneous samples, aiming to uncover misrepresented or stripped metadata and files masquerading as others. This also let us detect modifications that weren’t initially evident. To give more breadth and accuracy to the model, we sourced affected versions of the library from external sources and trusted partners. This deep-dive analysis is pivotal in identifying subtle issues that could indicate more extensive problems within the files.

The Role of FACT and Proactive Behaviour in Future Cyberthreats

As we navigate an ever-evolving software supply chain and its security, having the correct tools to triage quickly and effectively becomes imperative. FACT has proven its value in past incidents, notably during the Log4j vulnerability crisis, where our product enabled customers to identify and mitigate risks within hours rather than weeks. The ability to react quickly to new threats can drastically reduce the window of opportunity for attackers. FACT both aids in immediate threat mitigation and supports ongoing vigilance through continuous monitoring and updates. This proactive approach is essential for maintaining the integrity of systems and protecting critical infrastructure.

How FACT can play a critical role in triaging future supply chain incidents

Advanced Decomposition of Binaries: FACT uses sophisticated methods to effectively find and extract nested binaries, creating a robust foundation for granular analysis that helps pinpoint vulnerabilities and increase the efficiency of triage processes.

Enhanced Metadata Extrapolation: By extracting, correlating, and normalizing metadata across the platform, FACT provides deeper insights into known security risks, facilitating a more informed and proactive security posture.

Extensive YARA Scanning: FACT employs the best-in-class YARA ruleset from our partners at Nextron Systems, which is run against every extracted binary within our cloud environment. This ensures that the most effective and up-to-date detection logic is applied to identify known threats as quickly as possible.

Holistic Component Analysis: FACT offers a comprehensive view of how individual components are integrated across various product libraries within your organization. This holistic approach is crucial for understanding the full scope of a potential threat and for ensuring that all vectors of attack are addressed.

Lessons Learned

This backdoor incident is a potent reminder of the persistent and evolving nature of cyber threats. Targeting the open source community is hardly novel and will no doubt be repeated, given the maintainers of these libraries are underpaid, overworked, and essentially volunteering their time. Open-source software is simply more susceptible to these types of attacks, underpinning the need for more support and resources to maintain both security best practices and the mental health of developers. Without tools like FACT, product managers and security analysts could spend an enormous amount of time determining if their systems include a vulnerable version of the latest affected library. This challenge makes supply chain attacks on corporate entities both silent and effective, as tracking such vulnerabilities within an ecosystem can be exceptionally intricate without the right tools.

Fortunately, in the case of the XZ backdoor event, the malicious code was introduced into very recent versions of the library so it hadn’t yet had the chance to be widely adopted (as evidenced by the fact that none of our customers were using those versions). Nevertheless, we were able to quickly reassure our customers that they were at no risk from this threat and to spare them the costly task of wading through false positives to find it. In turn, they could reassure their customers that their software was free of the vulnerability.

This event also highlights the critical need for rapid response and ongoing vigilance in cybersecurity. Tools like FACT — as part of a wide range of security capabilities — are essential for managing and mitigating threats efficiently and helping to protect critical digital infrastructure.

Logic-based attacks, such as those hidden in code from what was believed to be a trusted maintainer, are extremely difficult to detect. But by having all of your deployed software scanned by FACT and knowing where it all is (including new updates), you will be able to quickly track down affected products in your environment and begin assigning remediation efforts — just like after the Log4j discovery or whatever new vulnerability is discovered tomorrow.

Marcello Delcaro

Marcello is a software analyst and renaissance man with an extremely wide set of skills. (If we need something done, we give it to Marcello — he’ll figure it out.) Marcello manages all of our customer pilot projects and implementation projects.

Previous Post View All