"We got lucky": What the XZ Utils backdoor says about the strength and insecurities of open source

Linux has just dodged a serious security threat in the form of a mysterious backdoor added to a key library found in many distributions.  

The backdoor was found in the XZ Utils library and could have allowed an attacker to compromise SSHD authentication, granting unauthorized access to the entire system remotely. Fortunately, however, it was spotted before it had been widely incorporated.

XZ is a data compression format present in nearly every Linux distribution which helps compress and then decompress large file formats for sharing via file transfers.

“With a library this widely used, the severity of this vulnerability poses a threat to the entire Linux ecosystem,” the Kali Linux team explained in an advisory. “Luckily, this issue was caught quickly so the impact was significantly less than it could have been”.

The backdoor is quite complex, according to analysis from Akamai. Instead of pushing parts of the backdoor to the public git repository, these were only included in source code tarball releases.

“This caused parts of the backdoor to remain relatively hidden, while still being used during the build process of dependent projects,” it said.

It’s not clear who added the backdoor into the library or why, but it seems to have been a very sophisticated attempt to introduce malicious code. It appears that a developer joined the project and contributed to it for two years and took on more responsibility before their account was used to introduce the rogue code.

In this case, the backdoor was found relatively quickly after another developer spotted some odd behavior around liblzma (part of the xz package) and decided to do a bit of digging.

After posting his findings online, the Linux distributions affected by the backdoor also issued warnings.

OpenSUSE said the rolling release distribution openSUSE Tumbleweed and openSUSE MicroOS included the affected version of the XZ / liblzma library between March 7 and March 28.

RedHat also revealed its Fedora Linux 40 beta had contained two affected versions of xz libraries, and the vulnerability affected Kali Linux between March 26 to March 29.

Meanwhile, Debian said compromised packages were part of the Debian testing, unstable and experimental distributions, with versions ranging from 5.5.1alpha-0.1 (uploaded on 2024-02-01), up to and including 5.6.1-1.

The package has since been reverted to use the upstream 5.4.5 code.

CISA recommended that developers and users downgrade XZ Utils to an uncompromised version - such as XZ Utils 5.4.6 Stable - and hunt for any malicious activity.

“This backdoor almost became one of the most significant intrusion enablers ever,” said Akamai, because if widely implemented the flaw would have given attackers access to any Linux machine running an infected distro.

“This obviously raises a lot of concerns. We got lucky,” Akamai added. “If this backdoor was not detected by a curious engineer, how long would it have remained active? And perhaps even more concerning: What if this has happened before?”

XZ Utils incident highlights the pros and cons of open source

It's an incident which reflects the complicated nature of open source development - and how the actions of a small group of developers can have an outsized impact across the software supply chain. 

The Open Source Security Foundation said situations like this “remind us all that we need to remain vigilant within the open source software ecosystem”

“Open source is about well-intentioned humans donating their time and talents to help solve problems, and sadly this can be compromised,” the foundation added.

However, the open nature of open source stopped the bad code from getting very far, the foundation noted.

“The nature of open source software allowed this vulnerability to be discovered, reported, and addressed in a short period of time due to the diligence and oversight of the community.

Beyond that, the way that open source packages cycle from “experimental” to “stable” releases meant that the compromised packages were contained to a narrow distribution.”

As many will recall from XKCD, all modern digital infrastructure is underpinned by the maintainers of obscure software projects that few people know about, but on which everyone relies.

This latest incident, which many are warning could have turned out far, far worse, is just the latest reminder of that.