I am fascinated by a story of how a Microsoft engineer discovered a major, heavily disguised, backdoor security breach that was years in the making, and nearly implemented. Pixabay   BackgroundHidden in a widely use compression utility was a software backdoor that would allow someone remote access to entire systems.This was a multi-year endeavor by user named Jia Tan, @JiaT75 who gained trust over many years. His account is now suspended everywhere.HackerNews has this interesting snip.

Microsoft security researcher Andres Freund has been credited with discovering and reporting the issue on Friday.

The heavily obfuscated malicious code is said to have been introduced over a series of four commits to the Tukaani Project on GitHub by a user named JiaT75.

 The Long GameThese opensource projects are volunteer work. They pay nothing.The person normally responsible for the code, Lasse Collin (Larhzu), maintained the utility since 2009 but was suffering burnout.Jia Tan started contributing in the last 2-2.5 years and gained commit access, and then release manager rights, about 1.5 years ago. Backdoor Uncovered in Years-Long Hacking PlotMuch of this story is extremely geekish and difficult to understand. An article on Unicorn Riot is generally readable.Please consider Backdoor Uncovered in Years-Long Hacking Plot

A fascinating but ominous software story dropped on Friday: a widely used file compression software package called “xz utils” has a cleverly embedded system for backdooring shell login connections, and it’s unclear how far this dangerous package got into countless internet-enabled devices. It appears the persona that injected this played a long game, gaining the confidence of the legitimate main developer, and thus empowered to release new versions themselves.

Andreas Freund reported this Friday morning on an industry security mailing list, leading many experts to spend the day poking under rocks and peering into the abyss of modern digital insecurity: “The upstream xz repository and the xz tarballs have been backdoored,” Freund wrote. It cleverly pokes a hole in the SSH daemon (sshd), which is essential to modern-day computing at the most fundamental level.

The risks if this hadn’t been discovered were extreme: as noted expert @thegrugq put it: “The end game would be the ability to login to every Fedora, Debian and Ubuntu box on the internet. If it isn’t a state actor it should be…” 

Cryptographer Filippo Valsorda said, “This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library.”

The problem was uncovered after Freund noticed that the new version slowed down their PostgreSQL database tests, and they started debugging why this happened. It turns out the backdoor causes a tiny but noticeable slowdown in performance, a big win for picky benchmarking types everywhere.

As Minneapolis security expert Ian Coldwater noted, “Open source maintainer burnout is a clear and present security danger. What are we doing about that?”

This June 2022 message from the original developer confessing to burnout illustrates how Jia Tan gained control over the software:

“I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see.

It’s also good to keep in mind that this is an unpaid hobby project.

Anyway, I assure you that I know far too well about the problem that not much progress has been made. The thought of finding new maintainers has existed for a long time too as the current situation is obviously bad and sad for the project.

A new XZ Utils stable branch should get released this year with threaded decoder etc. and a few alpha/beta releases before that. Perhaps the moment after the 5.4.0 release would be a convenient moment to make changes in the list of project maintainer(s). Forks are obviously another possibility and I cannot control that. […]”Lasse Collin, xz-devel mailing list, June 8, 2022

Some observers suspect the personas badgering Collin by email may have also been sockpuppets trying to shake control away from him. In a detailed report ars technica warned that even older versions could have security problems since the bad actor made many binary test file changes over the years.

 Backdoor Story Unfolding Now Upstream Backdoor“Very annoying – the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it’s “great new features. We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.”“He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.” USA Security AlertThe US Cybersecurity & Infrastructure Security Agency (CISA) issued an alert on a Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094

CISA and the open source community are responding to reports of malicious code being embedded in XZ Utils versions 5.6.0 and 5.6.1. This activity was assigned CVE-2024-3094. XZ Utils is data compression software and may be present in Linux distributions. The malicious code may allow unauthorized access to affected systems.

 Industry-Wide Reckoning Needed“I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever,” commented Mastadon user @glyph.Here is an Interesting Timeline on how this was nearly implemented. Here’s the HeroWow, just wow.We were perhaps days away from this code being implemented.More By This Author:How Much Do Food Stamps, Social Security, And Medicare Support The Economy? Spending, Income, And Inflation Data Do Not Support Fed Interest Rate CutszCase-Shiller National Home Price Index Hits New Record High