The Internet Was Weeks Away From Disaster and No One Knew

A hacker discovered a fatal weakness in the world's most important operating system that could have compromised millions of internet servers.

• Millions of machines would have been accessible to whoever crafted the backdoor • Could enable spying, ransom demands, or taking down entire countries • Weeks away from catastrophic internet-wide compromise

The entire operating system rested on a single part maintained by a single person, and compromising that one part could infect almost any server on the internet.

The story began with a jammed printer at an MIT AI lab and escalated into a fundamental structural weakness in the open source ecosystem.

A Xerox 9700 laser printer at MIT's AI lab kept jamming, preventing researchers from collecting their printouts after waiting hours.

Richard Stallman wanted to write code to alert when the printer jammed, but Xerox refused to provide the source code, and the original developer declined to share it.

• Stallman quit MIT and established the Free Software Foundation in 1985 • Promoted four basic freedoms: run software for any purpose, study it, change it, and share it • Created the General Public License to protect these freedoms

Stallman started the GNU Project to build a Unix-compatible system from scratch, creating GCC compiler, Bash shell, and core utilities over seven years.

The GNU Project had utilities, shell, and tools, but was always missing the kernel—the core component that talks to hardware and manages memory.

In fall 1991, a young computer science student at the University of Helsinki was building his own kernel. After hearing Stallman speak, he adopted the General Public License.

The student initially wanted to call his kernel Free Unix or Freax, but his friend thought it sounded terrible, so he renamed it after himself: Linus, creating Linux.

• Linux kernel combined with GNU Project components created a full operating system • Code was open and free, allowing anyone to inspect, improve, and fix flaws • New model of software development took hold based on collaborative innovation

• Over 30 million Linux users estimated in major markets • Android with over 3 billion devices built on Linux kernel • Majority of internet servers in the world run Linux

Electronic vacuums, cameras, most TVs, and most consumer electronics run Linux, along with high-security military applications.

• Every single one of the top 500 supercomputers in the world runs Linux • Used in the Pentagon and on US nuclear submarines • Banks, manufacturers, hospitals, governments, and defense organizations rely on Linux servers

Linux's open nature allows developers to tweak it for their specific use cases, enabling it to cover all modern computing applications.

With enough eyeballs, all bugs are shallow—the belief that open source code scrutiny ensures security.

The open source movement relies on thousands of small tools and libraries, each doing different jobs like networking, security, or compression.

Many projects start when one person wants to fix a specific problem, often unpaid, coding on nights and weekends. If useful, millions of machines end up relying on one person's passion project.

The entire ecosystem can quietly rest on a project maintained by a single volunteer, with the open source community unaware of the vulnerability.

Lasse Collin developed XZ, a data compression tool since 2005 that became so effective it is now used in almost every major Linux distribution.

For 20 years, almost all work keeping XZ compatible with evolving hardware fell on Lasse, who was never paid but remained committed to the project.

• Community members pressured Lasse about unmerged patches and slow development • Lasse disclosed long-term mental health issues affecting his ability to maintain the project • Contributors complained about choking the repository and demanded a new maintainer

Jia Tan offered to step up as co-maintainer, taking some load off Lasse and eventually becoming the primary maintainer after years of helpful contributions.

In 1995, a hacker captured thousands of usernames and passwords sent over the Helsinki University of Technology campus network in plain text.

Tatu Ylonen created Secure Shell (SSH) combining two security steps: securing the channel through shared secret codes and authenticating users with RSA encryption.

• Uses mathematical key exchange (Diffie-Hellman) to establish secure connections without meeting in person • RSA encryption ensures that only the intended recipient can read messages • Authenticates that the remote machine is really who it claims to be

SSH became the maintenance backbone of the entire internet, adopted on almost every machine running Linux for secure remote control and access.

The most widely used open source SSH implementation is OpenSSH, one of the most closely examined projects because it is vitally important to server security worldwide.

Having a way to bypass authentication in secure shell is like having the master key to a hotel—it lets attackers into every room.

• OpenSSH is heavily scrutinized, but not all of its dependencies receive the same attention • Each program is stitched together from other programs called dependencies • Compromising a dependency can sneak an exploit into the main project

Lasse Collin's compression tool XZ is linked through a chain of dependencies to OpenSSH, making it an attractive target for attackers.

XZ's goal was to find a better way to compress data on Linux where compressed and decompressed data must return exactly the same.

• Assigns shorter codes to frequently appearing symbols and longer codes to rare ones • More common symbols naturally appear at top of tree, getting shorter codes • Weakness: doesn't recognize when entire chunks repeat in text

Instead of encoding individual symbols, keeps rolling dictionary of previously seen chunks and uses codes pointing back to matches rather than repeating the chunk.

• LZMA developed by Igor Pavlov around 1998 combines Lempel-Ziv algorithm with Markov chains to predict probable next chunks • Often shrinks files to about 70% of typical zip size • Lasse called it XZ because it sounded cool, released in 2009

Jia Tan emailed Rich Harris with new features for XZ in February 2024, immediately impressing him as a good coder, responsive, and helpful.

• Red Hat ships Fedora (free) and Red Hat Enterprise Linux (paid subscription) • RHEL has new major releases roughly every three years • RHEL 10 deadline was around March-April 2024, forcing Jia to act quickly

Jia started by making small changes to the XZ codebase like changing the bug report contact to his email and tweaking small tools.

Rich let his guard down as Jia proved himself, while Jia became increasingly insistent about getting updated XZ into Fedora releases.

The XZ code lives on GitHub tracked by Git, but Jia couldn't sneak in the payload as normal source code without it being obvious.

• Compression software contains binary blobs used to test compression and decompression functionality • Nobody reads these test blobs as they appear to be garbage data • Perfect place to hide a payload inside something that looks harmless

Jia slipped a small easy-to-miss change into the build code that hides among automatically generated code and quietly unpacks his payload into the XZ library.

Once the payload is inside XZ, it must pick the right time to act by targeting the SSH RSA authentication step.

Jia's goal is to compromise RSA authentication step in SSH connection process by slipping a malicious component that intercepts every key check.

• Modern applications use shared libraries with Global Offset Table (GOT) containing addresses to functions • RSA Decrypt comes from a shared crypto library, not from OpenSSH directly • Jia can overwrite the GOT entry telling SSH where RSA Decrypt is located

IFUNC is normally used for hardware optimization, but Jia uses it to run code early in program startup and locate the dynamic audit hook.

• Must slip payload after RSA Decrypt entry is filled legitimately but before GOT is marked Read Only • Dynamic audit hook fires when linker writes real RSA Decrypt address • This tiny window is the Goldilocks zone where hijacking becomes possible

The backdoor listens for a hidden master key during SSH connections, which is not a simple password but a mini cryptographic exchange.

• Backdoor code checks for a shared secret • Authenticates the user attempting access • Only runs payload if both checks pass

The backdoor runs a miniature version of SSH's encryption inside SSH itself, using encryption to ensure only attackers can get in instead of keeping them out.

• Wipes evidence of the backdoor ever firing to avoid SSH logging detection • Numerous safety checks ensure system compatibility without crashes • Designed to slip through only where it will run invisibly

Jia becomes frantic pushing to get compromised XZ into major Linux releases before anyone discovers it, getting it into Debian experimental and requesting Ubuntu inclusion.

After weeks of increasingly insistent messages, Rich adds updated XZ to pre-release Fedora, giving Jia access to pre-release versions of Fedora, Debian testing, and Ubuntu.

The backdoor has a memory leak bug—it grabs memory without releasing it, causing programs to grow over time and eventually crash.

• Developers use Valgrind tool to monitor memory operations and catch leaks • Valgrind detects invalid writes from Jia's code • Bug raises attention but Rich hasn't yet realized it's a backdoor

Jia claims that random data used to generate test files isn't reproducible, so he's replacing it with better random data.

In the updated code, Jia fixes the memory leak bug, making the backdoor work correctly without crashes.

The cover story is very plausible because it's a reasonable explanation for updating test blobs in compression software.

• Jia adds comments and modifications to IFUNC code around the fix • Changes look like he's improving how IFUNC works to fix Valgrind bug • Actually only hides the real fix without changing the malicious code functionality

Andres Freund, a Microsoft programmer working on Postgres, notices a slowdown of 400-500 milliseconds in connection times when testing unstable Debian in March 2024.

• Andres remembers seeing XZ problems with Valgrind weeks earlier • Traces delay back to OpenSSH update caused by XZ dependency • Notices binary test files were never actually used in tests

Andres realizes this isn't a bug but an intentional backdoor that hunts for audit hooks, decodes binary instructions, wraps everything in encryption, and covers its tracks with string obfuscation.

• Since XZ's security contact is Jia Tan, Andres emails Debian security team directly • Posts detailed report to public security mailing list • Triggers immediate Red Hat response and community investigation

Red Hat quickly rolled back Fedora versions and told all users to revert to uncompromised versions when the backdoor was discovered.

• Without Andres investigating, the slowdown would have been missed • His work for days hunting the issue saved millions of systems from compromise • Received kudos from Microsoft CEO and security community

Surprisingly muted mainstream news coverage despite the potential to compromise millions of systems across governments, banks, hospitals, and defense organizations.

• Millions of internet servers would have been compromised • Could enable spying, ransoms, or taking down entire countries • Would have affected RHEL 10 and most important computers in the world

The identity of Jia Tan remains unknown, though experts believe one person communicated with the project but a group of people worked for two and a half years.

• Aliases sound like Asian names and timestamps fall in UTC+8 Beijing time • But signs point this might be deliberate misdirection given meticulous operation • Other evidence suggests possible APT29 Russian state-backed group involvement

• Criminal organizations lack patience for two and a half years without financial return • State actors have motivation to spend millions on long-term infrastructure • Sock puppet accounts applied pressure as part of multi-stage social engineering

• Nine changes fall outside Beijing time into UTC+2 timezone • UTC+2 includes Israel and parts of Western Russia • Worked on Chinese New Year but not Christmas, adding to attribution confusion

Lasse gave the world a beautiful gift with XZ, but humanity responded by poisoning it and blaming him for not maintaining free software forever.

• Lasse maintained XZ for 20 years without payment • Community pressured him despite his mental health struggles • Implicitly blamed for not protecting the ecosystem single-handedly

The system has not supported maintainers of critical infrastructure projects, creating vulnerability when maintainers burn out or face pressure.

On Saturday evening, Lasse worked together on a RHEL 9 XZ bug workaround despite having every reason to refuse, demonstrating his brilliant commitment.

Community audit after XZ found almost nothing, suggesting many state-sponsored backdoors remain undiscovered in the ecosystem.

• Governments and militaries prepare for cyber escalation and geopolitical conflict • Private contractors working for states also incentivized to place backdoors • Too many incentivized actors for so few discovered backdoors

Some experts argue XZ reveals a fundamental flaw in open source, while others argue closed source is no better and may be worse.

• Attackers are getting more sophisticated and making fewer mistakes • Gloves are off in cyber warfare preparation • Linux community may not be fully ready for this level of threat

What took multiple years of social engineering and meticulous code in open source would be easier in closed source where a court order or company decision could hide backdoors.

• Only because XZ is open source was it picked apart, analyzed, and turned into security conversation • Closed source hackers might use court orders or brush breaches under the rug • Open source requires constant oversight instead of trust

The fundamental vulnerability isn't the code itself, it's the people who maintain it and the systems that don't support them enough.

• XZ backdoor demonstrates need for supporting open source maintainers • Security depends on preventing maintainer burnout and pressure • Community must invest in infrastructure maintainers to prevent future compromises