This whole thing has been consuming me over the whole weekend. The mechanisms are interesting and a collection of great obfuscations, the social engineering is a story that’s shamefully all too familiar for open source maintainers.
I find most interesting how they chose their attack vector of using BAD test data, it makes the rest of the steps incredibly easier when you have a good archive, manipulate it in a structured method (this should show on a graph of the binary pattern btw for future reference) then use it for a fuzzy bad data test. It’s great.
The rest of the techniques are banal enough except the most brilliant move seems to be that they could add “patches” or even whole new back doors using the same pattern on a different test file. Without being noticed.
Really really interesting, GitHub shouldn’t have hidden and removed the repo though. It’s not helpful at all to work through this whole drama.
Edit: I don’t mean to say this is banal in any way, but once the payload was decided and achieved through a super clever idea, the rest was just great obfuscation.
It’s got me suspicious of build-time dependency we have in an open source tool, where the dependency goes out of its way to prefer xz and we even discovered that it installs xz on the host machine if it isn’t already installed — as a convenience. Kinda weird because it didn’t do that for any other dependencies.
These long-games are kinda scary and until whatever “evil” is actually done you have no idea what is actually malicious or just weird.
> It’s got me suspicious of build-time dependency we have in an open source tool, where the dependency goes out of its way to prefer xz and we even discovered that it installs xz on the host machine if it isn’t already installed — as a convenience. Kinda weird because it didn’t do that for any other dependencies.
Have you considered reaching out to the maintainers of that project and (politely) asking them to explain? In lieu of recent events I don't think anyone would blame you, in fact you might even suggest they explain such an oddly specific side effect in a README or such.
> Have you considered reaching out to the maintainers of that project and (politely) asking them to explain?
That's kind of a catch-22, right? They'd explain with a seemingly good answer if they did it for actual reasons. They'll still explain with a seemingly good answer if they did it for nefarious reasons.
I don't have a good answer to this except to monitor this dependency and its changes.
I'm not sure it's a catch-22, there are many possibilities and subsequent outcomes. I would just want to satiate my curiosity.
You assume they'll have a good reason, but they may not. They could be a malicious actor but also suck at it, and you might spook them into reconsidering. Or they may be fully innocent and just suck at packaging and distributing software and you might help them learn something.
If they do have a good reason, it should stand on its own merits when presented. Anyone could be up to no good and they might be plotting the end of the world, but they also might coincidentally and not relatedly happen to like xz-utils a lot.
Is it possible that it was added by a bad actor, and that most of the individuals in contact will not have signed off on it? I mean, this whole thing started by an interested party digging deeper than anyone else had, you could trigger that for someone else. At the very least, what do you have to lose? If the best they can do is look like they're fine, then every possible side effect is to root up a frighteningly well hidden inflator
They’d definitely have a great answer up their sleeves.
The crazy thing is the devs responsible for the XZ backdoor are among us and will likely be here on HN, downvoting comments which are getting close to the truth and upvoting each other.
A main culprit seems to be the addition of binary files to the repo, to be used as test inputs. Especially if these files are “binary garbage” to prove a test fails. Seems like an obvious place to hide malicious stuff.
It is an obvious place for sure, but it also would have been picked up if the builds where a bit more transperant. That batch build script should have been questioned before approval.
There’s a severe and dangerous lack of paranoia in this dev space.
If I was letting someone maintain my codebase, I would 1 Billion percent be reviewing everything… if there was a binary added, I’d be building it myself and comparing the checksums.
Trust absolutely no-one. If you can give a close friend or a loved one a loan of money and it is so easy for them to never pay you back, it should be a reminder that devs you’ve never met are even more likely to scam you.
I think it's pretty pointless to speculate what you would have done considering the only reason "Jia Tan" was able to gain the access he did was because the original maintainer was burned out / busy with life / otherwise not able to dedicate the time needed for the project. If he can't maintain it himself he will be even less likely to adequately review his replacement. You can claim to have perfect opsec as much as you want but life has a way of screwing that up that no one is invulnerable against.
pretty unhealthy attitude to live by tbh. almost better being burned by a malicious payload once every 10 years than live in perpetual fear of being constantly scammed
I think that depends on the context. If you're maintaining one of the most widely used packages that is directly linked to by libsystemd and is included by pretty much every Linux distro as part of the base system? Then maybe some measure of paranoia is justified.
I think the OpenBSD developers are right to be as paranoid as they are. Anyone who is maintaining a security critical system should be on guard against these kinds of attacks.
> If you're maintaining one of the most widely used packages that is directly linked to by libsystemd and is included by pretty much every Linux distro as part of the base system? Then maybe some measure of paranoia is justified.
But whose problem is that? systemd chose to link against liblzma, not the other way around. I doubt the xz maintainer(s) make money off of the project, and I'm assuming it's a spare-time/side-project type thing. Why should the fact that the library is included in every distro and is a dependency of systemd affect the xz maintainers' obligations? The leader of systemd has been variously employed by RedHat and Microsoft... if they're choosing to pull in an external dependency for systemd and then making money off of selling their Linux distros/cloud services, it would seem they're the ones that could afford to take on the burden of reviewing everything with a fine-toothed comb, not the xz maintaners.
Thing is, sec should be taken seriously across the board.
I love what OpenBSD devs did - they seen an entire community of naive coders who didn’t give much of a crap about security and started something.
I have used OpenBSD for 10 years and thoroughly recommend it to anyone to give you a good slap around the face for how shit is done right.
I sort of agree, I think we should collectively raise the level of paranoia slight, by some tens of percentages, to remove a lot of negative outcomes - but I wouldn't expect off-hand hobby devs to even remotely apply the same level of risk management as OpenBSD does.
I find most interesting how they chose their attack vector of using BAD test data, it makes the rest of the steps incredibly easier when you have a good archive, manipulate it in a structured method (this should show on a graph of the binary pattern btw for future reference) then use it for a fuzzy bad data test. It’s great.
The rest of the techniques are banal enough except the most brilliant move seems to be that they could add “patches” or even whole new back doors using the same pattern on a different test file. Without being noticed.
Really really interesting, GitHub shouldn’t have hidden and removed the repo though. It’s not helpful at all to work through this whole drama.
Edit: I don’t mean to say this is banal in any way, but once the payload was decided and achieved through a super clever idea, the rest was just great obfuscation.