See https://www.openwall.com/lists/oss-security/2024/03/29/4. "" [...] == Compromised Release Tarball == One portion of the backdoor is *solely in the distributed tarballs*. For easier reference, here's a link to debian's import of the tarball, but it is also present in the tarballs for 5.6.0 and 5.6.1: https://salsa.debian.org/debian/xz-utils/-/blob/debian/unstable/m4/build-to-host.m4?ref_type=heads#L63 That line is *not* in the upstream source of build-to-host, nor is build-to-host used by xz in git. However, it is present in the tarballs released upstream, except for the "source code" links, which I think github generates directly from the repository contents: https://github.com/tukaani-project/xz/releases/tag/v5.6.0 https://github.com/tukaani-project/xz/releases/tag/v5.6.1 This injects an obfuscated script to be executed at the end of configure. This script is fairly obfuscated and data from "test" .xz files in the repository. This script is executed and, if some preconditions match, modifies $builddir/src/liblzma/Makefile to contain [...] """
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e12c7ce6dab9f016b3efdd0a774793865c486b8c commit e12c7ce6dab9f016b3efdd0a774793865c486b8c Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-29 16:14:28 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-29 16:14:53 +0000 profiles: add references to xz-utils mask See https://www.openwall.com/lists/oss-security/2024/03/29/4. Bug: https://bugs.gentoo.org/928134 Signed-off-by: Sam James <sam@gentoo.org> profiles/package.mask | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
We have it masked since yesterday, after Debian, suse, fedora, et. al all did the same: commit 6424767d42b1853c6e24b1cf0734d5a51d0e2e4d Author: Sam James <sam@gentoo.org> Date: Thu Mar 28 22:00:41 2024 +0000 profiles: mask >=app-arch/xz-utils-5.6.0 A serious bug is being investigated. Please downgrade ASAP until we have a fix. Signed-off-by: Sam James <sam@gentoo.org>
As a summary so far: * We had the (nominally) vulnerable versions in Gentoo. They were masked last night. * The posted injection script appears to not fire on Gentoo as it looks for .deb and .rpm specific files/environment variables in the build environment, but it's possible (I haven't yet verified) tht other versions had a different set of criteria. * In Gentoo, we don't patch net-misc/openssh with systemd-notify support which means liblzma, at least in the normal case, doesn't get loaded into the sshd process. * The backdoor which is injected by the script during the build process MAY OR MAY NOT only work on sshd, it's unclear (people report it checks argv[0] but no in-depth analysis has been done yet or at least shared publicly; it's possible it activates on other binaries).
After migrating to profile 23.0 last week, when updating @world, my terminal lit up with lzma flag in green on a like a dozen packages. I don't want to point fingers but I think it's worth checking why that was done and who did it, because my system worked without lzma in things like kmod for years. And this is especially bitter because I read that profile 23.0 is supposed to be "enhanced in security". I don't see how linking liblzma into everything enhances security.
(In reply to mayoi from comment #4) > After migrating to profile 23.0 last week, when updating @world, my terminal > lit up with lzma flag in green on a like a dozen packages. I don't want to > point fingers but I think it's worth checking why that was done and who did > it, because my system worked without lzma in things like kmod for years. > That's completely unrelated. We just changed defaults because lots of stuff uses lzma and zstd support. It was done a while ago after discussion on IRC: commit 99a7cb9e0b1728ca75242ddfee6357dc008bd1cd Author: Andreas K. Hüttel <dilfridge@gentoo.org> Date: Sun Nov 13 19:26:40 2022 +0100 profiles: Set USE="xz zstd" in 23.0 Signed-off-by: Andreas K. Hüttel <dilfridge@gentoo.org> I have ABSOLUTELY NO REASON to believe this was malicious at all, and I think it's unhelpful to go down that route. > And this is especially bitter because I read that profile 23.0 is supposed > to be "enhanced in security". I don't see how linking liblzma into > everything enhances security. Well, if we really want to get into that, we can talk about how BIND_NOW sort of made things interesting for the exploit in question. But let's try to keep this bug focused on facts please. There's already a lot of confusino out there.
(That said, I totally appreciate your point, I agree it's very unfortunate timing, I just don't think it was malicious at all, and I didn't oppose the change.)
Since the malicious party appears to be the upstream developer with the signing key, are we assuming any changes made by them to the (unmasked) previous releases will conflict with the manifest? Assuming those hypothetical malicious changes were not already present when the manifest was generated, of course.
(In reply to revamps.warbler-0w from comment #7) > Since the malicious party appears to be the upstream developer with the > signing key, are we assuming any changes made by them to the (unmasked) > previous releases will conflict with the manifest? Yes, you can safely assume this. I've also seen no evidence of tarballs being modified post-upload. I am still considering whether we should bring back the last release signed by Lasse Collin, rather than Jia Tan, to be safe. > Assuming those > hypothetical malicious changes were not already present when the manifest > was generated, of course. Yeah, this is the big challenge.
A note about the whiteboard status: we currently believe, as per sam's summary, that the likelyhood of an impact on Gentoo is small, and that would lead to something like an A4 severity level. However, with the backdoor not fully analyzed yet we want to take into account that the impact may be larger, and thus opt for a A0 severity, assuming a (currently unclear) worst case scenario where the backdoor could still be included and may act with root privileges. This severity also matches with the overall news about this vulnerability.
Since the exact scope of this is not 100% clear yet, two points we might want to consider: 1) The 'modules-compress' flag on gentoo-kernel(-bin) and external kernel modules uses xz-utils for compression. Would it be a good idea for users to recompile gentoo-kernel(-bin) after downgrading xz-utils if they have this flag enabled? Just to be on the safe side, or is this overkill? 2) (stable masked) generic-initramfs/generic-uki on gentoo-kernel(-bin) includes files from xz-utils. Do we want to rebuild these with an older version of xz-utils?
(In reply to Sam James from comment #8) > (In reply to revamps.warbler-0w from comment #7) > > > I am still considering whether we should bring back the last release signed > by Lasse Collin, rather than Jia Tan, to be safe. > > > Assuming those > > hypothetical malicious changes were not already present when the manifest > > was generated, of course. > > Yeah, this is the big challenge. This seems to me to be the only responsible option, pending a thorough review, considering what I’ve seen about them pushing distros to update to the compromised version. If you don’t trust them you can’t trust what they’ve signed, can you?
fyi, stage3-amd64-hardened-systemd-20240324T164906Z.tar.xz comes with the affected xz-utils version
(In reply to revamps.warbler-0w from comment #11) > This seems to me to be the only responsible option, pending a thorough > review, considering what I’ve seen about them pushing distros to update to > the compromised version. If you don’t trust them you can’t trust what > they’ve signed, can you? Yes, there's just issues with whether any packages use versioned symbols from >=5.4. I have done initial checks of 5.4..5.6 but not enough. 5.2..5.4 is harder. But yes, if we can't trust them, we can't trust them..
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/data/glsa.git/commit/?id=ad7cf37eb216318a2076f79b7aceee6389bc887b commit ad7cf37eb216318a2076f79b7aceee6389bc887b Author: GLSAMaker <glsamaker@gentoo.org> AuthorDate: 2024-03-29 21:48:56 +0000 Commit: Hans de Graaff <graaff@gentoo.org> CommitDate: 2024-03-29 21:53:10 +0000 [ GLSA 202403-04 ] XZ utils: Backdoor in release tarballs Bug: https://bugs.gentoo.org/928134 Signed-off-by: GLSAMaker <glsamaker@gentoo.org> Signed-off-by: Hans de Graaff <graaff@gentoo.org> glsa-202403-04.xml | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
+1 to going back to a release with a signature that we can trust more, even if not by much. At least provided it's not too painful to do so. Additionally it might be prudent to consider LD_BIND_NOT=1 in the default environment, since from my understanding it only affects performance but not compatibility (other than with the backdoor). Especially on systems that have not yet been downgraded and (soft-)rebooted. Additionally do we *really* need +extra-filters IUSE on xz-utils? Those BCJ coders in particular have been making me uncomfortable me for years ever since I learned about them. Finally, it's probably a good idea to consider patching libarchive with https://github.com/libarchive/libarchive/commit/6110e9c82d8ba830c3440f36b990483ceaaea52c ASAP, since it should be addressing a suspect change by the same persona. I'm sure this APT will keep on giving us more gifts, so happy holidays, I guess. ;)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=56f0bb584949a4b8946dd5e79e0398e73aaf06e0 commit 56f0bb584949a4b8946dd5e79e0398e73aaf06e0 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-29 22:45:41 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-29 22:52:30 +0000 app-arch/xz-utils: add/restore 5.4.2 This is the last release signed by Lasse Collin, the previous signer of xz-utils releases. Downgrade to this out of an abundance of caution. We are not aware of any issues that *specifically* require this. Note that the Manifest matches dfcc1f271fa3da8b8710c80737e85a7347f16ba0 from when 5.4.2 was removed from ::gentoo in the past. Bug: https://bugs.gentoo.org/928134 Signed-off-by: Sam James <sam@gentoo.org> app-arch/xz-utils/Manifest | 2 + app-arch/xz-utils/xz-utils-5.4.2.ebuild | 140 ++++++++++++++++++++++++++++++++ profiles/package.mask | 11 ++- 3 files changed, 152 insertions(+), 1 deletion(-)
Github has disabled the XZ repository entirely as of 1:29 AM UTC
Attempt to downgrade it while doing usual `emerge -uDN world` hangs at: >>> Installing (1 of 6) app-arch/xz-utils-5.4.2::gentoo * checking 191 files for package collisions # tail /var/log/portage/app-arch:xz-utils-5.4.2:20240330-060915.log * Final size of build directory: 14528 KiB (14.1 MiB) * Final size of installed tree: 1424 KiB ( 1.3 MiB) strip: x86_64-pc-linux-gnu-strip --strip-unneeded -N __gentoo_check_ldflags__ -R .comment -R .GCC.command.line -R .note.gnu.gold-version /usr/lib64/liblzma.so.5.4.2 /usr/bin/xzdec /usr/bin/lzmadec /usr/bin/xz /usr/bin/lzmainfo * checking 191 files for package collisions
(In reply to Alex Efros from comment #18) > Attempt to downgrade it while doing usual `emerge -uDN world` hangs at: Sorry, this wasn't related to portage, I suppose it hangs because Keybase hangs and portage probably has tried to get access to /keybase or something like that. Killing keybase-redirector process has solved the issue.
I see that all versions above (and including) 5.4.6-r1 have been masked and this is the correct action to take. However, I see that other distros revert back to 5.4.6, is this version also infected with this backdoor?
(In reply to Vasilis Lourdas from comment #20) > I see that all versions above (and including) 5.4.6-r1 have been masked and > this is the correct action to take. However, I see that other distros revert > back to 5.4.6, is this version also infected with this backdoor? In short: this is unknown. See the explanation in the profiles/package.mask and in the commit message [1]. [1] https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=56f0bb584949a4b8946dd5e79e0398e73aaf06e0
Lasse Collin provided a statement at <https://tukaani.org/xz-backdoor/>.
(In reply to Larry the Git Cow from comment #16) > commit 56f0bb584949a4b8946dd5e79e0398e73aaf06e0 > Author: Sam James <sam@gentoo.org> > AuthorDate: 2024-03-29 22:45:41 +0000 > Commit: Sam James <sam@gentoo.org> > CommitDate: 2024-03-29 22:52:30 +0000 > > app-arch/xz-utils: add/restore 5.4.2 > > This is the last release signed by Lasse Collin, the previous signer of > xz-utils > releases. Hi, Is it really the latest version signed by Lasse Collin? Tag is from Jia Tan: https://git.tukaani.org/?p=xz.git;a=tag;h=073fbd9a994f2e934f3eb3ecf5109041c24f6f24 Latest tag of Lasse Collin is: https://git.tukaani.org/?p=xz.git;a=tag;h=f52502e78bf84f516a739e8d8a1357f27eeea75f Some people try also to downgrade to an early version (without any commit of Jia Tan): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024#45
(In reply to Sébastien P. from comment #23) > (In reply to Larry the Git Cow from comment #16) > > commit 56f0bb584949a4b8946dd5e79e0398e73aaf06e0 > > Author: Sam James <sam@gentoo.org> > > AuthorDate: 2024-03-29 22:45:41 +0000 > > Commit: Sam James <sam@gentoo.org> > > CommitDate: 2024-03-29 22:52:30 +0000 > > > > app-arch/xz-utils: add/restore 5.4.2 > > > > This is the last release signed by Lasse Collin, the previous signer of > > xz-utils > > releases. > > Is it really the latest version signed by Lasse Collin? > Tag is from Jia Tan: > https://git.tukaani.org/?p=xz.git;a=tag; > h=073fbd9a994f2e934f3eb3ecf5109041c24f6f24 > Latest tag of Lasse Collin is: > https://git.tukaani.org/?p=xz.git;a=tag; > h=f52502e78bf84f516a739e8d8a1357f27eeea75f > > Some people try also to downgrade to an early version (without any commit of > Jia Tan): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024#45 According to Debian at <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024#5>: <QUOTE> I count a minimum of 750 commits or contributions to xz by Jia Tan, who backdoored it. This includes all 700 commits made after they merged a pull request in Jan 7 2023, at which point they appear to have already had direct push access, which would have also let them push commits with forged authors. Probably a number of other commits before that point as well. Reverting the backdoored version to a previous version is not sufficient to know that Jia Tan has not hidden other backdoors in it. Version 5.4.5 still contains the majority of those commits. Commits by them such as 18d7facd3802b55c287581405c4d49c98708c136 and ae5c07b22a6b3766b84f409f1b6b5c100469068a show that they were deep into analyzing the security of xz. They were well placed to insert a buffer overflow that could allow eg, a targeted xz file to cause arbitrary code execution. The impact of such a security hole could be much more stealthy and bad than the known backdoor since it would allow chaining xz with other unrelated software releases on an ongoing basis. The package should be reverted to a version before their involvement, which started with commit 6468f7e41a8e9c611e4ba8d34e2175c5dacdbeb4. Or their early commits vetted and revert to a later point, but any arbitrary commit by a known bad and malicious actor almost certainly has less value than the risk that a subtle change go unnoticed. I'd suggest reverting to 5.3.1. Bearing in mind that there were security fixes after that point for ZDI-CAN-16587 that would need to be reapplied. </QUOTE>
(In reply to Sébastien P. from comment #23) The ebuild verifies it using Lasse Collin's key with USE=verify-sig, even.
(In reply to Jeffrey Walton from comment #24) 5.3.1 would not be suitable to downgrade to, as it was a development release. xz-utils does odd/even for dev/stable. It would have to be 5.2.x.
(In reply to Sam James from comment #25) > (In reply to Sébastien P. from comment #23) > > The ebuild verifies it using Lasse Collin's key with USE=verify-sig, even. Ok. About verify-sig, I saw that Jia Tan one is still on portage: https://gitweb.gentoo.org/repo/gentoo.git/tree/sec-keys/openpgp-keys-jiatan It should be removed/masked since it can not be trusted anymore.
Only masked packages depend on this key anyway: # grep -r openpgp-keys-jiatan /usr/portage/**/*.ebuild /usr/portage/app-arch/xz-utils/xz-utils-5.4.6-r1.ebuild: BDEPEND+=" verify-sig? ( sec-keys/openpgp-keys-jiatan )" /usr/portage/app-arch/xz-utils/xz-utils-5.6.1.ebuild: BDEPEND+=" verify-sig? ( sec-keys/openpgp-keys-jiatan )" /usr/portage/app-arch/xz-utils/xz-utils-9999.ebuild: BDEPEND+=" verify-sig? ( sec-keys/openpgp-keys-jiatan )"
Yeah, that's why I haven't been in a hurry to do anything about it there, but I should add it to the mask at least.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c5494a3efcce376157635fb368ab0d9a5b7e7090 commit c5494a3efcce376157635fb368ab0d9a5b7e7090 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-30 19:38:13 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-30 19:41:45 +0000 profiles: mask sec-keys/openpgp-keys-jiatan too (essentially a noop though) Mask sec-keys/openpgp-keys-jiatan too just out of completeness, but this is really a noop given all packages using it were already masked. Two people have independently suggested it to me so may as well do it. Bug: https://bugs.gentoo.org/928134 Signed-off-by: Sam James <sam@gentoo.org> profiles/package.mask | 1 + 1 file changed, 1 insertion(+)
It looks like more analysis has revealed this is a RCE with the payload in the modulus of a public key: "The payload is extracted from the N value (the public key) passed to RSA_public_decrypt, checked against a simple fingerprint, and decrypted with a fixed ChaCha20 key before the Ed448 signature verification..." Also see <https://www.openwall.com/lists/oss-security/2024/03/30/36>.
Is there information currently on the scope of the backdoor? We know it attacked sshd as that was how it was discovered, but would this have any impact on something like zstd (another [de]compression program) which can be complied with USE +lzma? The linux kernel itself with CONFIG_XZ_DEC?
(In reply to Spencer from comment #32) > Is there information currently on the scope of the backdoor? We know it > attacked sshd as that was how it was discovered, but would this have any > impact on something like zstd (another [de]compression program) which can be > complied with USE +lzma? The linux kernel itself with CONFIG_XZ_DEC? Someone decrypted the strings in the payload (https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01) only mentions sshd. I would assume we would see more stuff like that if it was affecting others.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=29d5f89b75c1ab37b3346ac41840cf8fa4b3d3a8 commit 29d5f89b75c1ab37b3346ac41840cf8fa4b3d3a8 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-30 21:47:35 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-30 21:48:39 +0000 app-arch/xz-utils: drop 5.6.1 (noop really) On IRC, it was pointed out that non-Gentoo users might get confused if they see Gentoo has 5.6.1 in its repository even though it is masked. Let's just drop it to avoid confusion. It's better to do that than risk someone being confused - there's enough confusion out there anyway. In a minute, I'll probably adjust the live ebuild to delete the known bad test data too. Bug: https://bugs.gentoo.org/928134 Signed-off-by: Sam James <sam@gentoo.org> app-arch/xz-utils/Manifest | 2 - app-arch/xz-utils/xz-utils-5.6.1.ebuild | 141 -------------------------------- 2 files changed, 143 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=efe06b01128ce829eb4e4c6a5ed5f52a342ecbc0 commit efe06b01128ce829eb4e4c6a5ed5f52a342ecbc0 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-30 21:51:33 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-30 21:51:52 +0000 app-arch/xz-utils: delete known-bad test files in 9999 As I promised to do in 29d5f89b75c1ab37b3346ac41840cf8fa4b3d3a8 a moment ago. I really doubt many people are rocking 9999 and there's nothing known-bad in the git repo *that can use these* but the test files are dangerous anyway so let's just do it for completeness. Bug: https://bugs.gentoo.org/928134 Signed-off-by: Sam James <sam@gentoo.org> app-arch/xz-utils/xz-utils-9999.ebuild | 3 +++ 1 file changed, 3 insertions(+)
(In reply to Larry the Git Cow from comment #34) > On IRC, it was pointed out that non-Gentoo users might get confused if > they > see Gentoo has 5.6.1 in its repository even though it is masked. I got a similar confusion with Debian: https://packages.debian.org/sid/xz-utils => 5.6.1+really5.4.5-1 First, I saw 5.6.1 and did not pay attention to the next part… I also saw that Alpine stay with 5.6.1 and just switch from release to git archives: https://git.alpinelinux.org/aports/commit/main/xz/APKBUILD?id=982d2c6bcbbb579e85bb27c40be84072ca0b1fd9 Like ou say: “there's enough confusion out there anyway” :). In that case, it is important to look at security alert of your distro. Because each one can handle it in different ways.
I'm maintaining https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 as a FAQ.
(In reply to Spencer from comment #32) > Is there information currently on the scope of the backdoor? It is a RCE (remote code execution) backdoor, where sshd is used as the first step: It listens for connections, and when so patched, invokes the malignant liblzma, which in turn executes a stage 2 that finally executes the payload which is provided to sshd in a part of the encrypted public key given to it as the credential (wich doesn't need to be authentic to be harmful). > would this have any impact on something like zstd (another > [de]compression program) which can be complied with USE +lzma? > The linux kernel itself with CONFIG_XZ_DEC? sshd and indeed liblzma was apparently chosen deliberately for its convenience as a remove code execution attack vector, particularly as RedHat and others were friendly enough to patch them together (something to look into, eventually -- was this prepared a long time ago? When and by whom were such patches pushed?), but -- as far as we know yet -- there is no suggestion that authentication bypass was attempted or even intended, nor that archives other than the encrypted public keys were supposed to be used. If you can run a payload by simply sending a crafted public key, there is no need for authentication bypass... Where direct ssh access from the internet is not required, users should restrict sshd to accept connections only from the local network (and connect to it from the internet using wireshark, if needed).
(In reply to Alexander Wessel from comment #38) > wireshark WireGUARD, of course. I do that all the time. :/
I'll correct the above statement to __always__ VPN into any remote access daemon which has root and probably even just full local user rights. Exposing such things directly to the internet is asking for the next 0-day to hit you even if it's a once or twice per century kind of bug. WireGuard is quite simple to set up and costs nothing, so just use it. And best not to use SSH as the poor man's VPN here. ;)
BTW, its not all hardware. its only x86_64. The way it works is, the build system is broken is some archives. The backdoor is not in the code. Its in some object files left there. That get included in the final library, but are not built on said system. The are artifacts that remain in the archive. And they are x86_64 only. The backdoor is injected by a single test at the end of configure, which in turn use that binary test to uncompress some code and execute it, which in turn compromise the existing build system to inject the another code into the resulting library binary. So it has 2 parts. The one that injects it, and the one that gets injected. Second one at least is x86_64 only. And the tests in configure system also point to x86_64 only.
https://www.openwall.com/lists/oss-security/2024/03/29/4/1 This is part of what configure does. The modifications on configure already did before. The injection part. https://www.openwall.com/lists/oss-security/2024/03/29/4/2 This is the actual objected injected.
An important comment from oss-security mailing list message, <https://www.openwall.com/lists/oss-security/2024/03/31/4>: commit f9cf4c05edd14dedfe63833f8ccbe41b55823b00 (HEAD -> master, origin/master, origin/HEAD) Author: Lasse Collin <lasse.collin@...aani.org> Date: Sat Mar 30 14:36:28 2024 +0200 CMake: Fix sabotaged Landlock sandbox check. It never enabled it.
(In reply to revamps.warbler-0w from comment #17) > Github has disabled the XZ repository entirely as of 1:29 AM UTC Yes, and should remain that way for a very long time or indefinitely. The current version of xz that's not "compromised" need to be frozen and excluded from receiving updates. Happy threat hunting.
Please keep general commentary out of the bug - those are better placed on forums, mailing lists, IRC, etc. There's a lot of people CC'd to this for important developments.
