Gentoo compresses stage3 tarballs using pixz. In its default mode of operation, pixz generates "indexed" tarballs. When these indexed tarballs are decompressed using the "xz" utiltity, the resulting output contains the index data at the end of the file, which confuses tar when invoked with the --ignore-zeros option. This may confuse tools like "machinectl import-tar" from systemd. A possible workaround would be to pass "-t" to pixz to disable indexing. alternatively, we could stop using pixz altogether.
Created a PR for pyDeComp which resolves the issue: https://github.com/dol-sen/pyDeComp/pull/8
this should be resolved soon. I pinged dolsen to get pydecomp change in, should happen this weekend. if it does not we'll simply switch to xz. xz is horribly and unreasonably slow, so stage builds will see a noticeable increast in build times. pixz without index will be just as slow, so there's no loss really, unless we switch back to bz2/lbzip2 or gz/pigz, which produce larger files but are much faster (seconds instead of minutes, compared to xz).
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=50141b75d21e644e36be4d0b010345dbb0862161 commit 50141b75d21e644e36be4d0b010345dbb0862161 Author: Georgy Yakovlev <gyakovlev@gentoo.org> AuthorDate: 2021-11-04 01:06:04 +0000 Commit: Georgy Yakovlev <gyakovlev@gentoo.org> CommitDate: 2021-11-04 01:07:30 +0000 dev-python/pydecomp: add upstream patch to avoid pixz index Bug: https://bugs.gentoo.org/787194 Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org> dev-python/pydecomp/files/0.3-no-pixz-index.patch | 55 +++++++++++++++++++++++ dev-python/pydecomp/pydecomp-0.3-r2.ebuild | 27 +++++++++++ 2 files changed, 82 insertions(+)
using ppc64le stages as guinea pig with this patch merged. let's see if it's enough to make it load-able by machinectl import-tar/pull-tar
> sudo machinectl pull-tar --verify=checksum https://distfiles.gentoo.org/releases/ppc/autobuilds/20211117T192136Z/stage3-ppc64le-openrc-20211117T192136Z.tar.xz gy .... Download of https://distfiles.gentoo.org/releases/ppc/autobuilds/20211117T192136Z/stage3-ppc64le-openrc-20211117T192136Z.tar.xz complete. SHA256 checksum of https://distfiles.gentoo.org/releases/ppc/autobuilds/20211117T192136Z/stage3-ppc64le-openrc-20211117T192136Z.tar.xz is valid. Created new local image 'gy'. Operation completed successfully. Exiting. yes, some images are ok now. rest of releng stages need to install pydecomp-0.3-r2 catalyst also has a patch that generates .sha256 files posted and will emit those files: https://archives.gentoo.org/gentoo-catalyst/message/5b74ec38794a13238c56251bd8bcc3b7 https://distfiles.gentoo.org/releases/ppc/autobuilds/20211117T192136Z/stage3-ppc64le-openrc-20211117T192136Z.tar.xz.sha256 so machinectl --verify=checksum will work signing those sha256 files is also planned.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4093fbb2a5bb5b18a7a68ed9ee68fddc4b8d7bd8 commit 4093fbb2a5bb5b18a7a68ed9ee68fddc4b8d7bd8 Author: Georgy Yakovlev <gyakovlev@gentoo.org> AuthorDate: 2021-11-22 00:22:56 +0000 Commit: Georgy Yakovlev <gyakovlev@gentoo.org> CommitDate: 2021-11-22 00:25:30 +0000 dev-python/pydecomp: drop 0.3-r1 Bug: https://bugs.gentoo.org/787194 Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org> dev-python/pydecomp/pydecomp-0.3-r1.ebuild | 25 ------------------------- 1 file changed, 25 deletions(-)
we now just need pydecomp update so spread to catalyst hosts stages would become unbroken after some time, rest is done. I'll close this one.
