Created attachment 882781 [details] catalyst log Observed in m68k stage3 build, and I *think* this might be a portage problem: >>> Emerging binary (225 of 285) sys-apps/coreutils-9.4::gentoo >>> Installing (225 of 285) sys-apps/coreutils-9.4::gentoo >>> Completed (225 of 285) sys-apps/coreutils-9.4::gentoo >>> Emerging binary (226 of 285) sys-apps/acl-2.3.1-r2::gentoo >>> Failed to emerge sys-apps/acl-2.3.1-r2, Log file: >>> '/var/log/portage/sys-apps:acl-2.3.1-r2:20240118-153207.log' >>> Extracting info * Package: sys-apps/acl-2.3.1-r2:0 * Repository: gentoo * USE: elibc_glibc nls split-usr kernel_linux m68k * FEATURES: preserve-libs sandbox userpriv usersandbox /bin/mv: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory * ERROR: sys-apps/acl-2.3.1-r2::gentoo failed (setup phase): * error processing environment * * Call stack: * ebuild.sh, line 561: Called die * The specific snippet of code: * __preprocess_ebuild_env || die "error processing environment" * * If you need support, post the output of `emerge --info '=sys-apps/acl-2.3.1-r2::gentoo'`, * the complete build log and the output of `emerge -pqv '=sys-apps/acl-2.3.1-r2::gentoo'`. * The complete build log is located at '/var/log/portage/sys-apps:acl-2.3.1-r2:20240118-153207.log'. * For convenience, a symlink to the build log is located at '/var/tmp/portage/sys-apps/acl-2.3.1-r2/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/sys-apps/acl-2.3.1-r2/temp/environment'. * Working directory: '/var/tmp/portage/sys-apps/acl-2.3.1-r2/empty' * S: '/var/tmp/portage/sys-apps/acl-2.3.1-r2/work/acl-2.3.1' /bin/mv: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory * ERROR: sys-apps/acl-2.3.1-r2::gentoo failed: * error processing environment * And from there everything goes downhill... Full log attached
There's two coreutils-9.4 binary packages and I dont know which one is used. However, here's the Packages info for both: BDEPEND: app-arch/xz-utils dev-lang/perl BUILD_ID: 1 BUILD_TIME: 1693521885 CPV: sys-apps/coreutils-9.4 DEFINED_PHASES: configure install postinst prepare setup test unpack DEPEND: sys-apps/acl dev-libs/openssl:0/3= sys-apps/attr virtual/libintl EAPI: 8 IUSE: acl caps gmp hostname kill multicall nls +openssl selinux +split-usr static test vanilla xattr verify-sig KEYWORDS: ~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~loong ~m68k ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86 ~x86-linux LICENSE: GPL-3+ MD5: 1c265000b19098eaa3d2c7cad5936c1e PATH: sys-apps/coreutils/coreutils-9.4-1.gpkg.tar PROVIDES: m68k_32: libstdbuf.so RDEPEND: sys-apps/acl dev-libs/openssl:0/3= sys-apps/attr virtual/libintl !app-misc/realpath !<sys-apps/util-linux-2.13 !<sys-apps/sandbox-2.10-r4 !sys-apps/stat !net-mail/base64 !sys-apps/mktemp !<app-forensics/tct-1.18-r1 !<net-fs/netatalk-2.0.3-r4 REQUIRES: m68k_32: libacl.so.1 libattr.so.1 libc.so.6 libcrypto.so.3 RESTRICT: test SHA1: b93402310b813d304eab9467e4eaca6bb38b92ed SIZE: 3973120 USE: acl elibc_glibc kernel_linux m68k nls openssl split-usr xattr MTIME: 1693521964 REPO: gentoo BDEPEND: app-arch/xz-utils dev-lang/perl BUILD_ID: 2 BUILD_TIME: 1704362555 CPV: sys-apps/coreutils-9.4 DEFINED_PHASES: configure install postinst prepare setup test unpack DEPEND: sys-apps/acl dev-libs/openssl:0/3= sys-apps/attr virtual/libintl EAPI: 8 IUSE: acl caps gmp hostname kill multicall nls +openssl selinux +split-usr static test vanilla xattr verify-sig KEYWORDS: ~alpha ~amd64 arm arm64 hppa ~ia64 ~loong ~m68k ~mips ppc ppc64 ~riscv ~s390 sparc ~x86 ~x86-linux LICENSE: GPL-3+ MD5: 9120ad5d8c685500423df3ce1e8a4f53 PATH: sys-apps/coreutils/coreutils-9.4-2.gpkg.tar PROVIDES: m68k_32: libstdbuf.so RDEPEND: sys-apps/acl dev-libs/openssl:0/3= sys-apps/attr virtual/libintl !<sys-apps/util-linux-2.13 !<sys-apps/sandbox-2.10-r4 !sys-apps/stat !net-mail/base64 !sys-apps/mktemp !<app-forensics/tct-1.18-r1 !<net-fs/netatalk-2.0.3-r4 >=sys-libs/glibc-2.38-r8 REQUIRES: m68k_32: ld.so.1 libacl.so.1 libattr.so.1 libc.so.6 libcrypto.so.3 RESTRICT: test SHA1: 6dffc58e0aa3d99f7a092230eece525fb692e666 SIZE: 4003840 USE: acl elibc_glibc kernel_linux m68k nls openssl split-usr xattr MTIME: 1704362599 REPO: gentoo
Same just happened for an alpha stage3 build. >>> Emerging binary (228 of 289) virtual/perl-Math-BigInt-1.999.842::gentoo >>> Installing (228 of 289) virtual/perl-Math-BigInt-1.999.842::gentoo >>> Completed (228 of 289) virtual/perl-Math-BigInt-1.999.842::gentoo >>> Emerging binary (229 of 289) sys-apps/coreutils-9.4::gentoo >>> Installing (229 of 289) sys-apps/coreutils-9.4::gentoo >>> Completed (229 of 289) sys-apps/coreutils-9.4::gentoo >>> Emerging binary (230 of 289) sys-apps/acl-2.3.1-r2::gentoo >>> Failed to emerge sys-apps/acl-2.3.1-r2, Log file: >>> '/var/log/portage/sys-apps:acl-2.3.1-r2:20240121-170157.log' >>> Extracting info [32m * [39;49;00mPackage: sys-apps/acl-2.3.1-r2:0 [32m * [39;49;00mRepository: gentoo [32m * [39;49;00mUSE: nls elibc_glibc kernel_linux alpha [32m * [39;49;00mFEATURES: preserve-libs sandbox userpriv usersandbox /usr/bin/mv: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory [31;01m*[0m ERROR: sys-apps/acl-2.3.1-r2::gentoo failed (setup phase): [31;01m*[0m error processing environment [31;01m*[0m [31;01m*[0m Call stack: [31;01m*[0m ebuild.sh, line 561: Called die [31;01m*[0m The specific snippet of code: [31;01m*[0m __preprocess_ebuild_env || die "error processing environment" [31;01m*[0m [31;01m*[0m If you need support, post the output of `emerge --info '=sys-apps/acl-2.3.1-r2::gentoo'`, [31;01m*[0m the complete build log and the output of `emerge -pqv '=sys-apps/acl-2.3.1-r2::gentoo'`. [31;01m*[0m The complete build log is located at '/var/log/portage/sys-apps:acl-2.3.1-r2:20240121-170157.log'. [31;01m*[0m For convenience, a symlink to the build log is located at '/var/tmp/portage/sys-apps/acl-2.3.1-r2/temp/build.log'. [31;01m*[0m The ebuild environment file is located at '/var/tmp/portage/sys-apps/acl-2.3.1-r2/temp/environment'. [31;01m*[0m Working directory: '/var/tmp/portage/sys-apps/acl-2.3.1-r2/empty' [31;01m*[0m S: '/var/tmp/portage/sys-apps/acl-2.3.1-r2/work/acl-2.3.1' /usr/bin/mv: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory [31;01m*[0m ERROR: sys-apps/acl-2.3.1-r2::gentoo failed: [31;01m*[0m error processing environment [31;01m*[0m [31;01m*[0m Call stack: [31;01m*[0m misc-functions.sh, line 17: Called source '/var/tmp/portage/._portage_reinstall_.9a3h70z_/bin/ebuild.sh' [31;01m*[0m ebuild.sh, line 561: Called die [31;01m*[0m The specific snippet of code: [31;01m*[0m __preprocess_ebuild_env || die "error processing environment" [31;01m*[0m ...
Same observed in the loong stage3, with binary tar emerged before source acl [...] >>> Emerging binary (231 of 280) sys-apps/attr-2.5.2::gentoo >>> Installing (231 of 280) sys-apps/attr-2.5.2::gentoo >>> Completed (231 of 280) sys-apps/attr-2.5.2::gentoo >>> Emerging binary (232 of 280) virtual/acl-0-r2::gentoo >>> Installing (232 of 280) virtual/acl-0-r2::gentoo >>> Completed (232 of 280) virtual/acl-0-r2::gentoo >>> Emerging binary (233 of 280) app-arch/tar-1.35::gentoo >>> Installing (233 of 280) app-arch/tar-1.35::gentoo >>> Completed (233 of 280) app-arch/tar-1.35::gentoo >>> Emerging binary (234 of 280) sys-devel/gettext-0.22.4::gentoo >>> Installing (234 of 280) sys-devel/gettext-0.22.4::gentoo >>> Completed (234 of 280) sys-devel/gettext-0.22.4::gentoo >>> Emerging (235 of 280) sys-apps/acl-2.3.2::gentoo >>> Failed to emerge sys-apps/acl-2.3.2, Log file: >>> '/var/log/portage/sys-apps:acl-2.3.2:20240206-215741.log' [32m * [39;49;00mPackage: sys-apps/acl-2.3.2:0 [32m * [39;49;00mRepository: gentoo [32m * [39;49;00mMaintainer: base-system@gentoo.org [32m * [39;49;00mUSE: elibc_glibc kernel_linux loong nls split-usr [32m * [39;49;00mFEATURES: preserve-libs sandbox userpriv usersandbox /bin/tar: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory [31;01m*[0m ERROR: sys-apps/acl-2.3.2::gentoo failed (unpack phase): [31;01m*[0m unpack: failure unpacking acl-2.3.2.tar.xz [31;01m*[0m [31;01m*[0m Call stack: [31;01m*[0m ebuild.sh, line 136: Called src_unpack [31;01m*[0m environment, line 2109: Called default [31;01m*[0m phase-functions.sh, line 871: Called default_src_unpack [31;01m*[0m phase-functions.sh, line 898: Called __eapi0_src_unpack [31;01m*[0m phase-helpers.sh, line 807: Called unpack 'acl-2.3.2.tar.xz' [31;01m*[0m phase-helpers.sh, line 547: Called __unpack_tar 'xz -T32 -d' [31;01m*[0m phase-helpers.sh, line 372: Called __assert_sigpipe_ok 'unpack: failure unpacking acl-2.3.2.tar.xz' [31;01m*[0m isolated-functions.sh, line 41: Called die [31;01m*[0m The specific snippet of code: [31;01m*[0m [[ ${x} -ne 0 && ${x} -ne ${PORTAGE_SIGPIPE_STATUS:-141} ]] && die "$@" [31;01m*[0m [31;01m*[0m If you need support, post the output of `emerge --info '=sys-apps/acl-2.3.2::gentoo'`, [31;01m*[0m the complete build log and the output of `emerge -pqv '=sys-apps/acl-2.3.2::gentoo'`. [31;01m*[0m The complete build log is located at '/var/log/portage/sys-apps:acl-2.3.2:20240206-215741.log'. [31;01m*[0m For convenience, a symlink to the build log is located at '/var/tmp/portage/sys-apps/acl-2.3.2/temp/build.log'. [31;01m*[0m The ebuild environment file is located at '/var/tmp/portage/sys-apps/acl-2.3.2/temp/environment'. [31;01m*[0m Working directory: '/var/tmp/portage/sys-apps/acl-2.3.2/work' [31;01m*[0m S: '/var/tmp/portage/sys-apps/acl-2.3.2/work/acl-2.3.2'
Created attachment 884435 [details] glibc runtime cycle containing 29 packages from docker.io/gentoo/stage3:systemd-20240205 image The injected libc deps are definitely coming into play here. The runtime cycle for the docker.io/gentoo/stage3:systemd-20240205 image has 29 packages.
It should help to increase the priority of deps where there is runtime + buildtime intersection, as suggested in bug 921333 comment 12. Since injected libc deps are runtime only, they'll have lower priority. We could also possibly use soname dependencies to enrich priorities without doing full soname dependency accounting, ignoring soname dependencies that don't intersect with *DEPEND.
I made a big and very ugly reproducer (sorry), based on the stage builds. * Download the 750M .tar.xz from dev.gentoo.org/~dilfridge/dilfridge-m68k.tar.xz * Unpack it in /var/lib/machines Set up your system for a qemu chroot * Build app-emulation/qemu[static-user] with QEMU_USER_TARGETS="m68k" * Copy /usr/bin/qemu-m68k to /var/lib/machines/dilfridge-m68k/usr/bin * Configure binfmt-misc see e.g. https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Compiling_with_QEMU_user_chroot#Configuration Start the container via systemd-nspawn -M dilfridge-m68k On the emulated shell, run emerge -eavk world You'll see that tar[acl] is scheduled for binary install before acl.
(In reply to Andreas K. Hüttel from comment #6) > I made a big and very ugly reproducer (sorry), based on the stage builds. > > * Download the 750M .tar.xz from > dev.gentoo.org/~dilfridge/dilfridge-m68k.tar.xz > PS. Obviously I have this running here, so I can also provide debug output, log files, etc... Maybe easier.
Created attachment 884505 [details] output of "emerge -epvk world"
Created attachment 884506 [details] output of "emerge -epvk --debug world"
zac was (of course) right, get_smallest_cycle goes wild again with many 1-node cycles. We also have: ``` runtime cycle digraph (5 nodes): (app-arch/tar-1.35-1:0/0::gentoo, binary scheduled for merge) depends on (virtual/acl-0-r2-1:0/0::gentoo, binary scheduled for merge) (runtime) (sys-devel/gettext-0.22.4-1:0/0::gentoo, binary scheduled for merge) depends on (sys-apps/attr-2.5.2-r1:0/0::gentoo, ebuild scheduled for merge) (runtime) (virtual/acl-0-r2-1:0/0::gentoo, binary scheduled for merge) (runtime) (sys-apps/attr-2.5.2-r1:0/0::gentoo, ebuild scheduled for merge) depends on (sys-devel/gettext-0.22.4-1:0/0::gentoo, binary scheduled for merge) (buildtime) (sys-apps/acl-2.3.2-r1:0/0::gentoo, ebuild scheduled for merge) depends on (sys-apps/attr-2.5.2-r1:0/0::gentoo, ebuild scheduled for merge) (buildtime) (sys-devel/gettext-0.22.4-1:0/0::gentoo, binary scheduled for merge) (buildtime) (virtual/acl-0-r2-1:0/0::gentoo, binary scheduled for merge) depends on (sys-apps/acl-2.3.2-r1:0/0::gentoo, ebuild scheduled for merge) (runtime) ```
Popped up again, this time on hppa stage builds (with broken tar half the way)
mgorny hit this during dist-kernel binpkg builds using https://github.com/projg2/binpkg-docker. He's given me a set of binpkgs on jiji (our arm64 builder) where I can reproduce it for sed/acl. ``` >>> Extracting info * Package: acct-user/polkitd-0-r3:0 * Repository: gentoo * USE: arm64 kernel_linux elibc_glibc * FEATURES: preserve-libs userpriv >>> Extracting acct-user/polkitd-0-r3 sed: /lib64/libc.so.6: version `GLIBC_2.38' not found (required by /usr/lib64/libacl.so.1) * ERROR: acct-user/polkitd-0-r3::gentoo failed (instprep phase): * (no error message) * ``` Let me try extract it and I'll upload the reproducer here and then try go from there. Obviously, at worst, we can try dilfridge's one, but I was reluctant unless I *had* to given it involves qemu-user and hence is slow.
Created attachment 894430 [details] bug922629.bash OK, here's a partial reduction (almost-standalone script, but it needs a binpkg cache too).
(In reply to Sam James from comment #13) > OK, here's a partial reduction (almost-standalone script, but it needs a > binpkg cache too). https://dev.gentoo.org/~sam/bugs/gentoo/bug922629-binpkg-cache.tar
(In reply to Sam James from comment #14) > (In reply to Sam James from comment #13) > > OK, here's a partial reduction (almost-standalone script, but it needs a > > binpkg cache too). > > https://dev.gentoo.org/~sam/bugs/gentoo/bug922629-binpkg-cache.tar hm - I don't see any implicit libc deps in here. This container has sys-apps/portage-3.0.51 which implies whatever is building them is probably equally old (as it uses the same Dockerfile). -> this is probably different (and invalid), compared to the releng issue.
I'm going to look at dilfridge's reproducer now.
(In reply to Andreas K. Hüttel from comment #6) > > You'll see that tar[acl] is scheduled for binary install before acl. I guess the tree state has changed since then, as it looks OK now: ``` # emerge -epvk world | grep -E "(app-arch/tar|sys-apps/acl)" [ebuild N ] sys-apps/acl-2.3.2-r1::gentoo USE="nls -static-libs" 363 KiB [binary R ] app-arch/tar-1.35-1::gentoo USE="acl* nls* xattr* -minimal (-selinux) -verify-sig" 0 KiB [...] ``` I'll try a ::gentoo commit from around the time you posted this comment.
(if only we had REPO_REVISIONS sooner, eh...) OK, with 0267c0b5d945f1ed274b56a08f3076c061fb6541, I can reproduce: ``` [binary N ] virtual/acl-0-r2-1::gentoo USE="-static-libs" 0 KiB [binary R ] app-arch/tar-1.35-1::gentoo USE="acl* nls* xattr* -minimal (-selinux) -verify-sig" 0 KiB [binary R ] sys-devel/gettext-0.22.4-1::gentoo USE="acl* cxx* ncurses* nls* (xattr) -doc (-emacs) -git (-java) -openmp -static-libs -verify-sig" 0 KiB [ebuild R ] sys-apps/attr-2.5.2-r1::gentoo USE="nls* -debug -static-libs" 327 KiB [ebuild N ] sys-apps/acl-2.3.2-r1::gentoo USE="nls -static-libs" 363 KiB `` It took ages though because of qemu-user (670s), so going to try remove that from the equation with either emerge --root or mangling the binpkg index to pretend it's amd64 (given we just need a merge list, nothing else)...
Created attachment 894472 [details] test_tar_merge_order.py OK, I think I have a testcase.
(In reply to Sam James from comment #19) > Created attachment 894472 [details] > test_tar_merge_order.py > > OK, I think I have a testcase. The key was app-alternatives/tar.
I bisected with this and got commit 49e01d041c74680a81860b819daff812d83df02f Author: Zac Medico <zmedico@gentoo.org> Date: Mon Nov 27 19:42:17 2023 -0800 find_smallest_cycle: Optimize to traverse fewer nodes If gather_deps is traversing a cycle that is greater than or equal to the size of the current smallest_cycle, then abort early. Also abort early if we traverse a node encountered in a previous cycle for the same ignore_priority, since that means the two cycles are identical. On my laptop, this brings the emerge -pe @world time down to 3m28.884s, compared to 10m44.268s with portage-3.0.55. Bug: https://bugs.gentoo.org/918682 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/depgraph.py | 36 +++++++++++++++++++++++++++++++++--- lib/portage/tests/resolver/test_rebuild_ghostscript.py | 2 +- lib/portage/tests/resolver/test_runtime_cycle_merge_order.py | 7 +++++-- 3 files changed, 39 insertions(+), 6 deletions(-) If I revert on master, the testcase goes from XFAIL->XPASS.
Since we have a working test case, we should go ahead and revert 49e01d041c74680a81860b819daff812d83df02f and add the test case. Later we can try to optimize it again but without breaking the test case.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=96e4f95cc8c0d544d375b28394dafe8809c4bc9b commit 96e4f95cc8c0d544d375b28394dafe8809c4bc9b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-05-26 18:23:27 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-05-26 18:34:04 +0000 Revert "find_smallest_cycle: Optimize to traverse fewer nodes" This reverts commit 49e01d041c74680a81860b819daff812d83df02f in order to fix bug 922629. Later we can try to optimize it again but without breaking testTarMergeOrder. Bug: https://bugs.gentoo.org/922629 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/depgraph.py | 36 ++-------------------- .../tests/resolver/test_rebuild_ghostscript.py | 2 +- .../resolver/test_runtime_cycle_merge_order.py | 7 ++--- lib/portage/tests/resolver/test_tar_merge_order.py | 4 ++- 4 files changed, 9 insertions(+), 40 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=11bded10a2a4f143e8d7f8ccb2f91f2b6fed59b5 commit 11bded10a2a4f143e8d7f8ccb2f91f2b6fed59b5 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-05-26 14:44:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-05-26 18:27:02 +0000 tests: add testcase for app-arch/tar merge order with binpkgs In the bug, dilfridge reports releng@ is hitting the following: ``` $ emerge -epvk world | grep -E "(app-arch/tar|sys-apps/acl)" [ebuild N ] sys-apps/acl-2.3.2-r1::gentoo USE="nls -static-libs" 363 KiB [binary R ] app-arch/tar-1.35-1::gentoo USE="acl* nls* xattr* -minimal (-selinux) -verify-sig" 0 KiB [...] ``` Test for bug #922629 where binary app-arch/tar[acl] was merged before its dependency sys-apps/acl (with virtual/acl merged but unsatisfied). It (appears to be) a bad interaction with @system containing app-alternatives/tar plus a circular dependency on app-arch/tar. The USE change is also important, as e.g. dropping "sys-apps/attr nls" from package.use makes things okay. XFAIL'd for now. Bug: https://bugs.gentoo.org/922629 Signed-off-by: Sam James <sam@gentoo.org> Closes: https://github.com/gentoo/portage/pull/1332 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/resolver/meson.build | 1 + lib/portage/tests/resolver/test_tar_merge_order.py | 495 +++++++++++++++++++++ 2 files changed, 496 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1dd08f05355b14c7594d0e0ccf46cdb2b1486cbe commit 1dd08f05355b14c7594d0e0ccf46cdb2b1486cbe Author: Sam James <sam@gentoo.org> AuthorDate: 2024-06-04 01:31:07 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-06-04 01:31:07 +0000 sys-apps/portage: add 3.0.65 Closes: https://bugs.gentoo.org/930802 Closes: https://bugs.gentoo.org/787623 Closes: https://bugs.gentoo.org/922629 Closes: https://bugs.gentoo.org/927241 Closes: https://bugs.gentoo.org/930992 Closes: https://bugs.gentoo.org/932739 Closes: https://bugs.gentoo.org/932804 Closes: https://bugs.gentoo.org/933297 Closes: https://bugs.gentoo.org/933385 Closes: https://bugs.gentoo.org/933442 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.65.ebuild | 242 +++++++++++++++++++++++++++++++++ 2 files changed, 243 insertions(+)
