Created attachment 790271 [details] emerge.log XZ compressed I realise some packages depend on others before they can be rebuilt But it feels like after the initial batch of virtual packages are built, there are a lot that go serially Attaching an emerge world --empty with a -j100 which hopefully highlights the issue
<sam_> FireBurn: it'd be interesting if you could tell me if it still happens with --implicit-system-deps=n in EMERGE_DEFAULT_OPTS
>>> Installing (59 of 1483) media-libs/fdk-aac-2.0.2::gentoo >>> Installing (113 of 1483) app-text/scdoc-1.11.2-r1::gentoo >>> Installing (120 of 1483) acct-group/libreoffice-0::gentoo >>> Emerging (162 of 1483) acct-user/libreoffice-0::gentoo >>> Installing (117 of 1483) app-shells/quoter-4.2::gentoo >>> Installing (118 of 1483) app-text/libexttextcat-3.4.6::gentoo >>> Installing (124 of 1483) app-misc/editor-wrapper-4-r1::gentoo >>> Installing (119 of 1483) app-text/libnumbertext-1.0.10::gentoo >>> Installing (125 of 1483) virtual/libintl-0-r2::gentoo >>> Emerging (163 of 1483) sys-libs/timezone-data-2022a::gentoo >>> Installing (123 of 1483) sci-libs/suitesparseconfig-5.4.0::gentoo >>> Installing (122 of 1483) sys-devel/ucpp-1.3.5::gentoo >>> Installing (128 of 1483) acct-group/man-0-r1::gentoo >>> Emerging (164 of 1483) acct-user/man-1-r1::gentoo >>> Installing (129 of 1483) app-text/manpager-1::gentoo >>> Installing (130 of 1483) acct-group/ssmtp-0::gentoo >>> Installing (121 of 1483) dev-util/cppunit-1.15.1-r3::gentoo >>> Installing (127 of 1483) virtual/libiconv-0-r2::gentoo >>> Emerging (165 of 1483) sys-fs/dosfstools-4.2::gentoo >>> Installing (131 of 1483) acct-group/mail-0-r1::gentoo >>> Emerging (166 of 1483) acct-user/mail-0-r1::gentoo >>> Emerging (167 of 1483) acct-user/postmaster-0-r1::gentoo >>> Installing (111 of 1483) dev-lang/go-1.18.3::gentoo >>> Installing (126 of 1483) dev-util/pkgconf-1.8.0-r1::gentoo >>> Emerging (168 of 1483) virtual/pkgconfig-2-r1::gentoo >>> Installing (133 of 1483) acct-group/rtkit-0-r1::gentoo >>> Emerging (169 of 1483) acct-user/rtkit-0-r1::gentoo >>> Installing (132 of 1483) app-emulation/wine-gecko-2.47.2::gentoo >>> Installing (134 of 1483) dev-util/opencl-headers-2022.05.18::gentoo >>> Installing (13 of 1483) net-analyzer/sslscan-2.0.15::gentoo >>> Installing (135 of 1483) dev-lang/python-exec-2.4.9::gentoo >>> Installing (136 of 1483) acct-user/messagebus-0-r1::gentoo >>> Installing (139 of 1483) sys-devel/gcc-config-2.5-r1::gentoo >>> Installing (138 of 1483) sys-devel/binutils-config-5.4.1::gentoo >>> Installing (137 of 1483) games-util/esteam-0.20220522::steam-overlay >>> Installing (141 of 1483) acct-user/systemd-timesync-0-r1::gentoo >>> Installing (142 of 1483) acct-user/systemd-network-0-r1::gentoo >>> Installing (143 of 1483) acct-user/minidlna-0::gentoo >>> Installing (144 of 1483) acct-user/root-0-r1::gentoo >>> Emerging (170 of 1483) acct-user/nobody-0::gentoo >>> Installing (145 of 1483) acct-user/polkitd-0-r1::gentoo >>> Installing (153 of 1483) acct-user/systemd-coredump-0-r1::gentoo >>> Installing (147 of 1483) acct-user/systemd-resolve-0-r1::gentoo >>> Jobs: 145 of 1483 complete, 1 running Load avg: 3.8, 11.8, 17.0 Pretty sure it's getting the same issue
>>> Emerging (206 of 1483) virtual/acl-0-r2::gentoo >>> Installing (206 of 1483) virtual/acl-0-r2::gentoo >>> Emerging (207 of 1483) virtual/yacc-0::gentoo >>> Installing (207 of 1483) virtual/yacc-0::gentoo >>> Emerging (208 of 1483) virtual/perl-File-Spec-3.840.0::gentoo >>> Installing (208 of 1483) virtual/perl-File-Spec-3.840.0::gentoo >>> Emerging (209 of 1483) virtual/perl-ExtUtils-MakeMaker-7.640.0::gentoo >>> Installing (209 of 1483) virtual/perl-ExtUtils-MakeMaker-7.640.0::gentoo >>> Emerging (210 of 1483) virtual/perl-Encode-3.170.0::gentoo >>> Installing (210 of 1483) virtual/perl-Encode-3.170.0::gentoo >>> Emerging (211 of 1483) virtual/perl-Carp-1.520.0-r2::gentoo >>> Installing (211 of 1483) virtual/perl-Carp-1.520.0-r2::gentoo >>> Emerging (212 of 1483) virtual/perl-Exporter-5.770.0::gentoo >>> Installing (212 of 1483) virtual/perl-Exporter-5.770.0::gentoo >>> Emerging (213 of 1483) virtual/perl-Scalar-List-Utils-1.620.0::gentoo >>> Installing (213 of 1483) virtual/perl-Scalar-List-Utils-1.620.0::gentoo >>> Emerging (214 of 1483) virtual/tmpfiles-0-r3::gentoo >>> Installing (214 of 1483) virtual/tmpfiles-0-r3::gentoo >>> Emerging (215 of 1483) virtual/libcrypt-2::gentoo >>> Installing (215 of 1483) virtual/libcrypt-2::gentoo >>> Emerging (216 of 1483) virtual/libudev-232-r7::gentoo >>> Installing (216 of 1483) virtual/libudev-232-r7::gentoo >>> Emerging (217 of 1483) virtual/udev-217-r5::gentoo
The above is all output from and emerge --empty world -j100
emerge -j has been broken for years here It sometimes (randomly, rarely) does several package in //, but most of the time it does them serially, using <5% of cpu, almost no RAM, unless a package has log of files to compile (then make -j does its jobs). I triple checked all possible causes, and failed to understand why. It's not because of dependancies, and it's not because of -l
Whenever I am doing a world rebuild, I precalculate all deps and feed the list to emerge --oneshot --nodeps --keep-going --jobs=n
Can you provide mode details ? How do you 'precalculate' ?
I use emerge --pretend to make the list and then I use --nodeps for the actual merge. This way the dependency calculation does not get in the way of parallelization.
(In reply to Thomas Capricelli from comment #7) > Can you provide mode details ? How do you 'precalculate' ? Of course, that way dependencies are not respected, but the CPU is busy most of the time. Therefore is important to be up to date before world recompile. Here is my example, basically three stages: 1 - toolchain (two substages: small ones and big ones), 2 - majority of the world packages, 3 - big packages: emaint --fix all mkdir /tmp/costel &>/dev/null cd /tmp/costel rm world_list.txt /var/log/portage/elog/summary* &>/dev/null excluded='' for f in sys-devel/gcc-14 sys-devel/binutils sys-libs/glibc sys-devel/llvm-19 sys-devel/clang-19 sys-apps/portage sys-kernel/linux-headers \ sys-devel/gcc-config dev-libs/mpfr dev-libs/mpc dev-build/libtool sys-devel/mold net-libs/webkit-gtk app-office/libreoffice-2 ; \ do excluded="${excluded}${f}\|" ; done export excluded=${excluded::-2} for f in `emerge -pe world | grep ebuild | grep -v $excluded | gawk '{print $4;}'` ; do echo -n " ="$f ; done > /tmp/costel/world_list.txt unset excluded_ebuilds emerge -O1 --quiet-build sys-apps/portage sys-kernel/linux-headers sys-devel/gcc-config dev-libs/mpfr dev-libs/mpc dev-build/libtool \ sys-devel/binutils sys-devel/mold sys-libs/glibc emerge -O1 -j1 --quiet-build sys-devel/llvm sys-devel/clang sys-devel/gcc emerge -O1 `cat /tmp/costel/world_list.txt` emerge -O1 -j1 --quiet-build net-libs/webkit-gtk:4.1 app-office/libreoffice
I don't think you can safely do that if backtracking causes a particular solution to be chosen but another emerge set of commands/options leads to a different solution being chosen. I've even seen that myself a few times with protobuf.
(In reply to Sam James from comment #10) > I don't think you can safely do that if backtracking causes a particular > solution to be chosen but another emerge set of commands/options leads to a > different solution being chosen. > > I've even seen that myself a few times with protobuf. Yes, all safeties are off using that method and errors can happens. It is not something to recomand for general use. Why taking this risk, because IT IS A RISK involved ? To save time. But, with EMERGE_DEFAULT_OPTS="... --with-bdeps=y --complete-graph ..." and emerge -auvDN world done a priori, last error occurred to me with gcc 10. I use this method once per year, on gcc major update or on new build/system, just to be on the safe side. Bottom line, you have to be sure that all required bits are already there, you are doing just a rebuild, no update or downgrade, nothing on revdep-rebuild list and be prepared for unexpected failures.
