Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 893536 - frivolous rebuilds triggered by --rebuild-if-new-ver and/or --rebuild-if-new-rev
Summary: frivolous rebuilds triggered by --rebuild-if-new-ver and/or --rebuild-if-new-rev
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Dependencies (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-02-07 20:14 UTC by Daniel Pouzzner
Modified: 2023-08-03 13:27 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Pouzzner 2023-02-07 20:14:52 UTC
I'm seeing that when  emerge --update --rebuild-if-new-ver @world  pulls in sys-devel/gcc:13 (a deselected slot), several packages get marked for rebuild that apparently shouldn't be.  --rebuild-if-new-rev brings in even more, including system glibc, even though the system glibc was built+installed with gcc:11 selected, and 05gcc-x86_64-pc-linux-gnu.conf listing the gcc:11 libs first.  I don't know how to get emerge to show the dependency tree to explain what triggered each rebuild -- is there a way?

Example of the behavior:

# eselect gcc show
x86_64-pc-linux-gnu-11
# emerge --ask --oneshot --keep-going=y --update --rebuild-if-new-rev --rebuild-if-new-ver --autounmask-keep-masks --verbose --verbose-conflicts --backtrack=10 @world
These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 37.49 s.

[ebuild     U *] sys-devel/gcc-13.0.1_pre20230205-r1:13::gentoo [13.0.1_pre20230205:13::gentoo] USE="(cxx) doc fortran graphite (multilib) nls nptl openmp pgo (pie) sanitize ssp -ada (-cet) (-custom-cflags) -d -debug -default-stack-clash-protection -default-znow (-fixed-point) -go -hardened (-ieee-long-double) -jit (-libssp) -lto -objc -objc++ -objc-gc -pch -rust -systemtap -test -valgrind -vanilla -vtv -zstd" 0 KiB
[ebuild  rR   ~] sys-libs/glibc-2.36-r7:2.2::gentoo  USE="gd multiarch (multilib) profile ssp stack-realign (static-libs) -audit -caps (-cet) -compile-locales -crypt (-custom-cflags) -doc -hash-sysv-compat -headers-only -multilib-bootstrap -nscd -perl (-selinux) -suid -systemd -systemtap -test (-vanilla)" 0 KiB
[ebuild  rR   ~] app-crypt/libb2-0.98.1-r3::gentoo  USE="openmp -native-cflags -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U *] cross-riscv32-unknown-linux-gnu/gcc-13.0.1_pre20230205-r1:13::crossdev [13.0.1_pre20230205:13::crossdev] USE="cxx nls nptl pch pie ssp -ada -cet -custom-cflags -d -debug -default-stack-clash-protection -default-znow -doc -fixed-point -fortran -go -graphite -hardened -ieee-long-double -jit -libssp -lto (-multilib) -objc -objc++ -objc-gc -openmp -pgo -rust -sanitize -systemtap -test -valgrind -vanilla -vtv -zstd" 0 KiB
[ebuild     U *] cross-riscv64-unknown-linux-gnu/gcc-13.0.1_pre20230205-r1:13::crossdev [13.0.1_pre20230205:13::crossdev] USE="cxx nls nptl pch pie ssp -ada -cet -custom-cflags -d -debug -default-stack-clash-protection -default-znow -doc -fixed-point -fortran -go -graphite -hardened -ieee-long-double -jit -libssp -lto (-multilib) -objc -objc++ -objc-gc -openmp -pgo -rust -sanitize -systemtap -test -valgrind -vanilla -vtv -zstd" 0 KiB
[ebuild  rR   ~] sci-libs/hdf5-1.12.2-r5:0/1.12.2::gentoo  USE="examples fortran hl unsupported zlib -cxx -debug -mpi -szip -threads" 0 KiB
[ebuild  rR    ] sci-libs/amd-2.4.6-r1::gentoo  USE="fortran -doc" 0 KiB
[ebuild  rR    ] sci-libs/fftw-3.3.10:3.0/3::gentoo  USE="fortran openmp threads -doc -mpi -test (-zbus)" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="avx avx2 fma3 sse sse2 -fma4" 0 KiB
[ebuild  rR   ~] cross-riscv32-unknown-linux-gnu/glibc-2.36-r7:2.2::crossdev  USE="crypt multiarch ssp static-libs -audit -caps -cet -compile-locales -custom-cflags -doc -gd -hash-sysv-compat -headers-only (-multilib) -multilib-bootstrap -nscd -perl -profile (-selinux) -stack-realign -suid -systemd -systemtap -test -vanilla" 0 KiB
[ebuild  rR   ~] cross-riscv64-unknown-linux-gnu/glibc-2.36-r7:2.2::crossdev  USE="crypt multiarch ssp static-libs -audit -caps -cet -compile-locales -custom-cflags -doc -gd -hash-sysv-compat -headers-only (-multilib) -multilib-bootstrap -nscd -perl -profile (-selinux) -stack-realign -suid -systemd -systemtap -test -vanilla" 0 KiB
[ebuild  rR   ~] net-libs/nodejs-18.14.0:0/18::gentoo  USE="icu inspector npm snapshot ssl system-icu system-ssl -debug -doc -lto -pax-kernel -systemtap -test" CPU_FLAGS_X86="sse2" 0 KiB
[ebuild  rR    ] x11-misc/xautolock-2.2_p7::gentoo  0 KiB
[ebuild  rR    ] sys-cluster/openmpi-4.1.4::gentoo  USE="fortran ipv6 -cma -cuda -cxx -java -libompitrace -peruse -romio" ABI_X86="(64) -32 (-x32)" OPENMPI_FABRICS="-knem -ofed" OPENMPI_OFED_FEATURES="-control-hdr-padding -dynamic-sl -rdmacm -udcm" OPENMPI_RM="-pbs (-slurm)" 0 KiB
[ebuild  rR    ] media-gfx/transfig-3.2.5e-r1::gentoo  0 KiB
[ebuild  rR   ~] sci-libs/cfitsio-4.2.0:0/10::gentoo  USE="bzip2 curl -threads -tools" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2 ssse3" 0 KiB
[ebuild  rR   ~] sci-libs/lapack-3.11::gentoo  USE="-deprecated -doc -eselect-ldso -lapacke -test" 0 KiB
[ebuild     U ~] dev-python/numpy-1.24.2::gentoo [1.24.0::gentoo] USE="lapack -test" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR    ] sci-libs/arpack-3.8.0::gentoo  USE="-doc -examples -mpi" 0 KiB
[ebuild  rR    ] sci-libs/qrupdate-1.1.2-r2::gentoo  0 KiB
[ebuild  rR   ~] dev-python/bottleneck-1.3.6::gentoo  USE="-test" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR    ] dev-python/numexpr-2.8.4::gentoo  PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR   ~] dev-python/h5py-3.8.0::gentoo  USE="-doc -examples -test" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR   ~] dev-python/mpi4py-3.1.4::gentoo  USE="-doc -examples -test" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR    ] media-libs/vigra-1.11.1-r8::gentoo  USE="fftw hdf5 jpeg openexr png python tiff zlib -doc -mpi -test -valgrind" PYTHON_TARGETS="python3_9 python3_10" 0 KiB
[ebuild  rR    ] dev-python/scipy-1.10.0::gentoo  USE="fortran (test-rust) -doc -test" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR   ~] dev-python/pandas-1.5.3::gentoo  USE="X (-doc) -full-support -minimal -test" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR   ~] dev-lang/R-4.2.2::gentoo  USE="X cairo doc icu jpeg lapack nls openmp perl png readline tiff tk -java -lto -minimal (-prefix) -profile -test" 0 KiB
[ebuild  rR   ~] dev-python/matplotlib-3.6.3::gentoo  USE="cairo gtk3 (-doc) -examples -excel -latex -qt5 -test -tk -webagg -wxwidgets" PYTHON_TARGETS="python3_9 python3_10 -python3_11" 0 KiB
[ebuild  rR   ~] net-libs/webkit-gtk-2.38.4-r410:4.1/0::gentoo  USE="X egl gnome-keyring gstreamer introspection jpeg2k jumbo-build lcms pdf (seccomp) spell (-aqua) -avif -examples -gamepad -gles2-only -systemd -test -wayland" 0 KiB
[ebuild  rR   ~] net-libs/webkit-gtk-2.38.4-r500:5/0::gentoo  USE="X egl gnome-keyring gstreamer introspection jpeg2k jumbo-build lcms pdf (seccomp) spell (-aqua) -avif -examples -gamepad -gles2-only -systemd -test -wayland" 0 KiB
[ebuild  rR   ~] dev-lang/rust-1.67.0:stable/1.67::gentoo  USE="clippy doc rustfmt system-bootstrap system-llvm verify-sig -debug -dist (-llvm-libunwind) (-miri) -nightly (-parallel-compiler) -profiler -rust-analyzer -rust-src -test -wasm" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" LLVM_TARGETS="(X86) -AArch64 -AMDGPU -ARM -AVR -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -RISCV -Sparc -SystemZ -WebAssembly -XCore" 0 KiB
[ebuild  rR    ] sci-mathematics/octave-7.3.0-r2:0/7.3.0::gentoo  USE="X curl fftw glpk gnuplot gui hdf5 imagemagick opengl postscript qhull qrupdate readline sndfile sparse ssl zlib -doc -fltk -java -json -portaudio -sundials" 0 KiB

Total: 32 packages (4 upgrades, 28 reinstalls), Size of downloads: 0 KiB
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-02-07 20:24:31 UTC
Not looked at this properly yet, but I'd like to note that:
1. I'm surprised (but glad) to see people using --rebuild-if-new-rev --rebuild-if-new-ver and they're not very common options to use, even if they have quite limited application;

2. I feel like the "spurious rebuilds" when saying "rebuild everything with new versions of my deps" is at best a bit confusing. What behaviour are you expecting, exactly? Portage _does not_ analyse which version it was built against. It does not analyse SONAME dependencies or anything of the like for this. These packages depend on gcc and hence get a rebuild with a new version of gcc.

It's possible that someone could extend Portage to have it try figure out which GCC version things were built with (at least for C++, this will be easyish, not sure about C), but it isn't supposed to do that right now.

The ebuilds in your list, I think, all explicitly depend on gcc rather than the standard @system thing because of e.g. openmp or specifying a minimum version.
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-02-07 20:27:53 UTC
I do think there's value in something which tries to figure out if we used an old GCC (or glibc or ...), but it's also kind of related to the issue of "how do I rebuild only C/C++/... applications and not Python if it's unnecessary".

I'm not sure if there's enough info in the VDB for this, it might be close enough based on NEEDED for libc.so and such though. But that doesn't help with your problem wrt exact GCC versions...
Comment 3 Daniel Pouzzner 2023-02-07 20:29:49 UTC
>These packages depend on gcc and hence get a rebuild with a new version of gcc.

Ideally, I would want a package that depends on gcc to get a rebuild only if the new version of gcc at issue would be the one that the dependent package would actually use.  With deselected gcc, it isn't, implying that what's missing here is slot sensitivity in the dependency calculation.

And for the example I showed, I think that's all that's needed to make --rebuild-if-* have expected+unsurprising results.
Comment 4 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-02-07 20:33:22 UTC
Ah, I follow now. I think we could do that, although I can't promise to implement it personally immediately.
Comment 5 Daniel Pouzzner 2023-02-07 20:34:31 UTC
Understood.  Yeah I was just wondering what effort would be involved...
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-02-07 20:35:00 UTC
(In reply to Sam James from comment #4)
> Ah, I follow now. I think we could do that, although I can't promise to
> implement it personally immediately.

Sorry, I mean: it'd be possible to say "rebuild if installed before a slot was ever around", but it's not possible to make it aware of what slot was used without obsoleting gcc-config and making an ebuild handle the selected variant.

(This is similar to bug 283587.)
Comment 7 Daniel Pouzzner 2023-02-07 20:38:34 UTC
I think the rule I have in mind would just be "upgrade of slotted package X triggers rebuild of dependent package Y iff the slot of X is the currently selected slot."
Comment 8 Daniel Pouzzner 2023-08-03 13:27:19 UTC
Another example of the syndrome:


These are the packages that would be merged, in order:
[...]
[ebuild  NS   ~] sys-devel/binutils-2.41:2.41::gentoo [2.38-r2:2.38::gentoo, 2.39-r5:2.39::gentoo, 2.40-r7:2.40::gentoo] USE="doc nls plugins static-libs (-cet) -debuginfod -gold -gprofng -hardened% -multitarget -pgo -test -vanilla -zstd" 0 KiB
[...]
The following packages are causing rebuilds:

  (sys-libs/binutils-libs-2.41:0/2.41::gentoo, ebuild scheduled for merge) causes rebuilds for:
    (dev-db/mariadb-10.11.4:10.11/18::gentoo, ebuild scheduled for merge)
    (dev-lang/ocaml-4.14.1:0/4.14.1::gentoo, ebuild scheduled for merge)


# eselect binutils list
[...]
 [44] x86_64-pc-linux-gnu-2.38
 [45] x86_64-pc-linux-gnu-2.39
 [46] x86_64-pc-linux-gnu-2.40 *


The syndrome is fairly benign at least -- just some extra wheel-spinning.