For context: I'm in process of updating two outdated Gentoo systems. I perform updates gradually, by placing a cloned ebuild tree Git repository at /usr/portage and switching between revisions. Right now, I'm at commit 32df66059ef7f888962368d698f5172c9ddb15ea from 2023-04-07, thus I have stable portage-3.0.44-r1 and unstable portage-3.0.46 (which is currently stable, according to gpo.zugaina.org). After a recent full update ("emerge -u --deep --newuse --with-bdeps=y world"), I was doing "emerge -avt --depclean", and I noticed that it's going to remove dev-lang/python-2.7.18_p16-r1. I checked and found out that one installed package still depends on this version of python: ====== % qdepends firefox | grep python-2 www-client/firefox-52.9.0-r1: >=app-portage/elt-patches-20170815 sys-apps/findutils x11-libs/libXdamage >=app-text/hunspell-1.2:0/1.7= >=sys-devel/automake-1.15.1:1.15 >=x11-libs/startup-notification-0.8 ~sys-devel/autoconf-2.13 >=sys-libs/zlib-1.2.3 >=x11-libs/cairo-1.10[X] media-libs/fontconfig app-arch/unzip x11-libs/gdk-pixbuf >=virtual/libffi-3.0.10 virtual/pkgconfig >=dev-lang/python-2.7.5-r2:2.7[ncurses,sqlite,ssl,threads] ... ====== ====== % grep python /var/db/pkg/www-client/firefox-52.9.0-r1/* /var/db/pkg/www-client/firefox-52.9.0-r1/DEPEND:>=dev-libs/nss-3.28.3 >=dev-libs/nspr-4.13.1 >=dev-lang/yasm-1.1 virtual/opengl virtual/pkgconfig >=dev-lang/python-2.7.5-r2:2.7[ncurses,sqlite,ssl,threads] ... ====== (This package is long gone from the tree, but I'm keeping it installed for now, even though I don't have a ebuild for it. www-client/firefox is in world, too, and no other versions of firefox are installed - they are all masked by /etc/portage/package.mask.) When I try to depclean specifically python:2.7, emerge thinks it's ok, too: ====== % emerge -pvt --depclean --with-bdeps=y --complete-graph python:2.7 Calculating dependencies... done! >>> Calculating removal order... >>> These are the packages that would be unmerged: dev-lang/python selected: 2.7.18_p16-r1 protected: none omitted: 3.9.16_p3 All selected packages: =dev-lang/python-2.7.18_p16-r1 ====== (The options "--with-bdeps=y" and "--complete-graph" don't make a difference here, the output is the same without them.) Initially I had portage-3.0.44-r1 when I encountered this problem. Then I upgraded to portage-3.0.46 (which is the latest portage package that I currently have in my /usr/portage), but the problem still occurs. Reproducible: Always Portage 3.0.46 (python 3.9.16-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-10, glibc-2.36-r7, 6.5.0-pf5 x86_64) ================================================================= System uname: Linux-6.5.0-pf5-x86_64-AMD_Ryzen_5_5600X_6-Core_Processor-with-glibc2.36 KiB Mem: 32781612 total, 687480 free KiB Swap: 6291452 total, 5502064 free sh bash 5.1_p16-r2 ld GNU ld (Gentoo 2.39 p5) 2.39.0 distcc 3.4 x86_64-pc-linux-gnu [disabled] app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r2::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.36.0-r2::gentoo dev-lang/python: 2.7.18_p16-r1::gentoo, 3.9.16_p3::gentoo dev-lang/rust: 1.66.1::gentoo dev-util/cmake: 3.25.3::gentoo dev-util/meson: 1.0.1::gentoo sys-apps/baselayout: 2.13-r1::gentoo sys-apps/openrc: 0.46::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.13-r7::gentoo, 2.69-r5::gentoo, 2.71-r5::gentoo sys-devel/automake: 1.13.4-r2::gentoo, 1.16.5::gentoo sys-devel/binutils: 2.35.2::gentoo, 2.39-r4::gentoo sys-devel/binutils-config: 5.4.1::gentoo sys-devel/clang: 15.0.7-r1::gentoo sys-devel/gcc: 9.3.0-r2::gentoo, 10.4.1_p20230119-r1::gentoo, 12.2.1_p20230121-r1::gentoo sys-devel/gcc-config: 2.8::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/llvm: 12.0.0::gentoo, 15.0.7::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 6.1::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r7::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 volatile: True sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: --ipv4 local location: /usr/local/portage masters: gentoo priority: 0 volatile: True mrueg location: /var/lib/layman/mrueg masters: gentoo priority: 50 volatile: True nix-guix location: /var/lib/layman/nix-guix masters: gentoo priority: 50 volatile: True ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -mtune=amdfam10 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/easy-rsa /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=nocona -mtune=amdfam10 -O2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--autounmask=n --autounmask-keep-masks=y" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://gentoo.mirror.web4u.cz/ http://mirror.yandex.ru/gentoo-distfiles http://mirrors.linuxant.fr/distfiles.gentoo.org/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" MAKEOPTS="-j6 -l8" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/zsh" USE="X a52 aac aacplus acl acpi activities alsa amd64 bluetooth branding btrfs bzip2 cairo cdda cdr cli crypt cryptsetup cups dbus declarative device-mapper dri dts dvd dvdr elogind encode examples exif faac fat ffmpeg flac fortran gdbm gif gpm gtk gui iconv icu iproute2 ipv6 jfs jpeg kde kwallet lcms libglvnd libnotify libtirpc lvm mad mmx mng mp3 mp4 mpeg mtp multilib ncurses nls nptl ntfs objc ogg opengl openmp pam pango parted pcre pdf plasma png policykit ppds pulseaudio python qml qt5 readline sdl seccomp semantic-desktop sound spell split-usr sqlite sse sse2 ssl startup-notification svg test-rust threads tiff truetype udev udisks unicode upnp upower usb v4l vorbis widgets wxwidgets x264 xattr xcb xfs xft xml xv xvid zlib zsh-completion" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" RUBY_TARGETS="ruby30" USERLAND="GNU" VIDEO_CARDS="fbdev vesa radeon nouveau intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
zac, can you take a look?
This kind of behavior is expected if a matching package is listed in /etc/portage/profile/package.provided, so is that the case?
(In reply to Zac Medico from comment #2) > This kind of behavior is expected if a matching package is listed in > /etc/portage/profile/package.provided, so is that the case? No, it's not. I don't even have /etc/portage/profile on that system.
It looks like build time dependencies have the ignored dep priority attribute set for depclean actions, and that's causing them to be dropped. I'll try to make a test case.
(In reply to Zac Medico from comment #4) > It looks like build time dependencies have the ignored dep priority > attribute set for depclean actions, and that's causing them to be dropped. > I'll try to make a test case. From looking at the code, it seems like it could be ignored, but this test case does not reproduce it: > # Copyright 2024 Gentoo Authors > # Distributed under the terms of the GNU General Public License v2 > > from portage.tests import TestCase > from portage.tests.resolver.ResolverPlayground import ( > ResolverPlayground, > ResolverPlaygroundTestCase, > ) > > > class DepcleanBdepsTestCase(TestCase): > def testDepcleanBdeps(self): > installed = { > "www-client/firefox-52.9.0-r1": { > "EAPI": "8", > "DEPEND": "dev-lang/python:2.7", > }, > "dev-lang/python-2.7.18_p16-r1": {"SLOT": "2.7"}, > } > > world = ("www-client/firefox",) > > test_cases = ( > ResolverPlaygroundTestCase( > [], > options={"--depclean": True}, > success=True, > cleanlist=[], > ), > ResolverPlaygroundTestCase( > [], > options={"--depclean": True, "--with-bdeps": "n"}, > success=True, > cleanlist=["dev-lang/python-2.7.18_p16-r1"], > ), > ) > > playground = ResolverPlayground(installed=installed, world=world, debug=True) > try: > for test_case in test_cases: > playground.run_TestCase(test_case) > self.assertEqual(test_case.test_success, True, test_case.fail_msg) > finally: > playground.debug = False > playground.cleanup()
(In reply to Zac Medico from comment #5) > From looking at the code, it seems like it could be ignored Looking again at the code, the ignored attribute is only True for --with-bdeps=n here, so it should not be ignored: depend_root, edepend["DEPEND"], self._priority( cross=self._cross(pkg.root), buildtime=True, optional=(pkg.built or ignore_depend_deps), ignored=ignore_depend_deps, ), The test case debug output shows it as a "soft" dependency rather than "ignored": (www-client/firefox-52.9.0-r1:0/0::test_repo, installed) depends on (dev-lang/python-2.7.18_p16-r1:2.7/2.7::test_repo, installed) (soft)
It must be because the "threads" USE flag was dropped here: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=742eeca02a0d06a56f6a69af55bcbcdf42f18ba1 commit 742eeca02a0d06a56f6a69af55bcbcdf42f18ba1 Author: Michał Górny <mgorny@gentoo.org> Date: 2021-08-31 10:22:13 +0200 dev-lang/python: Enable threads unconditionally in <3.7 too Signed-off-by: Michał Górny <mgorny@gentoo.org> Because of that, your installed dev-lang/python-2.7.18_p16-r1 does not match the firefox dependency. Since it's a DEPEND, it's build-time only and does not trigger any warning or error when it is unsatisfied.
The "threads" USE flag first became optional in firefox ebuilds with this commit from bug 679582: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ed2ace908b4d5dc23d0f3b873f6d5df867f513d5 commit ed2ace908b4d5dc23d0f3b873f6d5df867f513d5 Author: Thomas Deutschmann <whissi@gentoo.org> Date: 2019-03-08 21:33:03 +0100 mozcoreconf-v6.eclass: add still required Python 2.7 dependency In commit 9d973365a74e58731b62caae87d4ad521c90d947 and af9298355290622e1ab2ec5253550e6b8196722a Python 3 support was added and PYTHON_COMPAT was moved from eclass to ebuild as requested by QA. However, Python 2 dependency, which is still required, was lost. This commit will bring back Python 2 dependency. Note that we use python-any-r1_pkg_setup in moz_pkgsetup to initialize Python 3 via PYTHON_COMPAT but we will switch back to Python 2 manually using python_export later. Closes: https://bugs.gentoo.org/679582 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
Note in bug 249166 comment #6 the idea that "package manager should be warning or erroring" about flags not listed in IUSE. We could definitely trigger a warning here.
The dbapi _match_use method and match_from_list function are the obvious places to detect and warn about the missing IUSE, but they we don't have access to the parent package in these locations, so there's not enough information to emit a really useful warning message here. So, we should probably add a function to detect this case and trigger the warning when we encounter an unsatisfied dependency in the depgraph.
So, is the following the correct summary of the situation: emerge --depclean doesn't pick up firefox's dependence on python because this dependency (1) is a build-time dependency and (2) isn't fully satisfied by the installed python package because of missing USE="threads"? Or the second factor (missing USE) would solely trigger this behaviour, too?
(In reply to Pavel Goran from comment #11) > So, is the following the correct summary of the situation: emerge --depclean > doesn't pick up firefox's dependence on python because this dependency (1) > is a build-time dependency With the default --with-bdeps=y behavior of depclean, build-time dependencies have an "optional" attribute set to True, and optional dependencies are silently ignored if they are unsatisfied. > and (2) isn't fully satisfied by the installed > python package because of missing USE="threads"? Or the second factor > (missing USE) would solely trigger this behaviour, too? Depclean has a boolean allow_missing_deps flag which is True if there are package arguments and False if there are no package arguments. Regardless of the allow_missing_deps flag, optional build-time dependencies are silently ignored if they are unsatisfied, but runtime dependencies trigger an error like this if they are unsatisfied and the allow_missing_deps flag is False (no package arguments): > * Dependencies could not be completely resolved due to > * the following required packages not being installed: > * > * >=dev-lang/python-2.7.5-r2:2.7[ncurses,sqlite,ssl,threads] pulled in by: > * www-client/firefox-52.9.0-r1 > * > * Have you forgotten to do a complete update prior to depclean? The > * most comprehensive command for this purpose is as follows: > * > * emerge --update --newuse --deep --with-bdeps=y @world Regardless of depclean options, this error will never trigger for the firefox dependency on python because it is build-time only. We can fix it to trigger special behavior for build-time dependencies that are unsatisfied due to USE. We can make it either a warning or an error. We can treat it more severely when the flag is missing from IUSE, as noted in bug 249166 comment #6, since this case is exceptional.
