Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 933553 - dev-python/setuptools-rust: can break setuptools: ModuleNotFoundError: No module named 'semantic_version'
Summary: dev-python/setuptools-rust: can break setuptools: ModuleNotFoundError: No mo...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
: 935149 (view as bug list)
Depends on: 934357
Blocks:
  Show dependency tree
 
Reported: 2024-06-04 23:20 UTC by anna
Modified: 2024-06-29 19:06 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description anna 2024-06-04 23:20:01 UTC
I was having a lot of difficulty getting dev-python/requests to emerge after a profile update and setting the new python version, but I found this thread https://forums.gentoo.org/viewtopic-t-1169236-start-0.html which mentioned the fix was to update dev-python/setuptools-rust first, so even though I was on the version mentioned in this thread I gave that a try, which seemed to rebuild the same version (1.9.0) but with the correct/updated python_targets flags set, then it built fine

Reproducible: Always

Steps to Reproduce:
 - Have requests in your emerge graph when updating after setting a new python version
 - Have it fail to emerge with the following error:
 ModuleNotFoundError: No module named 'semantic_version' 
 - Do a singular emerge with dev-python/setuptools-rust
 - Now requests should work fine
Actual Results:  
The dependencies should be set such so that setuptools-rust is updated first, since it seems necessary for requests to update


Portage 3.0.65 (python 3.12.3-final-0, default/linux/amd64/23.0/split-usr/desktop/plasma, gcc-13, glibc-2.39-r7, 6.9.2-gentoo x86_64)
=================================================================
System uname: Linux-6.9.2-gentoo-x86_64-AMD_Ryzen_9_7950X_16-Core_Processor-with-glibc2.39
KiB Mem:    31954864 total,    276160 free
KiB Swap:  134217724 total, 123878396 free
Timestamp of repository gentoo: Tue, 04 Jun 2024 18:30:00 +0000
Head commit of repository gentoo: 2df892ccf725e9fcfdd7033df6d9cfa8bfe25de6
Head commit of repository claytabase: 834bbfee97f02454383b27dfc16fd67eb568fca2

Timestamp of repository gogooverlay: Thu, 23 May 2024 11:18:47 +0000
Head commit of repository gogooverlay: b68708e58fce772605271b0475d79cd0b217d940

Timestamp of repository guru: Tue, 04 Jun 2024 16:18:49 +0000
Head commit of repository guru: d7d668ebc961d5a80ace6871fa97b7d16666853d

Timestamp of repository hering-overlay: Thu, 23 May 2024 09:48:39 +0000
Head commit of repository hering-overlay: 1d418fcdd0e9ea296fedf7125239f328bf7ff117

Timestamp of repository menelkir: Wed, 22 May 2024 04:18:49 +0000
Head commit of repository menelkir: 1ed74031e68244b42f8ee79099a21e9cd50f8144

Head commit of repository plex-overlay: 7fafa3bdbb97903931ddc1c07895c0fcf24e2573

Timestamp of repository robert7k: Tue, 04 Jun 2024 08:04:02 +0000
Head commit of repository robert7k: f7a46d297b212d5faabc9f8975d6ebe7de55b820

Timestamp of repository steam-overlay: Mon, 20 May 2024 17:23:25 +0000
Head commit of repository steam-overlay: 3aa8d3201d7bce32a1a0fffdf4afd9dbd0583048

Timestamp of repository torbrowser: Mon, 20 May 2024 17:23:26 +0000
Head commit of repository torbrowser: e864531f479e040a6516e2c34b630f187aee4338

Timestamp of repository usenet-overlay: Mon, 20 May 2024 17:23:36 +0000
Head commit of repository usenet-overlay: 0dd5e4d7816e23954528c3bbf737edc5d936b4fd

Head commit of repository brother-overlay: ee347f6e4c8f2eb4efcc8ca57f3494b4f34aedbb

sh bash 5.2_p26-r3
ld GNU ld (Gentoo 2.42 p3) 2.42.0
ccache version 4.10 [disabled]
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p26-r3::gentoo
dev-build/autoconf:        2.13-r8::gentoo, 2.72-r1::gentoo
dev-build/automake:        1.16.5-r2::gentoo
dev-build/cmake:           3.29.3::gentoo
dev-build/libtool:         2.4.7-r4::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.4.1::gentoo
dev-java/java-config:      2.3.3-r1::gentoo
dev-lang/perl:             5.38.2-r6::gentoo
dev-lang/python:           3.10.14_p1-r1::gentoo, 3.11.9-r1::gentoo, 3.12.3-r1::gentoo, 3.13.0_beta1_p3::gentoo
dev-lang/rust:             1.78.0::gentoo
dev-util/ccache:           4.10-r1::gentoo
sys-apps/baselayout:       2.15::gentoo
sys-apps/openrc:           0.54::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-devel/binutils:        2.42-r1::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           16.0.6::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-devel/gcc:             13.2.1_p20240503::gentoo, 14.1.1_p20240518::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/lld:             16.0.6::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-devel/llvm:            16.0.6::gentoo, 17.0.6::gentoo, 18.1.6::gentoo
sys-kernel/linux-headers:  6.9::gentoo (virtual/os-headers)
sys-libs/glibc:            2.39-r7::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    volatile: False
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts: 
    sync-rsync-verify-metamanifest: yes

XaetaCore
    location: /home/anna/xaetacore
    masters: gentoo
    volatile: True

claytabase
    location: /var/db/repos/claytabase
    sync-type: git
    sync-uri: https://github.com/claybie/claytabase.git
    masters: gentoo
    volatile: False

gogooverlay
    location: /var/db/repos/gogooverlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gogooverlay.git
    masters: gentoo
    volatile: False

guru
    location: /var/db/repos/guru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/guru.git
    masters: gentoo
    volatile: False

hering-overlay
    location: /var/db/repos/hering-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/hering-overlay.git
    masters: gentoo
    volatile: False

menelkir
    location: /var/db/repos/menelkir
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/menelkir.git
    masters: gentoo
    volatile: False

plex-overlay
    location: /var/db/repos/plex-overlay
    sync-type: git
    sync-uri: https://github.com/comio/plex-overlay
    masters: gentoo
    volatile: False

robert7k
    location: /var/db/repos/robert7k
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/robert7k.git
    masters: gentoo
    volatile: False

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/steam-overlay.git
    masters: gentoo
    volatile: False

torbrowser
    location: /var/db/repos/torbrowser
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/torbrowser.git
    masters: gentoo
    volatile: False

usenet-overlay
    location: /var/db/repos/usenet-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/usenet-overlay.git
    masters: gentoo
    volatile: False

brother-overlay
    location: /usr/local/portage/brother-overlay
    sync-type: git
    sync-uri: https://github.com/stefan-langenmaier/brother-overlay.git
    masters: gentoo
    priority: 9999
    volatile: True

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=znver3"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=znver3"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe -march=znver3"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=znver3"
GENTOO_MIRRORS="http://mirrors.rit.edu/gentoo/ http://gentoo-mirror.flux.utah.edu/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs"
LEX="flex"
MAKEOPTS="-j32"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RUSTFLAGS="-C target-cpu=native"
SHELL="/bin/bash"
USE="X a52 aac acl acpi activities alsa amd64 bluetooth branding bzip2 cairo cdda cdr cet crypt cups dbus declarative dri dts dvd dvdr elogind encode exif flac gdbm gif gpm gtk gui iconv icu ipv6 jpeg kde kf6compat kwallet lcms libnotify libtirpc mad mng mp3 mp4 mpeg multilib ncurses networkmanager nls nvidia ogg opengl openmp pam pango pcre pdf pipewire plasma png policykit ppds pulseaudio qml qt5 qt6 readline screencast sdl seccomp semantic-desktop sound spell split-usr ssl startup-notification svg test-rust tiff truetype udev udisks unicode upower usb vorbis vulkan wayland widgets wxwidgets x264 xattr xcb xft xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 aes avx avx2 avx512bw avx512cd avx512dq avx512f avx512vbmi avx512vl f16c fma3 pclmul popcnt rdrand sha sse3 sse4_1 sse4_2 sse4a ssse3" CURL_SSL="nss" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_12" PYTHON_TARGETS="python3_12" QEMU_SOFTMMU_TARGETS="x86_64" QEMU_USER_TARGETS="x86_64" RUBY_TARGETS="ruby31 ruby32" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2024-06-05 05:01:22 UTC
dev-python/requests does not depend on dev-python/setuptools-rust.

I have no clue why you'd get setuptools-rust rebuilt (?) before semantic-version but that looks like a bug in Portage.
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-05 05:04:56 UTC
Anna, I appreciate this is an ask, but if nobody can identify where the missing dependency is (if any), then it's a Portage bug -- and it's way easier to debug and fix those with a clean reproducer from a stage3.

Would you be willing to try grab a stage3 (from before the py3.12 migration, I don't think stage3s have been rebuilt yet) and give a set of steps which result in failure in it?
Comment 3 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-05 05:13:53 UTC
Also as a middle step, it might be enough if you can show me bad emerge -p ...  output from scroll back (the merge list is what's important)
Comment 4 Eli Schwartz gentoo-dev 2024-06-07 04:02:58 UTC
In #gentoo-python


17:13 <ztrawhcse> sam_: I reproduced the issue
17:13 <ztrawhcse> process:
17:13 <ztrawhcse> - have 3.11 system
17:13 <ztrawhcse> - run update to 3.12 with -p
17:14 <ztrawhcse> - check for some package in between semantic-version and setuptools-rust
17:14 <ztrawhcse> - add python_targets_python3_11 to that package
17:15 <ztrawhcse> then, run the update. semantic-version will be updated to 3.12-only, setuptools-rust will still be installed for 3.11 only, and you cannot build *other* packages for 3.11 until setuptools-rust gets updated too (to drop the broken 3.11 impl)



Example merge list:


[binary   R    ] dev-lang/python-exec-conf-2.4.6-2  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary     U  ] dev-python/ensurepip-setuptools-70.0.0-1 [69.5.1]
[ebuild     U  ] app-shells/gentoo-bashcomp-20240511 [20230313]
[ebuild     U  ] app-shells/bash-completion-2.14.0 [2.11]
[binary   R    ] sys-libs/libseccomp-2.5.5-r1-5  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] net-misc/rsync-3.3.0-2  PYTHON_SINGLE_TARGET="python3_12* -python3_11*"
[binary   R    ] app-misc/pax-utils-1.3.7-3  PYTHON_SINGLE_TARGET="python3_12* -python3_11*"
[binary   R    ] sys-apps/file-5.45-r4-7  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] app-crypt/gpgme-1.23.2-6  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] dev-build/meson-1.4.0-r1-2  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] dev-build/meson-format-array-0-3  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] dev-libs/libxml2-2.12.6-6  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] dev-python/certifi-3021.3.16-r4-3  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary   R    ] dev-python/charset-normalizer-3.3.2-3  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary   R    ] dev-python/idna-3.7-2  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary   R    ] dev-python/PySocks-1.7.1-r2-3  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary   R    ] dev-python/cython-3.0.10-3  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild   R    ] dev-python/installer-0.7.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[binary   R    ] dev-python/markupsafe-2.1.5-4  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary   R    ] dev-python/semantic-version-2.10.0-4  PYTHON_TARGETS="python3_12* -python3_11*"
[ebuild   R    ] dev-python/gpep517-16  PYTHON_TARGETS="python3_12*"
[binary   R    ] dev-libs/libxslt-1.1.39-4  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] dev-python/urllib3-2.2.1-4  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[binary     U  ] dev-python/jinja-3.1.4-2 [3.1.3] PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[ebuild  N     ] dev-python/flit-core-3.9.0  USE="-test" PYTHON_TARGETS="python3_11 python3_12 (-pypy3) -python3_10 (-python3_13)"
[binary   R    ] dev-python/requests-2.32.1-2  PYTHON_TARGETS="python3_12* -python3_11*"
[ebuild   R    ] dev-python/packaging-24.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild   R    ] dev-python/more-itertools-10.2.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[binary   R    ] app-portage/gemato-20.5-4  PYTHON_TARGETS="python3_12* -python3_11* (-python3_13)"
[ebuild   R    ] dev-python/ordered-set-4.1.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild     U  ] dev-python/platformdirs-4.2.2 [4.2.1] PYTHON_TARGETS="python3_12*"
[ebuild   R    ] dev-python/jaraco-context-5.3.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild   R    ] dev-python/wheel-0.43.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild   R    ] dev-python/jaraco-functools-4.0.1  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild   R    ] dev-python/jaraco-text-3.12.0  PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild     U  ] dev-python/setuptools-70.0.0 [69.5.1-r1] PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild     U  ] dev-python/setuptools-scm-8.1.0 [8.0.4] PYTHON_TARGETS="python3_12* (-python3_13)"
[ebuild     U  ] dev-python/lxml-5.2.2 [5.2.1] USE="examples*" PYTHON_TARGETS="python3_12* (-python3_13)"
[binary   R    ] dev-python/setuptools-rust-1.9.0-4  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] sys-apps/util-linux-2.39.3-r7-4  PYTHON_TARGETS="python3_12* -python3_11*"
[ebuild     U  ] sys-apps/systemd-255.7 [255.4] PYTHON_SINGLE_TARGET="python3_12* -python3_11*"
[binary   R    ] sys-apps/portage-3.0.63-r1-2  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] app-portage/gentoolkit-0.6.5-3  PYTHON_TARGETS="python3_12* -python3_11*"
[binary   R    ] dev-vcs/git-2.44.2-3  PYTHON_SINGLE_TARGET="python3_12* -python3_11*"



It failed trying to build dev-python/setuptools for 3.11 -- semantic_version was rebuilt for 3.11 -> 3.12, setuptools-rust was scheduled for later, this broke the 3.11 build of any package.

It is, once again, plugin autoloading. It's the setuptools-scm case but worse, since simply ensuring the entire package is merged won't help, and the PDEPEND hack isn't workable here.
Comment 5 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-07 08:02:29 UTC
Options so far, all of which feel bad:
1) setuptools PDEPENDing on all of its plugins;
2) source changes as Eli has suggested to e.g. setuptools-rust to defer loading semantic_version;
3) bundling all setuptools plugins inside the setuptools ebuild;
4) patch+maybe upstream a change to setuptools to warn instead on ImportError during plugin loading

Longer-term:
5) implement "tight groups", a variant of bug 918873 	

mgorny & zac: ideas?
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2024-06-07 12:20:56 UTC
Sam, do you have energy to try discussing this with setuptools upstream?  I think they may understand the problem and agree with making plugin errors nonfatal.
Comment 7 Eli Schwartz gentoo-dev 2024-06-07 12:30:57 UTC
That would be fantastic.

Unfortunately I fear that they will be extremely accommodating and understanding about the fact that it is a problem and the actual holdup will be that they cannot decide how to fix it without breaking backwards compatibility and then nothing ever changes.
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-07 13:07:04 UTC
Yes, let me try.
Comment 9 Zac Medico gentoo-dev 2024-06-08 15:26:21 UTC
(In reply to Sam James from comment #5)
> Options so far, all of which feel bad:
> 1) setuptools PDEPENDing on all of its plugins;

Yeah, since the plugin loading errors are fatal, it's certainly valid to list the plugins in PDEPEND. We can use a meta package like x11-base/xorg-drivers.
Comment 10 Zac Medico gentoo-dev 2024-06-08 23:38:20 UTC
(In reply to Eli Schwartz from comment #7)
> That would be fantastic.
> 
> Unfortunately I fear that they will be extremely accommodating and
> understanding about the fact that it is a problem and the actual holdup will
> be that they cannot decide how to fix it without breaking backwards
> compatibility and then nothing ever changes.

I can understand why they wouldn't want to ignore plugin loading errors, but maybe they would be willing to skip loading of plugins that are not explicitly specified as dependencies.

I did some poking around and found that the PDEPEND relationship arises from [setuptools.finalize_distribution_options] ocurring in files like /usr/lib/python3.12/site-packages/setuptools_rust-1.9.0.dist-info/entry_points.txt, and apparently setuptools loads these entry points even for plugins that are not explicitly specified as dependencies.
Comment 11 Zac Medico gentoo-dev 2024-06-08 23:45:59 UTC
(In reply to Zac Medico from comment #10)

Also note that if setuptools plugin loading were conditional on dependencies then it would have solved the setuptools plugin loading issues that triggered bug 663324, since those dependencies would prevent emerge from scheduling installation of setuptools plugins underneath jobs that depend on them.
Comment 12 Eli Schwartz gentoo-dev 2024-06-09 02:42:16 UTC
(In reply to Zac Medico from comment #9)
> (In reply to Sam James from comment #5)
> > Options so far, all of which feel bad:
> > 1) setuptools PDEPENDing on all of its plugins;
> 
> Yeah, since the plugin loading errors are fatal, it's certainly valid to
> list the plugins in PDEPEND. We can use a meta package like
> x11-base/xorg-drivers.

In this case we would end up dragging in rust as a build + install dependency for said plugin, and I very much doubt that is desirable behavior for the ecosystem. ;) It would be especially bad for wd40 profiles as it would prevent installing portage itself...

(In reply to Zac Medico from comment #10)
> I can understand why they wouldn't want to ignore plugin loading errors, but
> maybe they would be willing to skip loading of plugins that are not
> explicitly specified as dependencies.
> 
> I did some poking around and found that the PDEPEND relationship arises from
> [setuptools.finalize_distribution_options] ocurring in files like
> /usr/lib/python3.12/site-packages/setuptools_rust-1.9.0.dist-info/
> entry_points.txt, and apparently setuptools loads these entry points even
> for plugins that are not explicitly specified as dependencies.

Yes, the entrypoints schema is one of the few sources of metadata the python community has been able to agree on somehow and they have a fondness for using it for everything.

Unfortunately, this is a "feature" because historically they wanted to be able to do things like setuptools-scm modifying how unrelated packages include files into an sdist -- because setuptools-scm adds all VCS files -- without actually requiring the other package's agreement.

They may or may not still want that sort of flexibility, given the broader movement of the python community towards only ever touching source code inside a clean virtualenv with explicit build dependencies only. It would make a certain amount of sense to ditch the idea given that the move towards virtualenvs was in no small part motivated by the unpredictable unreliability of setuptools plugins installed for one package, infecting another package.

On the other hand, they may worry that packages need to both specify their plugins as dependencies, and request they be loaded, and say that it's our fault for not building in virtualenvs (disregarding that virtualenvs require breaking the network sandbox).
Comment 13 Zac Medico gentoo-dev 2024-06-09 05:16:18 UTC
(In reply to Eli Schwartz from comment #12)
> (In reply to Zac Medico from comment #9)
> > (In reply to Sam James from comment #5)
> > > Options so far, all of which feel bad:
> > > 1) setuptools PDEPENDing on all of its plugins;
> > 
> > Yeah, since the plugin loading errors are fatal, it's certainly valid to
> > list the plugins in PDEPEND. We can use a meta package like
> > x11-base/xorg-drivers.
> 
> In this case we would end up dragging in rust as a build + install
> dependency for said plugin, and I very much doubt that is desirable behavior
> for the ecosystem. ;) It would be especially bad for wd40 profiles as it
> would prevent installing portage itself...

We should be able to use use.mask or package.use.mask in the wd40 profile to toggle the dependency, no? If there's no rust then we should toggle off the dependency on setuptools-rust, yes?
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-09 18:28:07 UTC
I've filed https://github.com/pypa/setuptools/issues/4417.
Comment 15 Eli Schwartz gentoo-dev 2024-06-10 01:22:45 UTC
Submitted my workaround as a PR to setuptools-rust.
Comment 16 Larry the Git Cow gentoo-dev 2024-06-11 14:29:06 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=50031116133a3a8b300f0473196e7703b374f8f9

commit 50031116133a3a8b300f0473196e7703b374f8f9
Author:     Eli Schwartz <eschwartz93@gmail.com>
AuthorDate: 2024-06-11 14:08:25 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-06-11 14:25:47 +0000

    dev-python/setuptools-rust: backport upstream patch to unbreak complex merge
    
    When rebuilding world with python 3.12 it's very easy to end up with a
    merge order that results in building setuptools-rust's dependencies for
    3.12 only, but not yet merging setuptools-rust. This then breaks all
    other packages relying on setuptools, due to plugin autoloading.
    
    Fix this by making setuptools-rust only depend on other packages when it
    is actually used, but not in the APIs invoked by the plugin autoloading.
    This means that the package only has to be fully merged in time for
    other packages that have an explicit BDEPEND on it, which is fine as
    that is already the case.
    
    Bug: https://github.com/PyO3/setuptools-rust/pull/437
    Closes: https://bugs.gentoo.org/933553
    Signed-off-by: Eli Schwartz <eschwartz93@gmail.com>
    Signed-off-by: Sam James <sam@gentoo.org>

 ...tools-rust-1.9.0-delay-non-stdlib-imports.patch | 156 ++++++++++++++++
 .../setuptools-rust-1.9.0-r1.ebuild                | 198 +++++++++++++++++++++
 2 files changed, 354 insertions(+)
Comment 17 Larry the Git Cow gentoo-dev 2024-06-25 01:32:31 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ef2c9d2102b3c06f1c346097e4dd85cdf19b1d59

commit ef2c9d2102b3c06f1c346097e4dd85cdf19b1d59
Author:     Eli Schwartz <eschwartz93@gmail.com>
AuthorDate: 2024-06-23 04:50:18 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-06-25 01:31:25 +0000

    dev-python/setuptools: add strong blocker for old versions of setuptools-rust
    
    This version is no longer packaged in the tree. It contains a runtime dependency
    on typing_extensions, and triggers pathological dependency graph bugs
    due to its nature as a setuptools plugin. The issue trivially manifests
    when changing PYTHON_TARGETS.
    
    The ONLY way to fix this is to update/uninstall setuptools-rust before
    recompiling setuptools itself. Uninstalling is reasonably cheap, since
    the package is only ever used as a bdep for other packages. This also
    means it may end up not getting upgraded in a timely fashion.
    
    Strong blockers are a strong tool to use here, but:
    - it is an old version so panicking and refusing to do anything until it
      gets manually removed may be warranted
    - it does in fact prevent even successfully running src_compile() for
      setuptools itself, if setuptools-rust is in partially-merged state due
      to typing_extensions being queued before setuptools and
      setuptools-rust being queued *after* setuptools
    
    This "fixes" world updates by telling people exactly which package to
    manually remove rather than inscrutably failing at compile time with
    python tracebacks and leaving them to guess which package to manually
    remove. It could be wished that users would be aided through the power
    of automatically resolved dependency graphs, but there's no good options
    here and at least the error message is clear.
    
    No revbump, because existing packages that successfully compiled don't
    have the issue and it only occurs during certain world updates.
    
    Bug: https://bugs.gentoo.org/933553
    Signed-off-by: Eli Schwartz <eschwartz93@gmail.com>
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-python/setuptools/setuptools-69.0.3-r1.ebuild | 1 +
 dev-python/setuptools/setuptools-69.5.1-r1.ebuild | 1 +
 dev-python/setuptools/setuptools-70.0.0.ebuild    | 1 +
 3 files changed, 3 insertions(+)
Comment 18 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-29 19:06:03 UTC
*** Bug 935149 has been marked as a duplicate of this bug. ***