https://docs.python.org/3/library/nis.html This optional python module which is scheduled for removal in python 3.13, is conditionally compiled if libnsl / libtirpc are available at build time. The ebuild tries to disable it unconditionally, along with conditionally enabling gdbm, sqlite, ssl, ncurses, readline, tk. It does so with the following code run *after* configure: ``` for mod in "${disable_modules[@]}"; do echo "MODULE_${mod}_STATE=disabled" done >> Makefile || die ``` This is: - too late to affect configure-time search attempts - ineffective, because the Makefile doesn't check these variables when trying to compile said modules and install them As far as I know, the *only* reason these variables are created by the CPython Makefile is to create runtime lookup variables that can be used like this: ``` import sysconfig if sysconfig.get_config_var('MODULE__GDBM_STATE') == 'yes': print('sysconfig says the gdbm module was logged as built') ``` (Of course you can just import it and check that way...) A couple of modules are correctly disabled. gdbm is influenced by --with-dbmliborder="", and readline passes --without-readline. Reproducible: Always Steps to Reproduce: 1. Install python 3.9, 3.10, 3.11 2. Install po4a as a build dependency of another package, which drags in opensp and eventually libnsl 3. Install python 3.12 4. Depclean po4a Alternatively: ``` sudo USE="-sqlite -ncurses -ssl -tk" emerge -a python:3.12 ``` Actual Results: !!! existing preserved libs: >>> package: net-libs/libnsl-2.0.0-r1 * - /usr/lib64/libnsl.so.3 * - /usr/lib64/libnsl.so.3.0.0 * used by /usr/lib/python3.12/lib-dynload/nis.cpython-312-x86_64-linux-gnu.so (dev-lang/python-3.12.0_rc1_p6) Notice with some surprise that this issue only showed up for one version of python. Alternatively: Notice that the resulting installation has sqlite, ncurses, ssl, and tk built. (Specifically for tk, the C extension is built and installed, but the pure python interface is rm -r'ed based on the USE flag.) Expected Results: libnsl should have been cleanly removed, because the ebuild says it is explicitly telling the build system to not build nis. Alternatively, nis should be optional via a USE flag, and when compiled, it should be a dependency. Alternatively, sqlite, ncurses, ssl, and tk should not have been built or installed. Useless info: Portage 3.0.49 (python 3.11.5-final-0, default/linux/amd64/17.1/desktop/systemd/merged-usr, gcc-12, glibc-2.37-r3, 6.1.46-gentoo-dist x86_64) ================================================================= System uname: Linux-6.1.46-gentoo-dist-x86_64-AMD_Ryzen_7_4700U_with_Radeon_Graphics-with-glibc2.37 KiB Mem: 15737024 total, 4838204 free KiB Swap: 16777212 total, 15068924 free Timestamp of repository gentoo: Sun, 03 Sep 2023 23:30:01 +0000 Head commit of repository gentoo: fcb9439bc2e682fcb3a080dbed565fa4580aa9ba sh bash 5.1_p16-r6 ld GNU ld (Gentoo 2.40 p5) 2.40.0 app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r6::gentoo dev-java/java-config: 2.3.1-r1::gentoo dev-lang/perl: 5.38.0-r1::gentoo dev-lang/python: 2.7.18_p16-r1::gentoo, 3.8.18::gentoo, 3.9.18::gentoo, 3.10.13::gentoo, 3.11.5::gentoo, 3.12.0_rc1_p6::gentoo dev-lang/rust: 1.69.0-r1::gentoo dev-util/cmake: 3.26.5-r2::gentoo dev-util/meson: 1.2.1-r1::gentoo sys-apps/baselayout: 2.14::gentoo sys-apps/sandbox: 2.37::gentoo sys-apps/systemd: 253.6::gentoo sys-devel/autoconf: 2.13-r7::gentoo, 2.71-r6::gentoo sys-devel/automake: 1.16.5-r1::gentoo sys-devel/binutils: 2.40-r5::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/clang: 16.0.6::gentoo sys-devel/gcc: 12.3.1_p20230526::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/lld: 16.0.6::gentoo sys-devel/llvm: 14.0.6-r4::gentoo, 16.0.6::gentoo sys-devel/make: 4.4.1-r1::gentoo sys-kernel/linux-headers: 6.1::gentoo (virtual/os-headers) sys-libs/glibc: 2.37-r3::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 volatile: False sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes personal location: /var/db/repos/personal masters: gentoo volatile: False Binary Repositories: binhost priority: 9999 sync-uri: https://gentoo.osuosl.org/releases/amd64/binpackages/17.1/x86-64 Installed sets: @devstuff, @meson-testdeps ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/var/cache/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-march=native -O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance binpkg-request-signature buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles getbinpkg ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-march=native -O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="X a52 aac acl acpi alsa amd64 branding bzip2 cairo cdda cdr cli crypt cups dbus dri dts dvd dvdr encode exif flac fortran gdbm gif gpm gtk gui iconv icu ipv6 jpeg lcms libnotify libtirpc mad mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds qt5 readline sdl seccomp sound spell ssl startup-notification svg systemd test-rust tiff truetype udev udisks unicode upower usb vorbis vulkan wxwidgets x264 xattr xcb xft xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31" VIDEO_CARDS="amdgpu radeon radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, MAKEOPTS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
From doing more spelunking, I have noticed that: - in python 3.11, `MODULE_*_STATE` was used by setup.py as a transition while moving stuff into the Makefile, to check whether setup.py was responsible to build it or perform e.g. pkg-config lookups - in previous versions of python, gentoo had a gentoo-specific patch to update setup.py's `DISABLED_MODULE_LIST = []` from an environment variable, hence why setting $PYTHON_DISABLE_MODULES before running emake did something -- and why I couldn't find it in the CPython sources This just goes to show that there never really was a way to control automagic dependencies, and gentoo didn't rely on one either -- but did patch one in. It lacked comment docs ;) and more importantly does not work with 3.12 because the transition is over and the Makefile is no longer used. A new hack is needed, which I am currently testing.
(In reply to Eli Schwartz from comment #1) > [...] > This just goes to show that there never really was a way to control > automagic dependencies, and gentoo didn't rely on one either -- but did > patch one in. I dug them out as I was curious too: * <3.7: https://gitweb.gentoo.org/proj/python-gentoo-patches.git/tree/patches/02_all_disable_modules_and_ssl.patch?h=3.6 * >=3.7: https://gitweb.gentoo.org/fork/cpython.git/commit/?h=gentoo&id=cc91802c4c8639c4b0962728319dd5f76ade21b1 etc (we switched from a separate patch repo)
Created attachment 869745 [details, diff] dev-lang/python: fix 3.12 support for USE-disabled modules Here is a patch tested in a native compile (but not a cross compile! someone should please verify that). The log starts off with: ``` checking for LIBNSL... yes checking for rpc/rpc.h... yes checking for LIBSQLITE3... yes checking for sqlite3.h... yes checking for sqlite3_bind_double in -lsqlite3... yes checking for sqlite3_column_decltype in -lsqlite3... yes checking for sqlite3_column_double in -lsqlite3... yes checking for sqlite3_complete in -lsqlite3... yes checking for sqlite3_progress_handler in -lsqlite3... yes checking for sqlite3_result_double in -lsqlite3... yes checking for sqlite3_set_authorizer in -lsqlite3... yes checking for sqlite3_trace_v2 in -lsqlite3... yes checking for sqlite3_value_double in -lsqlite3... yes checking for sqlite3_load_extension in -lsqlite3... yes checking for sqlite3_serialize in -lsqlite3... yes checking for --enable-loadable-sqlite-extensions... yes checking for TCLTK... yes checking for gdbm.h... yes checking for gdbm_open in -lgdbm... yes checking for ndbm.h... no checking for ndbm presence and linker args... () checking for gdbm/ndbm.h... yes checking for gdbm-ndbm.h... no checking for library containing dbm_open... -lgdbm_compat checking for db.h... no checking for --with-dbmliborder... checking for _dbm module CFLAGS and LIBS... ... checking for curses.h... yes checking for ncurses.h... yes checking for CURSES... yes checking curses module flags... ncursesw (CFLAGS: -D_DEFAULT_SOURCE -I/usr/include/ncursesw, LIBS: -lncursesw -ltinfow ) checking for panel.h... yes checking for PANEL... yes checking panel flags... panelw (CFLAGS: -I/usr/include/ncursesw -D_DEFAULT_SOURCE , LIBS: -lpanelw ) checking for term.h... yes checking whether mvwdelch is an expression... yes checking whether WINDOW has _flags... yes checking for curses function is_pad... yes checking for curses function is_term_resized... yes checking for curses function resize_term... yes checking for curses function resizeterm... yes checking for curses function immedok... yes checking for curses function syncok... yes checking for curses function wchgat... yes checking for curses function filter... yes checking for curses function has_key... yes checking for curses function typeahead... yes checking for curses function use_env... yes ... checking whether compiling and linking against OpenSSL works... yes checking for --with-openssl-rpath... checking whether OpenSSL provides required ssl module APIs... yes checking whether OpenSSL provides required hashlib module APIs... yes checking for --with-ssl-default-suites... python checking for --with-builtin-hashlib-hashes... md5,sha1,sha2,sha3,blake2 ... configure: creating Modules/Setup.local configure: creating Makefile maksetup: 'nis' was handled by previous rule. maksetup: '_curses' was handled by previous rule. maksetup: '_curses_panel' was handled by previous rule. maksetup: '_sqlite3' was handled by previous rule. maksetup: '_ssl' was handled by previous rule. maksetup: '_hashlib' was handled by previous rule. maksetup: '_tkinter' was handled by previous rule. configure: If you want a release build with all stable optimizations active (PGO, etc), please run ./configure --enable-optimizations >>> Source configured. ``` and ends off with: ``` The following modules are *disabled* in configure script: _curses _curses_panel _dbm _gdbm _hashlib _sqlite3 _ssl _tkinter nis readline Checked 111 modules (31 built-in, 69 shared, 1 n/a on linux-x86_64, 10 disabled, 0 missing, 0 failed on import) >>> Source compiled. ```
Created attachment 869746 [details, diff] optional followup patch to use the same approach for consistency on older pythons
Comment on attachment 869745 [details, diff] dev-lang/python: fix 3.12 support for USE-disabled modules >+ # Force-disable modules we don't want built. >+ # See Modules/Setup for docs on how this works. Setup.local contains our local deviations. >+ { >+ echo "*disabled*" >+ echo "nis" >+ use gdbm || echo "_gdbm _dbm" >+ use sqlite || echo "_sqlite3" >+ use ssl || echo "_hashlib _ssl" >+ use ncurses || echo "_curses _curses_panel" >+ use readline || echo "readline" >+ use tk || echo "_tkinter" >+ >+ } > Modules/Setup.local Use `cat > Modules/Setup.local <<-EOF || die` approach here, with $(usex) inside.
(In reply to Michał Górny from comment #5) > Use `cat > Modules/Setup.local <<-EOF || die` approach here, with $(usex) > inside. I assume the thing that matters here is the die. I think the grouped commands is more elegant than cat with command substitution, and it's easy to do this: ``` { echo "foo" echo "bar" } > outfile || die
The Gentoo style tends to be to prefer explicit comsubs, so deviating from that is confusing, even if it's technically equivalent. You want to, at a glance, be able to know it's checking USE flags.
Created attachment 870058 [details, diff] v2: dev-lang/python: fix 3.12 support for USE-disabled modules
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=654f15afd89cab6ce21931c71491c962de67bdfe commit 654f15afd89cab6ce21931c71491c962de67bdfe Author: Eli Schwartz <eschwartz93@gmail.com> AuthorDate: 2023-09-04 18:15:19 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-09-07 22:26:12 +0000 dev-lang/python: fix 3.12 support for USE-disabled modules python's build system doesn't provide straightforward support for forcing modules to not be built. A couple different approaches have been taken throughout the years, sometimes relying on gentoo-specific patches, which are all broken in python 3.12 as it doesn't use distutils and a setup.py anymore. Adopt a new approach based on semi-documented internals of the CPython build system. Closes: https://bugs.gentoo.org/913596 Signed-off-by: Eli Schwartz <eschwartz93@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.12.0_rc2_p1-r1.ebuild | 528 +++++++++++++++++++++++++ 1 file changed, 528 insertions(+)
Created attachment 870060 [details, diff] optional followup patch to use the same approach for consistency on older pythons rerolled the followup patch with the same changes
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4fefeac023536fdbbc50e7cced091b2ab80013bd commit 4fefeac023536fdbbc50e7cced091b2ab80013bd Author: Eli Schwartz <eschwartz93@gmail.com> AuthorDate: 2023-09-04 18:53:03 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-09-07 22:58:42 +0000 dev-lang/python: port older versions to use the same USE disable approach Followup to commit 654f15afd89cab6ce21931c71491c962de67bdfea. It should produce the exact same effective package. [sam: Chuck in revbump to be safe, especially given these older branches tend to only get updates for sec fixes which we want to stable quickly, I'd rather find out about any problems now than later when they're bumped.] Bug: https://bugs.gentoo.org/913596 Signed-off-by: Eli Schwartz <eschwartz93@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.10.13-r1.ebuild | 500 +++++++++++++++++++++++++++++ dev-lang/python/python-3.11.5-r1.ebuild | 526 +++++++++++++++++++++++++++++++ dev-lang/python/python-3.8.18-r1.ebuild | 430 +++++++++++++++++++++++++ dev-lang/python/python-3.9.18-r1.ebuild | 490 ++++++++++++++++++++++++++++ 4 files changed, 1946 insertions(+)
