Last night I upgraded from python-3.10.10_p3 to python-3.10.11, and now any script that imports the "ssl" module is broken, showing following error: vala ~ # python Python 3.10.11 (main, May 14 2023, 08:19:54) [GCC 12.2.1 20230428] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import ssl Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.10/ssl.py", line 99, in <module> import _ssl # if we can't import it, let the error propagate SystemError: module _ssl uses unknown slot ID 1735421541 Incidentally, I also have python-3.11.3 installed (last night, updated from python-3.11.2_p2 to python-3.11.3), and its ssl module works: vala /var/log/portage # cd vala ~ # python3.11 -c "import ssl" vala ~ # echo $? 0 Ordinarily, I would just switch to python3.11 via python-exec.conf and try to rebuild python3.10, but unfortunately, one of the programs affected by this is the almighty "emerge", and I cannot seem to find a way to make it use python3.11 instead of 3.10, probably because portage is only built for 3.10. So now I have a broken system and do not know what to do. Reproducible: Didn't try Steps to Reproduce: Install python-3.10.11 Actual Results: "import ssl" doesn't work Expected Results: "import ssl" should work vala ~ # emerge --info Traceback (most recent call last): File "/usr/lib/python-exec/python3.10/emerge", line 46, in <module> import portage File "/usr/lib/python3.10/site-packages/portage/__init__.py", line 12, in <module> import asyncio File "/usr/lib/python3.10/asyncio/__init__.py", line 8, in <module> from .base_events import * File "/usr/lib/python3.10/asyncio/base_events.py", line 34, in <module> import ssl File "/usr/lib/python3.10/ssl.py", line 99, in <module> import _ssl # if we can't import it, let the error propagate SystemError: module _ssl uses unknown slot ID 1735421541 Oops :( I'm ready to provide necessary info upon request.
I'm heading off for a bit but hopefully someone else can take a look. But first thought is: anything in /usr/local at all? (Run find on it if you can and share the output). I'll check the diff between those versions as well later if nobody beats me to it.
My /usr/local is almost empty, only containing a package tree sync script that I use in a nightly cron job. vala /usr/local # find /usr/local /usr/local /usr/local/sbin /usr/local/sbin/.keep /usr/local/sbin/portage-update /usr/local/.keep /usr/local/share /usr/local/share/doc /usr/local/share/doc/.keep /usr/local/share/.keep /usr/local/share/man /usr/local/share/man/.keep /usr/local/share/man/whatis /usr/local/lib /usr/local/lib/.keep /usr/local/src /usr/local/src/.keep /usr/local/games /usr/local/games/.keep /usr/local/man /usr/local/bin /usr/local/bin/.keep vala /usr/local # cat /usr/local/sbin/portage-update #!/bin/sh emerge --color n --nospinner --sync >/dev/null && eix-update -q && emerge --color n --nospinner -Dupv world vala /usr/local # I have no idea what the "whatis" file is, but it is and empty, unused file: vala /usr/local # ls -l /usr/local/share/man/whatis -rw-r--r-- 1 root root 0 Oct 18 2019 /usr/local/share/man/whatis vala /usr/local # which whatis /usr/bin/whatis vala /usr/local #
Oh, I seem to have made a mistake in the "Hardware" field of this bug, as I forgot that this system is running x86, not x86_64. Correcting now, with apologies.
Looking further at Python internals, I guess "_ssl" refers to file /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-i386-linux-gnu.so, which is an ELF library: vala /var/log/portage # file /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-i386-linux-gnu.so /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-i386-linux-gnu.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped So it might be interesting to add that during the last night's emerge run, gcc was among the packages upgraded (from gcc-12.2.1_p20230121-r1 to gcc-12.2.1_p20230428-r1), and I am fairly certain that it was emerged before python, so the current python3.10 I have was built using this newer gcc. Should I attach build logs from python and/or gcc? They are each several megabytes large, so I do not want to do that unless it helps.
Build log from Python 3.10.11 may help. Also please paste the output of: strace python3.10 -c 'import _ssl' |& grep _ssl Do you have PYTHONPATH set?
Created attachment 861671 [details] dev-lang:python-3.10.11:20230514-061729.log.gz Compressed build log attached. Strace output: vala ~ # strace python3.10 -c 'import _ssl' |& grep _ssl execve("/usr/bin/python3.10", ["python3.10", "-c", "import _ssl"], 0xbfc6e098 /* 22 vars */) = 0 statx(AT_FDCWD, "/usr/lib/python3.10/lib-dynload/_ssl.cpython-310-i386-linux-gnu.so", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=175200, ...}) = 0 openat(AT_FDCWD, "/usr/lib/python3.10/lib-dynload/_ssl.cpython-310-i386-linux-gnu.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3 write(2, "SystemError: module _ssl uses un"..., 57SystemError: module _ssl uses unknown slot ID 1735421541 vala ~ #
I do not have PYTHONPATH set.
Hmm, I don't see anything suspicous. Could you try loading some other of the .so modules from Python (like _sha256, _hashlib, _socket...) and see if any of them exhibits the same problem? Did you try rebuilding Python 3.10 again?
Some of them work vala ~ # python3.10 Python 3.10.11 (main, May 14 2023, 08:19:54) [GCC 12.2.1 20230428] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import _sha256 >>> import _socket >>> But _hashlib even causes a segfault: vala ~ # gdb python3.10 GNU gdb (Gentoo 12.1 vanilla) 12.1 Copyright (C) 2022 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python3.10... Reading symbols from /usr/lib/debug//usr/bin/python3.10.debug... (No debugging symbols found in /usr/lib/debug//usr/bin/python3.10.debug) (gdb) run Starting program: /usr/bin/python3.10 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/libthread_db.so.1". Python 3.10.11 (main, May 14 2023, 08:19:54) [GCC 12.2.1 20230428] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import _hashlib Program received signal SIGSEGV, Segmentation fault. 0xb7fd8860 in _dl_relocate_object () from /lib/ld-linux.so.2 (gdb) bt #0 0xb7fd8860 in _dl_relocate_object () from /lib/ld-linux.so.2 #1 0xb7fd68de in dl_open_worker_begin () from /lib/ld-linux.so.2 #2 0xb7962d59 in _dl_catch_exception () from /lib/libc.so.6 #3 0xb7fd5e20 in dl_open_worker () from /lib/ld-linux.so.2 #4 0xb7962d59 in _dl_catch_exception () from /lib/libc.so.6 #5 0xb7fd6258 in _dl_open () from /lib/ld-linux.so.2 #6 0xb787f9b8 in dlopen_doit () from /lib/libc.so.6 #7 0xb7962d59 in _dl_catch_exception () from /lib/libc.so.6 #8 0xb7962e22 in _dl_catch_error () from /lib/libc.so.6 #9 0xb7fe39bd in _rtld_catch_error () from /lib/ld-linux.so.2 #10 0xb787f457 in _dlerror_run () from /lib/libc.so.6 #11 0xb787fa7f in dlopen () from /lib/libc.so.6 #12 0xb7db1ea6 in _PyImport_FindSharedFuncptr () from /usr/lib/libpython3.10.so.1.0 #13 0xb7d7bbc0 in _PyImport_LoadDynamicModuleWithSpec () from /usr/lib/libpython3.10.so.1.0 #14 0xb7d78ae4 in _imp_create_dynamic () from /usr/lib/libpython3.10.so.1.0 #15 0xb7cc29f2 in cfunction_vectorcall_FASTCALL () from /usr/lib/libpython3.10.so.1.0 #16 0xb7c80f8a in PyVectorcall_Call () from /usr/lib/libpython3.10.so.1.0 #17 0xb7c810e1 in _PyObject_Call () from /usr/lib/libpython3.10.so.1.0 #18 0xb7c81150 in PyObject_Call () from /usr/lib/libpython3.10.so.1.0 #19 0xb7d52f5e in do_call_core () from /usr/lib/libpython3.10.so.1.0 #20 0xb7c38208 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #21 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #22 0xb7c811e9 in _PyFunction_Vectorcall () from /usr/lib/libpython3.10.so.1.0 #23 0xb7c3ba21 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #24 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #25 0xb7c811e9 in _PyFunction_Vectorcall () from /usr/lib/libpython3.10.so.1.0 #26 0xb7c399e7 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #27 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #28 0xb7c811e9 in _PyFunction_Vectorcall () from /usr/lib/libpython3.10.so.1.0 #29 0xb7c3a456 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #30 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #31 0xb7c811e9 in _PyFunction_Vectorcall () from /usr/lib/libpython3.10.so.1.0 #32 0xb7c3a456 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #33 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #34 0xb7c811e9 in _PyFunction_Vectorcall () from /usr/lib/libpython3.10.so.1.0 #35 0xb7c3a456 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #36 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #37 0xb7c811e9 in _PyFunction_Vectorcall () from /usr/lib/libpython3.10.so.1.0 #38 0xb7c819e1 in object_vacall () from /usr/lib/libpython3.10.so.1.0 #39 0xb7c81cd7 in _PyObject_CallMethodIdObjArgs () from /usr/lib/libpython3.10.so.1.0 #40 0xb7d7a90c in PyImport_ImportModuleLevelObject () from /usr/lib/libpython3.10.so.1.0 #41 0xb7d5089f in import_name () from /usr/lib/libpython3.10.so.1.0 #42 0xb7c37726 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.10.so.1.0 #43 0xb7d54e1d in _PyEval_Vector () from /usr/lib/libpython3.10.so.1.0 #44 0xb7d54fc2 in PyEval_EvalCode () from /usr/lib/libpython3.10.so.1.0 #45 0xb7d943c7 in run_eval_code_obj () from /usr/lib/libpython3.10.so.1.0 #46 0xb7d9461b in run_mod () from /usr/lib/libpython3.10.so.1.0 #47 0xb7d95881 in PyRun_InteractiveOneObjectEx () from /usr/lib/libpython3.10.so.1.0 #48 0xb7d96585 in _PyRun_InteractiveLoopObject () from /usr/lib/libpython3.10.so.1.0 #49 0xb7d96f05 in _PyRun_AnyFileObject () from /usr/lib/libpython3.10.so.1.0 #50 0xb7d96fcf in PyRun_AnyFileExFlags () from /usr/lib/libpython3.10.so.1.0 #51 0xb7db57c6 in Py_RunMain () from /usr/lib/libpython3.10.so.1.0 #52 0xb7db5b90 in pymain_main () from /usr/lib/libpython3.10.so.1.0 #53 0xb7db5c76 in Py_BytesMain () from /usr/lib/libpython3.10.so.1.0 #54 0x00401088 in main () (gdb) Unfortunately, I cannot rebuild python, because I cannot get emerge to work. The only way would be to build it manually (the build log should have all the patches and the exact configure/make commands), and I'm not sure if emerge could even be pointed to use such built python in a different install prefix.
Can't you start emerge using: python3.11 /usr/bin/emerge ?
Alas, no: vala ~ # python3.11 /usr/bin/emerge /usr/lib/python-exec/python3.11/emerge: this Python implementation (python3.11) is not supported by the script.
Ah. Clone the Portage git repo and run it via python3.11 from the git checkout then ;-).
Oh, that's a neat trick, thanks! Python 3.10 rebuild finished OK, and now all seems fixed: vala ~ # python Python 3.10.11 (main, May 14 2023, 18:00:53) [GCC 12.2.1 20230428] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import _ssl >>> import _hashlib >>> import _sha256 >>> import _socket >>> vala ~ # emerge --info Portage 3.0.44 (python 3.10.11-final-0, default/linux/x86/17.0/hardened, gcc-12, glibc-2.36-r7, 6.1.19-gentoo i686) ================================================================= System uname: Linux-6.1.19-gentoo-i686-Intel_Core_i7_9xx_-Nehalem_Class_Core_i7-with-glibc2.36 KiB Mem: 2053892 total, 427304 free KiB Swap: 2097148 total, 1665704 free Timestamp of repository gentoo: Sun, 14 May 2023 01:15:01 +0000 Head commit of repository gentoo: 380c42e72b56c19de92e11ef6197cbaa23861524 sh bash 5.1_p16-r2 ld GNU ld (Gentoo 2.39 p6) 2.39.0 app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r2::gentoo dev-lang/perl: 5.36.0-r2::gentoo dev-lang/python: 3.10.11::gentoo, 3.11.3::gentoo dev-util/cmake: 3.25.2::gentoo dev-util/meson: 1.0.1::gentoo sys-apps/baselayout: 2.13-r1::gentoo sys-apps/openrc: 0.46::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.71-r5::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.39-r5::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/gcc: 12.2.1_p20230428-r1::gentoo sys-devel/gcc-config: 2.10::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/make: 4.4.1::gentoo sys-kernel/linux-headers: 6.1::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r7::gentoo Repositories: gentoo location: /usr/gentoo/portage sync-type: rsync sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage priority: -1000 volatile: True sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="@FREE" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=core2 -mtune=pentium -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.1/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=core2 -mtune=pentium -O2 -pipe" DISTDIR="/usr/gentoo/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -march=i686 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -march=i686 -pipe" GENTOO_MIRRORS="http://gentoo.wheel.sk/ http://tux.rainside.sk/gentoo/ http://gd.tuwien.ac.at/opsys/linux/gentoo/" LC_ALL="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" MAKEOPTS="-j2" PKGDIR="/usr/gentoo/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="acpi bash-completion bzip2 cli crypt curl dri fortran gdbm gmp gnutls hardened icecastflac iconv idn imap ipv6 libglvnd libtirpc logrotate lvm mp3 ncurses nntp nptl nptlonly ntp oggvorbis open_perms openmp openssl pam pcre peer_perms perl pic pie python readline seccomp sieve smp split-usr ssl sslog ssp test-rust type1 ubac unconfined unicode usb vim-syntax x86 xattr xtpax zlib" ABI_X86="32" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 ssse3" CURL_SSL="gnutls" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en sk" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11 python3_10" RUBY_TARGETS="ruby30" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LANG, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS I have no idea what caused this problem. Seems like something that happens when RAM is going bad, but this is a VM, and running memtest86 on the host does not show any errors... I saved the faulty python build via quickpkg before the rebuild, so in case someone wants to take a look at the built files, they're available.
If there's a possibility of memory corruption, that is some relief. I was afraid we're dealing with some race condition that sometimes results in broken builds. I'm afraid I don't have time nor skills to figure out what went wrong. It might be worthwhile to check what "slot ID" was supposed to be there, and e.g. figure out if it might have been corrupted via some bit flip or perhaps the file was built with slot ID from another Python version.
Digging into the Python module system, I notice that the error mentions "slot ID" not "slot value". There are only two defined slot IDs: (from moduleobject.h) #define Py_mod_create 1 #define Py_mod_exec 2 The slot ID from the error (1735421541) looks pretty random, almost like an uninitialized 32bit variable. It does NOT look like a random bit flip - there would have been several bits flipped to get such value from the initially correct value 1 or 2. The question is still how this could have happened. I'm currently running repeated rebuilds of the python package hoping to reproduce this, but so far I got nothing. If there is a bug, it could be in any one of the ELF module files, so not fun!
