0:06:18 load avg: 21.72 [390/427] test_urllib2_localnet passed 0:06:18 load avg: 21.72 [391/427] test_urlparse passed 0:06:19 load avg: 21.58 [392/427] test_utf8source passed 0:06:19 load avg: 21.58 [393/427] test_uu passed 0:06:20 load avg: 21.58 [394/427] test_uuid passed usage: [-h] [-u {uuid1,uuid3,uuid4,uuid5}] [-n NAMESPACE] [-N NAME] : error: Incorrect number of arguments. uuid3 requires a namespace and a name. Run 'python -m uuid -h' for more information. usage: [-h] [-u {uuid1,uuid3,uuid4,uuid5}] [-n NAMESPACE] [-N NAME] : error: Incorrect number of arguments. uuid3 requires a namespace and a name. Run 'python -m uuid -h' for more information. ------------------------------------------------------------------- This is an unstable amd64 chroot image at a tinderbox (==build bot) name: 17.1_no_multilib_systemd-j5-20230303-130045 ------------------------------------------------------------------- gcc-config -l: [1] x86_64-pc-linux-gnu-10 [2] x86_64-pc-linux-gnu-12 [3] x86_64-pc-linux-gnu-13 * clang/llvm (if any): clang version 15.0.7 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/lib/llvm/15/bin Configuration file: /etc/clang/clang.cfg /usr/lib/llvm/15 15.0.7 Python 3.10.10 Available Ruby profiles: [1] ruby27 (with Rubygems) [2] ruby30 (with Rubygems) * Available Rust versions: [1] rust-1.67.1 * The following VMs are available for generation-2: 1) IcedTea JDK 3.16.0 [icedtea-bin-8] 2) Eclipse Temurin JDK 11.0.18_p10 [openjdk-bin-11] *) Eclipse Temurin JDK 17.0.6_p10 [openjdk-bin-17] 4) Eclipse Temurin JDK 8.362_p09 [openjdk-bin-8] Available Java Virtual Machines: [1] icedtea-bin-8 [2] openjdk-bin-8 [3] openjdk-bin-11 [4] openjdk-bin-17 system-vm The Glorious Glasgow Haskell Compilation System, version 9.0.2 php cli (if any): HEAD of ::gentoo commit 2fb5d4ce35806654fe42b770b69fd758b070fb28 Author: Repository mirror & CI <repomirrorci@gentoo.org> Date: Wed Mar 8 05:32:02 2023 +0000 2023-03-08 05:32:02 UTC emerge -qpvO dev-lang/python [ebuild U ] dev-lang/python-3.12.0_alpha6 [3.12.0_alpha5] USE="bluetooth gdbm libedit ncurses pgo readline sqlite ssl tk verify-sig -build -ensurepip -examples -hardened -lto -test (-valgrind)"
Created attachment 856865 [details] emerge-info.txt
Created attachment 856866 [details] dev-lang:python-3.12.0_alpha6:20230308-155545.log
Created attachment 856867 [details] emerge-history.txt.bz2
Created attachment 856868 [details] etc.portage.tar.bz2
BTW I do jet -j5 but this seems to be ignored : └─timeout,19437 --signal=15 --kill-after=5m 48h bash -c emerge --update dev-lang/python └─emerge,19439 /usr/lib/python-exec/python3.10/emerge --update dev-lang/python ├─python3.10,28059 /usr/lib/portage/python3.10/pid-ns-init 7943 │ └─python3.10,28073 /usr/lib/portage/python3.10/pid-ns-init 250 250 250 18 0,1,2 /usr/bin/sandbox [dev-lang/python-3.12.0_alpha6] sandbox /usr/lib/portage/python │ └─sandbox,28082,portage /usr/lib/portage/python3.10/ebuild.sh compile │ └─bash,28083 /usr/lib/portage/python3.10/ebuild.sh compile │ └─bash,28105 /usr/lib/portage/python3.10/ebuild.sh compile │ └─bash,28111 /usr/lib/portage/python3.10/ebuild-helpers/emake CPPFLAGS= CFLAGS= LDFLAGS= │ └─make,28116 -j5 CPPFLAGS= CFLAGS= LDFLAGS= │ └─make,2065 run_profile_task │ └─sh,2191 -e -c LD_LIBRARY_PATH=/var/tmp/portage/dev-lang/python-3.12.0_alpha6/work/Python-3.12.0a6 ./python -m test -j5 --pgo-exten │ └─python,2195 -m test -j5 --pgo-extended -x test_gdb -u-network -x test_asyncio -x test_httpservers -x test_logging -x test_mult │ └─python,18733 -u -m test.regrtest --worker-args [{"testdir": null, "verbose": 0, "quiet": false, "exclude": true, "single": │ └─make,17992 -C /var/tmp/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_9c0w8vzc/tmp7e65emhe/python-build -j21
emerge --info dev-lang/python --verbose=n Portage 3.0.45.2 (python 3.10.10-final-0, default/linux/amd64/17.1/no-multilib/systemd, gcc-13, glibc-2.36-r7, 6.1.15 x86_64) ================================================================= System Settings ================================================================= System uname: Linux-6.1.15-x86_64-AMD_Ryzen_9_5950X_16-Core_Processor-with-glibc2.36 KiB Mem: 131830352 total, 23583568 free KiB Swap: 268435448 total, 268371960 free Timestamp of repository gentoo: Wed, 08 Mar 2023 05:32:02 +0000 Head commit of repository gentoo: 2fb5d4ce35806654fe42b770b69fd758b070fb28 sh bash 5.2_p15-r2 ld GNU ld (Gentoo 2.40 p2) 2.40.0 ccache version 4.7.4 [disabled] app-misc/pax-utils: 1.3.7::gentoo app-shells/bash: 5.2_p15-r2::gentoo dev-java/java-config: 2.3.1::gentoo dev-lang/perl: 5.36.0-r2::gentoo dev-lang/python: 3.10.10_p2::gentoo, 3.11.2_p1::gentoo, 3.12.0_alpha5::gentoo dev-lang/rust: 1.67.1::gentoo dev-util/ccache: 4.7.4::gentoo dev-util/cmake: 3.25.2::gentoo dev-util/meson: 1.0.1::gentoo sys-apps/baselayout: 2.13-r1::gentoo sys-apps/sandbox: 2.30-r1::gentoo sys-apps/systemd: 253.1::gentoo sys-devel/autoconf: 2.13-r7::gentoo, 2.71-r5::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.40-r2::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/clang: 14.0.6-r1::gentoo, 15.0.7-r1::gentoo sys-devel/gcc: 12.2.1_p20230304::gentoo, 13.0.1_pre20230305::gentoo sys-devel/gcc-config: 2.10::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/lld: 14.0.6-r1::gentoo, 15.0.7::gentoo sys-devel/llvm: 14.0.6-r2::gentoo, 15.0.7::gentoo sys-devel/make: 4.4.1::gentoo sys-kernel/linux-headers: 6.2::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r7::gentoo sys-libs/libselinux: 3.4::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: https://github.com/gentoo-mirror/gentoo.git priority: -1000 volatile: False ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-pipe -march=native -fno-diagnostics-color -O2" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /etc/grs/systems.conf /etc/stunnel/stunnel.conf /usr/share/gnupg/qualified.txt /usr/share/qpsmtpd/plugins /usr/share/themes/oxygen-gtk/gtk-2.0 /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-pipe -march=native -fno-diagnostics-color -O2" DISTDIR="/var/cache/distfiles" EMERGE_DEFAULT_OPTS="--deep --verbose --verbose-conflicts --nospinner --quiet-build --tree --color=n --ask=n" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-pipe -march=native -fno-diagnostics-color -O2" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-pipe -march=native -fno-diagnostics-color -O2" GENTOO_MIRRORS="https://ftp.fau.de/gentoo https://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ https://mirror.netcologne.de/gentoo/ http://ftp.halifax.rwth-aachen.de/gentoo/" LANG="C.UTF8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0" LEX="flex" MAKEOPTS=" -j5" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="accessibility acl afs amd64 berkdb bzip2 cairo calendar cddb clamav cli connman crypt css cups dbi dri dvb eds fastcgi fortran freewnn gdbm geoip graphviz imagemagick imlib infiniband inotify iodbc ipv6 jack jumbo-build kde latex libcaca libglvnd libtirpc lm-sensors lua m17n-lib memcached motif mpeg mpi ncurses nptl offensive opengl openmp orc osc pam pdf posix ppds pulseaudio readline scanner screencast seccomp session slang sndfile sockets split-usr ssl systemd tcpd test-rust tk truetype udev unicode v4l verify-sig wavpack wifi wmf wxwidgets xattr xcb xmlrpc xvid zip zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 aes avx avx2 f16c fma3 pclmul popcnt rdrand sha sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="cs da de-DE en-CA en-US hi iu km kn mi mni nl prs rw sd su sw-TZ uk" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby27 ruby30" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel dummy" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS ================================================================= Package Settings ================================================================= dev-lang/python-3.10.10_p2::gentoo was built with the following: USE="bluetooth gdbm libedit ncurses pgo readline sqlite ssl tk verify-sig xml -build -ensurepip -examples -hardened -lto -test (-valgrind)" CFLAGS="-pipe -march=native -fno-diagnostics-color -O2 -fwrapv" CXXFLAGS="-pipe -march=native -fno-diagnostics-color -O2 -fwrapv" FEATURES="sfperms unmerge-logs buildpkg-live strict userpriv usersync parallel-fetch network-sandbox config-protect-if-modified preserve-libs ebuild-locks xattr binpkg-logs merge-sync binpkg-dostrip usersandbox binpkg-docompress userfetch binpkg-multi-instance ipc-sandbox unmerge-orphans unknown-features-warn distlocks assume-digests sandbox multilib-strict pid-sandbox protect-owned fixlafiles qa-unresolved-soname-deps" dev-lang/python-3.11.2_p1::gentoo was built with the following: USE="bluetooth gdbm libedit ncurses pgo readline sqlite ssl tk verify-sig -build -ensurepip -examples -hardened -lto -test (-valgrind)" CFLAGS="-pipe -march=native -fno-diagnostics-color -O2 -fwrapv" CXXFLAGS="-pipe -march=native -fno-diagnostics-color -O2 -fwrapv" FEATURES="sfperms unmerge-logs buildpkg-live strict userpriv usersync parallel-fetch network-sandbox config-protect-if-modified preserve-libs ebuild-locks xattr binpkg-logs merge-sync binpkg-dostrip usersandbox binpkg-docompress userfetch binpkg-multi-instance ipc-sandbox unmerge-orphans unknown-features-warn distlocks assume-digests sandbox multilib-strict pid-sandbox protect-owned fixlafiles qa-unresolved-soname-deps" dev-lang/python-3.12.0_alpha5::gentoo was built with the following: USE="bluetooth gdbm libedit ncurses pgo readline sqlite ssl tk verify-sig -build -ensurepip -examples -hardened -lto -test (-valgrind)" CFLAGS="-pipe -march=native -fno-diagnostics-color -O2 -fwrapv" CXXFLAGS="-pipe -march=native -fno-diagnostics-color -O2 -fwrapv" FEATURES="binpkg-logs preserve-libs strict xattr unmerge-logs fixlafiles sfperms binpkg-multi-instance merge-sync parallel-fetch sandbox usersandbox binpkg-docompress unmerge-orphans protect-owned distlocks network-sandbox userfetch ebuild-locks ipc-sandbox assume-digests userpriv unknown-features-warn config-protect-if-modified buildpkg-live multilib-strict usersync binpkg-dostrip qa-unresolved-soname-deps pid-sandbox"
Ah, so it happens during the profiling phase, not the actual test phase. How often can you reproduce it? Can you also reproduce it with USE=-pgo when running src_test()?
(In reply to Michał Górny from comment #7) > Ah, so it happens during the profiling phase, not the actual test phase. > > How often can you reproduce it? > > Can you also reproduce it with USE=-pgo when running src_test()? Adding "-pgo" to /etc/.../*.use/python let "ebuild ... compile" run fine within 2 min. Without that line emerge stucks at 425. BTW src_test) - this is independent from FEATURES=test, right ?
and -j5 is ignored for [420/427] and obove
Without USE=pgo, tests are run only during src_test() (i.e. FEATURES=test). With USE=pgo, they are additionally run during src_compile() to provide profiling data.
FWIW I do now run "ebuild -...digest" within that image and it hangs here: 0:06:30 load avg: 21.21 [426/427] test_venv passed
and starts to span more and more x86_64-pc-linux(4037)───cc1(4038)
0:06:30 load avg: 21.21 [426/427] test_venv passed ^Csandbox:stop caught signal 2 in pid 4 Kill <TestWorkerProcess #5 running test=test_tools pid=12690 time=1 hour 21 min> process group 17-1-no-multilib-systemd-j5-20230303-130045 /var/db/repos/gentoo/dev-lang/python #
(In reply to Toralf Förster from comment #11) > FWIW I do now run "ebuild -...digest" within that image and it hangs here: > > 0:06:30 load avg: 21.21 [426/427] test_venv passed Similar here: 0:05:00 load avg: 6.81 [425/427] test_venv passed 0:08:11 load avg: 3.41 [426/427] test_compile passed [hang] top shows two pythons running under emerge running at 100% CPU. One is almsto 2 hours of CPU time now, and still hung.
I confirm USE="-pgo" allows the package to build and install.
(In reply to Gary E. Miller from comment #14) > (In reply to Toralf Förster from comment #11) > > FWIW I do now run "ebuild -...digest" within that image and it hangs here: > > > > 0:06:30 load avg: 21.21 [426/427] test_venv passed > > Similar here: > > 0:05:00 load avg: 6.81 [425/427] test_venv passed > 0:08:11 load avg: 3.41 [426/427] test_compile passed > [hang] > > top shows two pythons running under emerge running at 100% CPU. One is > almsto 2 hours of CPU time now, and still hung. which let me wonder if there's a missing flush() in the emerge call of "ebuild ...compile" -or- whether it is now really 426# and not 425#.
(In reply to Toralf Förster from comment #16) > (In reply to Gary E. Miller from comment #14) > > (In reply to Toralf Förster from comment #11) > > > FWIW I do now run "ebuild -...digest" within that image and it hangs here: > > > > > > 0:06:30 load avg: 21.21 [426/427] test_venv passed > > > > Similar here: > > > > 0:05:00 load avg: 6.81 [425/427] test_venv passed > > 0:08:11 load avg: 3.41 [426/427] test_compile passed > > [hang] > > > > top shows two pythons running under emerge running at 100% CPU. One is > > almsto 2 hours of CPU time now, and still hung. > > which let me wonder if there's a missing flush() in the emerge call of > "ebuild ...compile" -or- whether it is now really 426# and not 425#. Or 427, or after 427.
I haven't seen this issue on 3.12.0_alpha5: # qlop -Hv python | grep alpha 2023-02-14T14:47:22 >>> dev-lang/python-3.12.0_alpha5: 1 minute, 29 seconds 2023-02-14T15:03:33 >>> dev-lang/python-3.12.0_alpha5: 4 minutes, 12 seconds 2023-02-14T18:20:03 >>> dev-lang/python-3.12.0_alpha5: 6 minutes, 27 seconds 2023-03-08T19:25:42 >>> dev-lang/python-3.12.0_alpha6: 36 minutes, 49 seconds 2023-03-11T17:25:59 >>> dev-lang/python-3.12.0_alpha6: 58 minutes, 58 seconds (the alpha6 builds were finished by killing the profiling which let the ebuild to finish merging successfully). I've restored the already removed alpha5 ebuild and built it again (to make sure that the hang isn't caused by something in system which was changed at the same time as alpha6 upgrade) and alpha5 still doesn't get stuck: 2023-03-12T11:55:59 >>> dev-lang/python-3.12.0_alpha5: 9 minutes, 25 seconds The alpha5 and alpha6 ebuild are identical except the filename, so it's probably something introduced in upstream, if someone wants to bisect that: cpython $ git log --oneline v3.12.0a5..v3.12.0a6 | wc -l 235 I'll save build.log from both to see if I spot some interesting difference (some tests failed to build in alpha5 - not fatal for the build, but maybe if they fixed the test in alpha6 then it might be the one getting stuck or something like that).
I've noticed that gentoo patches are a bit different between a5 and a6: https://github.com/python/cpython/issues/100221 already merged in a6 https://github.com/python/cpython/pull/100328/ added for a6, but that doesn't seem to cause it (I've rebuilt alpha5 ebuild with alpha6 patches applied and it didn't get stuck). from build.log diff I can see: in alpha5 the test_tools test fails immediately: ``` 0:01:28 load avg: 19.69 [424/427] test_tools failed (1 error) test test_tools failed creating the script to be frozen at /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/app.py copying the source tree into /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython... configuring python in /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/python-build... building python in /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/python-build... installing python into /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/python-installation... freezing /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/app.py... CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw']' returned non-zero exit status 2. --- STDOUT --- make[2]: Entering directory '/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw' gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c config.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c frozen.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M___future__.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M___main__.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__aix_support.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__collections_abc.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__compat_pickle.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__compression.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__osx_support.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__py_abc.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__pydecimal.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__sitebuiltins.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__strptime.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__threading_local.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M__weakrefset.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M_abc.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M_argparse.c gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -IObjects -IInclude -IPython -I. -I/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include -c M_ast.c make[2]: Leaving directory '/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw' --- STDERR --- In file included from config.c:18: /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include/Python.h:12:10: fatal error: pyconfig.h: No such file or directory 12 | #include "pyconfig.h" | ^~~~~~~~~~~~ compilation terminated. In file included from frozen.c:279: /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha5/temp/test_python_6mu7qr8z/tmp8l4okaqw/cpython/Include/Python.h:12:10: fatal error: pyconfig.h: No such file or directory 12 | #include "pyconfig.h" | ^~~~~~~~~~~~ compilation terminated. make[2]: *** [Makefile:390: config.o] Error 1 make[2]: *** Waiting for unfinished jobs.... make[2]: *** [Makefile:392: frozen.o] Error 1 ---- END ---- 0:01:35 load avg: 18.65 [425/427] test_multiprocessing_forkserver passed 0:02:16 load avg: 17.73 [426/427] test_multiprocessing_spawn passed 0:02:27 load avg: 15.38 [427/427] test_concurrent_futures passed ``` while alpha6 shows an error and then continues for 100+ minutes even with using -j42 (from 64 on my builder): ``` 1:44:19 load avg: 1.61 [427/427] test_tools failed (1 error) creating the script to be frozen at /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/app.py copying the source tree into /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/cpython... configuring python in /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-build... CalledProcessError: Command '['/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/cpython/python', '-c', 'import sysconfig; print(sysconfig.get_config_var("CONFIG_ARGS"))']' returned non-zero exit status 1. --- STDOUT --- --- STDERR --- Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/cpython/Lib/sysconfig.py", line 740, in get_config_var return get_config_vars().get(name) ^^^^^^^^^^^^^^^^^ File "/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/cpython/Lib/sysconfig.py", line 723, in get_config_vars _init_config_vars() File "/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/cpython/Lib/sysconfig.py", line 670, in _init_config_vars _init_posix(_CONFIG_VARS) File "/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/cpython/Lib/sysconfig.py", line 536, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named '_sysconfigdata__linux_x86_64-linux-gnu' ---- END ---- building python parallel='-j42' in /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-build... installing python into /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-installation... CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-build', 'install']' returned non-zero exit status 2. --- STDOUT --- make[2]: Entering directory '/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-build' Creating directory /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-installation/bin Rebuilding with profile guided optimizations: Creating directory /tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-installation/include rm -f profile-clean-stamp make[2]: Leaving directory '/tmp/tmpfs/portage/dev-lang/python-3.12.0_alpha6/temp/test_python_fx7r8c03/tmplr4kzl3i/python-build' test test_tools failed --- STDERR --- /usr/bin/install: cannot remove '/usr/lib64/libpython3.12.so.1.0': Permission denied make[2]: *** [Makefile:2031: altbininstall] Error 1 make[2]: *** Waiting for unfinished jobs.... ---- END ---- Total duration: 1 hour 44 min Tests result: FAILURE ``` Suspicious commits from: https://github.com/python/cpython/compare/v3.12.0a5...v3.12.0a6 https://github.com/python/cpython/commit/c3a178398c199038f3a0891d09f0363ec73f3b38 https://github.com/python/cpython/commit/dfc2e065a2e71011017077e549cd2f9bf4944c54 https://github.com/python/cpython/commit/1d194235e4d5981b5fea25c75318d61189103a58
I can confirm that reverting https://github.com/python/cpython/commit/c3a178398c199038f3a0891d09f0363ec73f3b38 makes test_tools to fail quickly again like it was in alpha5.
Thanks for chasing that down Martin.
If you try build Python from source manually, outside of Portage, can someone hitting this bug reproduce it?
I have observed that it does not hang completely, it just takes a day or so to complete.
(In reply to Jeremy Stent from comment #23) > I have observed that it does not hang completely, it just takes a day or so > to complete. I was not paying enough attention the first time. It runs until it is out of disk space (apparently recursively trying to profile itself), and then it slowly recovers.
Same for dev-lang/python-3.12.0_alpha7
I've commented at https://github.com/python/cpython/pull/102152#issuecomment-1498140151. I think I can reproduce this on one machine now too.
I still have pgo enabled, and when I was upgrading from 3.12.0_alpha6 to 3.12.0_alpha7 yesterday it didn't get stuck and finished in reasonable 9 minutes: # qlop -Htv python | tail 2023-02-27T10:14:31 >>> dev-lang/python-3.10.10_p2: 5 minutes, 17 seconds 2023-03-08T19:25:42 >>> dev-lang/python-3.12.0_alpha6: 36 minutes, 49 seconds 2023-03-11T13:30:27 >>> dev-lang/python-3.11.2_p1: 14 minutes, 52 seconds 2023-03-11T13:47:56 >>> dev-lang/python-3.10.10_p2: 12 minutes, 6 seconds 2023-03-11T17:25:59 >>> dev-lang/python-3.12.0_alpha6: 58 minutes, 58 seconds 2023-03-12T11:55:59 >>> dev-lang/python-3.12.0_alpha5: 9 minutes, 25 seconds 2023-03-15T12:40:16 >>> dev-lang/python-3.11.2_p2: 5 minutes, 35 seconds 2023-03-15T12:45:51 >>> dev-lang/python-3.10.10_p3: 4 minutes, 49 seconds 2023-04-05T10:33:15 >>> dev-lang/python-3.12.0_alpha7: 8 minutes, 59 seconds 2023-04-06T00:54:06 >>> dev-lang/python-3.10.11: 4 minutes, 57 seconds but then I wasn't paying attention to python-3.11.3 update and that got stuck for at least 2,5 hours before I've noticed and killed it.
Thanks for your comment on the PR - I get it (more) now, sorry for misunderstanding. Okay, given hanging like this is really not acceptable, I'll skip this test for now, and then try investigate a bit more.
Agreed, I've added "-x test_tools" in profile_task_flags in python-3.11.3.ebuild and it finished also reasonably fast: 2023-04-06T01:23:11 >>> dev-lang/python-3.11.3: 9 minutes, 19 seconds In upstream source I've also noticed: configure.ac: PROFILE_TASK='-m test --pgo --timeout=$(TESTTIMEOUT)' maybe we can use --timeout as well? It seems available or enabled since 3.10: Makefile.pre.in:TESTTIMEOUT= 1200 Makefile.pre.in: $(TESTRUNNER) -j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) Misc/NEWS.d/3.10.0a5.rst:Pass ``--timeout=$(TESTTIMEOUT)`` option to the default profile task
And with "--timeout=1200" instead of "-x test_tools" it gets correctly killed after 20 minutes: 0:02:30 load avg: 11.79 [426/427] test_concurrent_futures passed 0:20:14 load avg: 53.02 [427/427] test_tools crashed (Exit code 1) Timeout (0:20:00)! Thread 0x00007fc60044b740 (most recent call first): File "/tmp/tmpfs/portage/dev-lang/python-3.11.3/work/Python-3.11.3/Lib/selectors.py", line 415 in select File "/tmp/tmpfs/portage/dev-lang/python-3.11.3/work/Python-3.11.3/Lib/subprocess.py", line 2075 in _communicate ... File "/tmp/tmpfs/portage/dev-lang/python-3.11.3/work/Python-3.11.3/Lib/test/regrtest.py", line 43 in _main File "/tmp/tmpfs/portage/dev-lang/python-3.11.3/work/Python-3.11.3/Lib/test/regrtest.py", line 47 in <module> File "<frozen runpy>", line 88 in _run_code File "<frozen runpy>", line 198 in _run_module_as_main Total duration: 20 min 14 sec Tests result: FAILURE so the build took about 20 minutes more: 2023-04-06T01:36:18 >>> dev-lang/python-3.11.3: 28 minutes, 17 seconds I think the best solution is to add both "-x test_tools" and "--timeout=1200" to keep the builds "fast" and prevent some future test getting crazy.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=33f6f605fb2bb432134103d3de13d8ebe9f5b146 commit 33f6f605fb2bb432134103d3de13d8ebe9f5b146 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-04-05 23:45:18 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-04-06 00:31:46 +0000 dev-lang/python: skip test_tools for PGO Skip the 'test_tools' test when running PGO for now to avoid hanging the build (and also kind of fork-bombing the system with recursive cpython build attempts). Interestingly, not seen this when running the actual regular testsuite, but I suppose far fewer people actually run that, so could just be a frequency thing. Bug: https://bugs.gentoo.org/828535 Bug: https://bugs.gentoo.org/850154 Bug: https://bugs.gentoo.org/903890 Closes: https://bugs.gentoo.org/900429 Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.10.11.ebuild | 4 ++++ dev-lang/python/python-3.11.3.ebuild | 4 ++++ dev-lang/python/python-3.12.0_alpha7.ebuild | 4 ++++ dev-lang/python/python-3.9.16_p3.ebuild | 4 ++++ 4 files changed, 16 insertions(+) Additionally, it has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=82692290c64f6dd2f11f25625489dab7f7749fab commit 82692290c64f6dd2f11f25625489dab7f7749fab Author: Sam James <sam@gentoo.org> AuthorDate: 2023-04-05 23:53:32 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-04-06 00:31:47 +0000 dev-lang/python: add timeout for PGO task for >= 3.10 Add a timeout matching the upstream default PROFILE_TASK given how often we've seen hangs here. Bug: https://bugs.gentoo.org/828535 Bug: https://bugs.gentoo.org/850154 Bug: https://bugs.gentoo.org/900429 Bug: https://bugs.gentoo.org/903890 Thanks-to: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.10.11.ebuild | 7 ++++++- dev-lang/python/python-3.11.3.ebuild | 7 ++++++- dev-lang/python/python-3.12.0_alpha7.ebuild | 7 ++++++- 3 files changed, 18 insertions(+), 3 deletions(-)
