Seemingly at random, it *might* be after it sends an email but I'm not sure yet, emerge will exit after completing install of one package before starting the next. [ERROR] Exception in callback AsynchronousTask._exit_listener_cb(<bound method...7f0f2987c930>>) handle: <Handle AsynchronousTask._exit_listener_cb(<bound method...7f0f2987c930>>)> Traceback (most recent call last): File "/usr/lib/python3.11/asyncio/events.py", line 84, in _run self._context.run(self._callback, *self._args) File "/usr/lib/python3.11/site-packages/_emerge/AsynchronousTask.py", line 209, in _exit_listener_cb listener(self) File "/usr/lib/python3.11/site-packages/_emerge/EbuildMerge.py", line 87, in _merge_exit self._start_exit_hook(self.returncode) File "/usr/lib/python3.11/site-packages/_emerge/EbuildMerge.py", line 96, in _start_exit_hook AsyncTaskFuture(future=self.exit_hook(self)), ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/_emerge/EbuildBuild.py", line 645, in _install_exit self._async_unlock_builddir() File "/usr/lib/python3.11/site-packages/_emerge/EbuildBuild.py", line 433, in _async_unlock_builddir portage.elog.elog_process(self.pkg.cpv, self.settings) File "/usr/lib/python3.11/site-packages/portage/elog/__init__.py", line 188, in elog_process m.process(mysettings, str(key), mod_logentries, mod_fulllog) File "/usr/lib/python3.11/site-packages/portage/elog/mod_custom.py", line 21, in process retval = portage.process.spawn_bash(mylogcmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/process.py", line 164, in spawn_bash return spawn(args, opt_name=opt_name, **keywords) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/process.py", line 701, in spawn retval = loop.run_until_complete(pid.wait()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/portage/util/_eventloop/asyncio_event_loop.py", line 179, in _run_until_complete return self._loop.run_until_complete(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/asyncio/base_events.py", line 630, in run_until_complete self._check_running() File "/usr/lib/python3.11/asyncio/base_events.py", line 589, in _check_running raise RuntimeError('This event loop is already running') RuntimeError: This event loop is already running Terminated I've only seen this running a general "update everything" e.g. emerge -uDN @world, running the command again happily continues where it left off, but obviously that doesn't re-do the package after which it failed. I've only witnessed this the past few days, I know there's been changes around async stuff, but given the intermittent nature, it's hard to be sure which version(s) this applies to. If I do happen to stumble across a pattern as to when it's happening, I'll update this. Reproducible: Sometimes Steps to Reproduce: 1. emerge -uDN @world Actual Results: Emerge exits with the above Python stack trace Expected Results: Emerge should continue until all packages have been merged. Portage 3.0.63 (python 3.11.8-final-0, default/linux/amd64/17.1/desktop/plasma/systemd/merged-usr, gcc-13, glibc-2.39-r1, 6.7.6 x86_64) ================================================================= System uname: Linux-6.7.6-x86_64-Intel-R-_Core-TM-_i7-6700_CPU_@_3.40GHz-with-glibc2.39 KiB Mem: 32764724 total, 20570484 free KiB Swap: 33554428 total, 33554428 free Timestamp of repository gentoo: Fri, 01 Mar 2024 08:33:46 +0000 Head commit of repository gentoo: fac8a61ba59ea4d4e375784bedb319da1324802a Head commit of repository kdab-overlay: 2814d9fb642ef2634b9767273f0e541ca03f3880 sh bash 5.2_p26 ld GNU ld (Gentoo 2.42 p3) 2.42.0 distcc 3.4 x86_64-pc-linux-gnu [enabled] ccache version 4.9.1 [enabled] app-misc/pax-utils: 1.3.7::gentoo app-shells/bash: 5.2_p26::gentoo dev-build/autoconf: 2.13-r8::gentoo, 2.72-r1::gentoo dev-build/automake: 1.16.5-r2::gentoo dev-build/cmake: 3.28.3::gentoo dev-build/libtool: 2.4.7-r2::gentoo dev-build/make: 4.4.1-r1::gentoo dev-build/meson: 1.3.2::gentoo dev-java/java-config: 2.3.3-r1::gentoo dev-lang/perl: 5.38.2-r1::gentoo dev-lang/python: 3.11.8_p1::gentoo, 3.12.2_p1::gentoo dev-lang/rust: 1.75.0-r1::gentoo dev-util/ccache: 4.9.1::gentoo sys-apps/baselayout: 2.14-r2::gentoo sys-apps/sandbox: 2.38::gentoo sys-apps/systemd: 255.4::gentoo sys-devel/binutils: 2.42-r1::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/clang: 17.0.6::gentoo sys-devel/gcc: 10.5.0::gentoo, 12.3.1_p20240209::gentoo, 13.2.1_p20240210::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/lld: 17.0.6::gentoo sys-devel/llvm: 17.0.6::gentoo sys-kernel/linux-headers: 6.7::gentoo (virtual/os-headers) sys-libs/glibc: 2.39-r1::gentoo Repositories: gentoo location: /usr/portage sync-type: git sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 volatile: True sync-git-pull-extra-opts: --depth 1000 Octal location: /home/dang/work/portage masters: gentoo volatile: True RandomLAN location: /home/randomdan/dev/portage masters: gentoo volatile: True kdab-overlay location: /var/lib/layman/kdab-overlay sync-type: git sync-uri: https://git.randomdan.homeip.net/git/kdab masters: gentoo priority: 50 volatile: True ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="@FREE linux-fw-redistributable no-source-code vim.org MSttfEULA free-noncomm CC-Sampling-Plus-1.0 PUEL FESTIVAL mysql_fdw" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O3 -march=skylake -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.9/conf /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d" CXXFLAGS="-O3 -march=skylake -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live ccache config-protect-if-modified distcc distlocks ebuild-locks fail-clean fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-backup unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://portage.randomdan.homeip.net/" LANG="en_GB.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -O3 -march=native -pipe" LEX="flex" LINGUAS="en en_GB" MAKEOPTS="-j20 -l10" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/tmp" SHELL="/bin/bash" USE="X a52 acl acpi activities alsa amd64 amt autoshutdown autoupdate bash-completion boot branding brotli bzip2 c++0x cairo caps cdda cddb cdr clamonacc clang cli cryptsetup cscope css curl custom-cflags custom-optimization cxx dbus declarative device-mapper dot dracut dri dts dv dvb dvd dvdr encode exif fam ffmpeg firewall firmware flac fontconfig frei0r fuse gallium gcrypt gdbm gif git gles1 gles2 glib gnome-keyring gold gpm graphite graphviz grub gtk gui harfbuzz http2 hwaccel iconv icu ilt imagemagick inkjar inspector introspection ipv6 ithreads javascript jemalloc jit jpeg jpeg2k json jumbo-build kdesu kernel-install kwallet lame lcms ldap libkms libnotify libtirpc lightning llvm lm-sensors lto lvm lz4 lzma lzo mad markdown minizip mng mp3 mp4 mpeg mpm-prefork mtp mysql ncurses network network-cron nls nohourlyupdate nscd nuv nvme odbc offensive offload ogg openconnect openexr opengl openmp opus pam pango pcre pcre-jit pcre16 pcre32 pdf perl pipewire pkcs11 plasma plymouth png policykit postgres postproc ppds profiler pulseaudio python qml qt5 readline rtmp savedconfig screencast seccomp secure-delete silc smp sndfile sound spell spi sqlite ssd ssh ssl startup-notification svg svg2 symlink syslog system-act system-av1 system-boost system-bootloader system-bootstrap system-cmark system-crontab system-ffmpeg system-harfbuzz system-heimdal system-icu system-info system-ipxe system-jpeg system-lcms system-leveldb system-libcxx system-libevent system-libs system-libvpx system-libyaml system-llvm system-lua system-lz4 system-man system-mathjax system-mesa system-mitkrb5 system-numpy system-png system-python system-qemu system-seabios system-sqlite system-ssl system-tbb system-webp system-wfconfig system-wide system-wlroots system-zlib systemd taglib test-rust theora threads thumbnail tiff truetype udev udisks uefi unicode upower usb user-session uuid vaapi vim-pager vim-syntax virtualbox visio vlc vorbis vpx vulkan wayland widevine widgets winbind wxwidgets x264 xattr xcb xft xinerama xml xmp xv xvfb xvid zip zlib zstd" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="macro actions alias allowmethods auth_basic authn_alias authn_anon authn_core authn_dbm authn_file authz_core authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias proxy proxy_http cache_socache cache_disk remoteip" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="evdev joystick" KERNEL="linux" L10N="en en-GB" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres12 postgres14" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Clarification on the exact command being run (it's in a script I'd forgotten about) emerge -uqDN @world @system --keep-going --with-bdeps=y --verbose-conflicts --backtrack=100 -k
Can't be coincidence, 12 package update today, did 8 of 12, 8th was grub, email sent and then it errored out. Mail settings are in a make.defaults in a custom profile: 12 PORTAGE_ELOG_MAILSUBJECT="\${HOST} - package \${PACKAGE}" 13 PORTAGE_ELOG_MAILURI="dan@randomdan.homeip.net smtp.random.lan" 14 PORTAGE_ELOG_SYSTEM="mail" I don't *think* they're a problem because that's worked for years and the mails DO get sent... Mar 02 12:23:37 danielg exim[125590]: [1\2] 2024-03-02 12:23:37 Warning: purging the environment. Mar 02 12:23:37 danielg exim[125590]: [2/2] Suggested action: use keep_environment. Mar 02 12:23:37 danielg exim[125590]: 2024-03-02 12:23:37 1rgOOv-00000000Wfe-0pYv <= root@danielg.random.lan U=root P=local S=766 Mar 02 12:23:38 danielg exim[125592]: 2024-03-02 12:23:38 1rgOOv-00000000Wfe-0pYv => dan.goodliffe@randomdan.homeip.net R=smarthost T=smarthost_smtp H=virtualip1.random.lan [10.10.0.253] X=TLS1.3:TLS_AES_256_GCM_SHA384:256 CV=yes K C="250- 788 byte chunk, total 788\\n250 OK id=1rgOOv-0000000> Mar 02 12:23:38 danielg exim[125592]: 2024-03-02 12:23:38 1rgOOv-00000000Wfe-0pYv Completed Can't see anything amiss here.
(In reply to Dan Goodliffe from comment #2) > Can't be coincidence, 12 package update today, did 8 of 12, 8th was grub, > email sent and then it errored out. > > Mail settings are in a make.defaults in a custom profile: > > 12 PORTAGE_ELOG_MAILSUBJECT="\${HOST} - package \${PACKAGE}" > 13 PORTAGE_ELOG_MAILURI="dan@randomdan.homeip.net smtp.random.lan" > 14 PORTAGE_ELOG_SYSTEM="mail" Did you see another traceback from portage? For the mail elog module there's not the same issue as the custom module shown in comment #0 because the mail elog module does not call the portage.process module (issues with the portage.process module arose from changes related to bug 916566).
Created attachment 886559 [details] Screenshot of whole process
Picked an arbitrary small package that I knew would email out, merged that on its own. Screenshot attached.
(In reply to Dan Goodliffe from comment #4) > Created attachment 886559 [details] > Screenshot of whole process This is the same traceback as comment #0, and is for elog/mod_custom rather than elog/mod_mail. Are you sure you don't have "custom" in your PORTAGE_ELOG_SYSTEM?
Damnit, apologies I do... I've overridden make.defaults in make.conf for some reason and I'd totally forgotten I'd done it or why... PORTAGE_ELOG_SYSTEM="custom" PORTAGE_ELOG_COMMAND="/usr/bin/mailx -s '${PACKAGE}' dan.goodliffe@randomdan.homeip.net < '${LOGFILE}'" (I can only presume this made sense to me at the time for a reason that maybe doesn't exist now) Removing those and using the default mail solution works fine, so it's an issue with the above or with custom, not with email specifically... that was coincidence.
Okay, great, so there's nothing wrong with mod_mail and mod_custom is fixed in https://github.com/gentoo/portage/pull/1295.
It would be nice to add async send to mod_mail still, but it's not critical for now because it doesn't use the event loop to send so it won't trigger the "RuntimeError: This event loop is already running" error.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c3ebdbb42e72335ca65335c855a82b99537c7606 commit c3ebdbb42e72335ca65335c855a82b99537c7606 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-03-03 06:30:50 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-03-03 06:30:50 +0000 elog/mod_custom: Spawn processes in background Since elog_process is typically called while the event loop is running, hold references to spawned processes and wait for them asynchronously, ultimately waiting for them if necessary when the AsyncioEventLoop _close_main method calls _async_finalize via portage.process.run_coroutine_exitfuncs(). ConfigProtectTestCase is useful for exercising this code, and this little make.globals patch can be used to test failure during finalize with this error message: !!! PORTAGE_ELOG_COMMAND failed with exitcode 1 --- a/cnf/make.globals +++ b/cnf/make.globals @@ -144 +144,2 @@ PORTAGE_ELOG_CLASSES="log warn error" -PORTAGE_ELOG_SYSTEM="save_summary:log,warn,error,qa echo" +PORTAGE_ELOG_SYSTEM="save_summary:log,warn,error,qa echo custom" +PORTAGE_ELOG_COMMAND="/bin/false" Bug: https://bugs.gentoo.org/925907 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/elog/mod_custom.py | 73 +++++++++++++++++++++-- lib/portage/process.py | 29 +++++++++ lib/portage/util/_eventloop/asyncio_event_loop.py | 8 ++- 3 files changed, 105 insertions(+), 5 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f1e53b9368c215451e3af78b557d8b86a2db9677 commit f1e53b9368c215451e3af78b557d8b86a2db9677 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-04-28 00:31:29 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-04-28 00:31:29 +0000 sys-apps/portage: add 3.0.64 Closes: https://bugs.gentoo.org/925481 Closes: https://bugs.gentoo.org/675748 Closes: https://bugs.gentoo.org/712222 Closes: https://bugs.gentoo.org/920330 Closes: https://bugs.gentoo.org/925213 Closes: https://bugs.gentoo.org/925863 Closes: https://bugs.gentoo.org/925888 Closes: https://bugs.gentoo.org/925907 Closes: https://bugs.gentoo.org/926048 Closes: https://bugs.gentoo.org/926221 Closes: https://bugs.gentoo.org/927059 Closes: https://bugs.gentoo.org/927632 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.64.ebuild | 242 +++++++++++++++++++++++++++++++++ 2 files changed, 243 insertions(+)
