Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 937384 - app-portage/gentoolkit-0.6.7: FEATURES="distcc" eclean hangs at exit: coroutine 'ProxyManager.stop' was never awaited
Summary: app-portage/gentoolkit-0.6.7: FEATURES="distcc" eclean hangs at exit: corouti...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, REGRESSION
Depends on: 933499
Blocks:
  Show dependency tree
 
Reported: 2024-08-05 15:15 UTC by Matt Whitlock
Modified: 2024-09-11 01:30 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Whitlock 2024-08-05 15:15:13 UTC
After `eclean distfiles` has completed its work, it hangs indefinitely until terminated by a signal, at which time it emits the following:

^C
Exception ignored in atexit callback: <function _exit_function at 0x7fecd3c64040>
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/util.py", line 360, in _exit_function
Traceback (most recent call last):
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    p.join()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 149, in join
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/lib/portage/python3.12/socks5-server.py", line 226, in run_socks5_server
    await asyncio.get_running_loop().create_future()
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/portage/python3.12/socks5-server.py", line 238, in <module>
    res = self._popen.wait(timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 43, in wait
    asyncio.run(run_socks5_server(sys.argv[1]))
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 27, in poll
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
    pid, sts = os.waitpid(self.pid, flag)
  File "/usr/lib/python3.12/asyncio/runners.py", line 123, in run
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python-exec/python3.12/eclean", line 27, in exithandler
    raise KeyboardInterrupt()
KeyboardInterrupt
    sys.exit(1)
SystemExit: 1
/usr/lib/python3.12/site-packages/portage/process.py:219: RuntimeWarning: coroutine 'ProxyManager.stop' was never awaited
  func(*targs, **kargs)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Task was destroyed but it is pending!
task: <Task pending name='Task-4979' coro=<MultiprocessingProcess._proc_join() running at /usr/lib/python3.12/site-packages/portage/process.py:453> wait_for=<Future pending cb=[sleep.<locals>.cancel_callback() at /usr/lib/python3.12/site-packages/portage/util/futures/_asyncio/__init__.py:229, Task.task_wakeup()]> cb=[MultiprocessingProcess._proc_join_done()]>


The hang occurs only if "distcc" is present in FEATURES. A workaround is to pass FEATURES='-distcc' when invoking eclean.
Comment 1 Mike Gilbert gentoo-dev 2024-08-05 16:48:07 UTC
Please provide emerge --info.
Comment 2 Zac Medico gentoo-dev 2024-08-05 17:45:29 UTC
The ProxyManager.stop coroutine was added for bug 925240.
Comment 3 Zac Medico gentoo-dev 2024-08-05 18:03:13 UTC
The ProxyManager.stop coroutine is supposed to be called when the even loop is closing. I portage we explicitly close the event loop at exit but I wonder if we could somehow avoid that in eclean here.
Comment 4 Matt Whitlock 2024-08-05 19:52:33 UTC
# emerge --info app-portage/gentoolkit
Portage 3.0.65 (python 3.12.4-final-0, default/linux/amd64/23.0/desktop/plasma, gcc-14, glibc-2.40, 6.6.41-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-6.6.41-gentoo-x86_64-AMD_Ryzen_9_7940HS_w-_Radeon_780M_Graphics-with-glibc2.40
KiB Mem:    63588836 total,  57700700 free
KiB Swap:   75495420 total,  72145856 free
Timestamp of repository gentoo: Sat, 27 Jul 2024 13:30:00 +0000
Head commit of repository gentoo: 74e73ea4d240a000e5a15dfdd689da272d81cb1b
sh dash 0.5.12
ld GNU ld (Gentoo 2.42 p6) 2.42.0
distcc 3.4 x86_64-pc-linux-gnu [disabled]
ccache version 4.10.1 [enabled]
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p26-r8::gentoo
dev-build/autoconf:        2.72-r1::gentoo
dev-build/automake:        1.16.5-r2::gentoo, 1.17-r1::gentoo
dev-build/cmake:           3.30.1::gentoo
dev-build/libtool:         2.4.7-r4::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.5.0::gentoo
dev-lang/perl:             5.40.0::gentoo
dev-lang/python:           3.12.4_p2::gentoo, 3.13.0_beta4::gentoo
dev-lang/rust:             1.79.0::gentoo
dev-util/ccache:           4.10.1::gentoo
sys-apps/baselayout:       2.15::gentoo
sys-apps/openrc:           0.54.2::gentoo
sys-apps/sandbox:          2.39::gentoo
sys-devel/binutils:        2.42-r2::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           18.1.8::gentoo
sys-devel/gcc:             14.1.1_p20240720::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/lld:             18.1.8::gentoo
sys-devel/llvm:            18.1.8-r2::gentoo
sys-kernel/linux-headers:  6.9::gentoo (virtual/os-headers)
sys-libs/glibc:            2.40::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    volatile: False
    sync-rsync-extra-opts: 
    sync-rsync-verify-metamanifest: no
    sync-rsync-verify-max-age: 3
    sync-rsync-verify-jobs: 1

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O3 -ggdb -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/X11/xkb /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/i18n/locales/en_US /usr/share/sddm/scripts/Xsetup /usr/share/themes/oxygen-gtk/gtk-3.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dbus-1 /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d"
CXXFLAGS="-march=native -O3 -ggdb -pipe"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--nospinner --unordered-display --verbose-conflicts --autounmask=n --backtrack=100 --usepkg --load-average=16"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live ccache compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict strict-keepdir unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync warn-on-large-env"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="        http://home.mattwhitlock.com/   http://mirror.leaseweb.com/gentoo/      http://mirrors.rit.edu/gentoo/  http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ "
INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd/system /usr/lib/sysusers.d     /etc/xinetd.d   /usr/share/bash-completion /usr/share/fish /usr/share/zsh   /usr/share/cmake/Help /usr/share/gtk-doc        /usr/share/locale/*/ -/usr/share/locale/en -/usr/share/locale/en_US"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O2 -Wl,--as-needed"
LEX="flex"
LINGUAS="en_US en"
MAKEOPTS="-j16 -l24"
PKGDIR="/var/cache/binpkgs"
PORTAGE_COMPRESS="xz"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git --exclude=/CACHEDIR.TAG"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X a52 aac acpi alsa amd64 branding bzip2 cairo cdda cdparanoia cdr crypt cups curl dbus declarative dri dts dvd dvdr elogind encode evdev exif ffmpeg flac fontconfig gif git gmp gtk gui iconv icu id3tag ipv6 jpeg kde kf6compat kwallet lame lcms libnotify libsamplerate libtirpc lzma mad matroska mng mp3 mp4 mpeg multilib ncurses networkmanager ogg opengl opus pam pango pcre pdf php pipewire plasma png policykit postgres ppds pulseaudio qml qt5 quicktime readline sctp sdl seccomp smp sound spell ssl startup-notification subversion svg taglib tcmalloc test-rust theora threads tiff truetype udev udisks unicode upower usb v4l vaapi vdpau vnc vorbis vpx vulkan wayland webkit webp widgets x264 xcb xft xml xv xvid xvmc zlib zstd" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 avx512_bf16 avx512_bitalg avx512_vbmi2 avx512_vnni avx512_vpopcntdq avx512bw avx512cd avx512dq avx512f avx512ifma avx512vbmi avx512vl f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 vpclmulqdq" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en-US en" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LLVM_TARGETS="AMDGPU" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_12" PYTHON_TARGETS="python3_12" RUBY_TARGETS="ruby31 ruby32" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

=================================================================
                        Package Settings
=================================================================

app-portage/gentoolkit-0.6.7::gentoo was built with the following:
USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_12 -pypy3 -python3_10 -python3_11"
CFLAGS="-pipe -march=native -O3 -ggdb"
CXXFLAGS="-pipe -march=native -O3 -ggdb"
Comment 5 Zac Medico gentoo-dev 2024-08-06 04:55:53 UTC
Since eclean-dist doesn't really need the socks5 proxy, we can avoid the problem by skipping the socks5 proxy as in https://github.com/gentoo/portage/pull/1365.
Comment 6 Zac Medico gentoo-dev 2024-08-06 05:09:09 UTC
Something like this would suppress the issue in eclean, since closing the event loop stops the socks5 proxy:

--- a/pym/gentoolkit/eclean/cli.py
+++ b/pym/gentoolkit/eclean/cli.py
@@ -20,6 +20,7 @@ import sys
 import time
 
 import portage
+from portage.util._eventloop.global_event_loop import global_event_loop
 from portage.output import green, red, turquoise, white, yellow
 
 import gentoolkit.pprinter as pp
@@ -745,4 +746,6 @@ if __name__ == "__main__":
     except KeyboardInterrupt:
         print("Aborted.")
         sys.exit(130)
+    finally:
+        global_event_loop().close()
     sys.exit(0)
Comment 7 Zac Medico gentoo-dev 2024-08-06 05:24:36 UTC
There's already an atexit hook in portage that's supposed to close loops, so we can also investigate why that doesn't work.
Comment 8 Zac Medico gentoo-dev 2024-08-06 06:05:01 UTC
(In reply to Zac Medico from comment #7)
> There's already an atexit hook in portage that's supposed to close loops, so
> we can also investigate why that doesn't work.

If the loop hasn't been closed, then portage's run_exitfuncs atexit hook will misbehave because the _exithandlers list will contain coroutine functions that run_exitfuncs currently doesn't handle correctly.

If run_exitfuncs skips the coroutine functions and leaves them in the _exithandlers list, then it's possible that one of the non-coroutine atexit functions will close the event loop, triggering a call to run_coroutine_exitfuncs that will cause a successful cleanup.
Comment 9 Larry the Git Cow gentoo-dev 2024-08-07 14:39:45 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=9e6451c88e3da11e0eb7b0bd6b1497c5ca4fb67f

commit 9e6451c88e3da11e0eb7b0bd6b1497c5ca4fb67f
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-08-06 04:48:26 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-08-07 14:39:25 +0000

    doebuild.spawn: Skip socks5 proxy for "depend" phase
    
    Skip the socks5 proxy for the "depend" phase. It should not be needed
    because we only allow bash builtin commands during this phase.
    
    Since the socks5 proxy requires portage's event loop to be explictly
    closed before exit, skipping it will allow programs like eclean-dist
    to avoid the need to explicitly close portage's event loop before exit.
    
    Bug: https://bugs.gentoo.org/937384
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/package/ebuild/doebuild.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 10 Zac Medico gentoo-dev 2024-08-07 14:52:07 UTC
The fix merged in https://github.com/gentoo/portage/pull/1365 should suppress the issue.

Ultimately we'll also want to merge https://github.com/gentoo/portage/pull/1366 in order to correct the underlying problem, but there's an unresolved issue with atexit handlers not working as expected, and it's tricky to examine the registered atexit handlers due to https://github.com/python/cpython/issues/61388.
Comment 11 Zac Medico gentoo-dev 2024-08-08 03:59:06 UTC
I was able to get the atexit hook working in https://github.com/gentoo/portage/pull/1366, including a test case for a standalone program using the portage API like eclean does.
Comment 12 Larry the Git Cow gentoo-dev 2024-08-10 18:49:12 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d6710ee0cdab2a212ff70503f9699f1be4660bb4

commit d6710ee0cdab2a212ff70503f9699f1be4660bb4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-08-08 14:58:31 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-08-08 14:58:31 +0000

    run_exitfuncs: Support loop close via hook
    
    Handle the case where the loop has not been explicitly closed before
    exit. In this case, run_exitfuncs previously tried to call coroutine
    functions as though they were normal functions, which obvously would
    not behave correctly.
    
    Solve this problem by storing the coroutine functions in a separate
    _coroutine_exithandlers list that belongs exclusively to the
    run_coroutine_exitfuncs function, so that it is safe to close the
    loop and call run_coroutine_exitfuncs from inside a run_exitfuncs
    hook. A _thread_weakrefs_atexit hook already exists that will close
    weakly referenced loops. The _thread_weakrefs_atexit hook is now
    fixed to release its lock when it closes a loop, since the same
    lock may need to be re-acquired when run_coroutine_exitfuncs runs.
    
    The included test case demonstrates that run_exitfuncs will run
    via an atexit hook and correctly terminate the socks5 proxy in a
    standalone program using the portage API (like eclean).
    
    Due to a deadlock that will occur if an _exit_function atexit hook
    from the multiprocessing module executes before run_exitfuncs, a
    portage.process._atexit_register_run_exitfuncs() function needs to
    be called in order to re-order the hooks after the first process
    has been started via the multiprocessing module. The natural place
    to call this is in the ForkProcess class, using a global variable
    to trigger the call just once.
    
    Fixes: c3ebdbb42e72 ("elog/mod_custom: Spawn processes in background")
    Bug: https://bugs.gentoo.org/937384
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/process.py                        | 37 ++++++++++++---
 lib/portage/tests/__init__.py                 | 19 ++++++++
 lib/portage/tests/util/test_socks5.py         | 66 ++++++++++++++++++++++++---
 lib/portage/util/_async/ForkProcess.py        |  8 ++++
 lib/portage/util/futures/_asyncio/__init__.py | 24 ++++++----
 5 files changed, 133 insertions(+), 21 deletions(-)
Comment 13 Larry the Git Cow gentoo-dev 2024-09-11 01:30:55 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=02d0e00a1ba811b39140d10e17488f7fc3916534

commit 02d0e00a1ba811b39140d10e17488f7fc3916534
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-09-11 01:30:10 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-09-11 01:30:30 +0000

    sys-apps/portage: add 3.0.66
    
    Closes: https://bugs.gentoo.org/435066
    Closes: https://bugs.gentoo.org/907061
    Closes: https://bugs.gentoo.org/910560
    Closes: https://bugs.gentoo.org/933433
    Closes: https://bugs.gentoo.org/934220
    Closes: https://bugs.gentoo.org/934514
    Closes: https://bugs.gentoo.org/934784
    Closes: https://bugs.gentoo.org/935830
    Closes: https://bugs.gentoo.org/936273
    Closes: https://bugs.gentoo.org/937384
    Closes: https://bugs.gentoo.org/937485
    Closes: https://bugs.gentoo.org/937740
    Closes: https://bugs.gentoo.org/937888
    Closes: https://bugs.gentoo.org/937891
    Closes: https://bugs.gentoo.org/938127
    Closes: https://bugs.gentoo.org/933499
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.66.ebuild | 227 +++++++++++++++++++++++++++++++++
 2 files changed, 228 insertions(+)