I've had multiple reports in #gentoo-portage of emerge hanging, and the issue appears to be both difficult to reproduce and debug (I've never experience it myself). In one pdb debug session with Zero_Chaos, it seemed as though there was a missing read event for a PipeReader instance. When we faked the read event, we saw that all of the expected data was sitting in the pipe: > (Pdb) p list(self._running_tasks.values())[0]._current_task._current_task._current_task._digest_pipe_reader.input_files > {'input': 40} > (Pdb) p list(self._running_tasks.values())[0]._current_task._current_task._current_task._digest_pipe_reader._output_handler(40) > None > (Pdb) p p list(self._running_tasks.values())[0]._current_task._current_task._current_task._digest_pipe_reader._read_data > [b'SHA1=25e55233a750937eb6f74c1b1bc2b584e9ee92ef\nMD5=3700a73435636e13cabd202088e35f5f\n'] Apparently the changes that have triggered this issue are all related to bug 709746, beginning with this commit which added the AsynchronousTask async_start method: https://gitweb.gentoo.org/proj/portage.git/commit/?id=d66e9ec0b10522528d62e18b83e012c1ec121787 Note that there's a special type of hang reported in bug 711322 which can be suppressed by the emerge --load-average option, and that's an entirely different type of hang which is more easily understood.
Due to the elusive nature of this bug, I think the best course of action is to revert all changes related to bug 709746 until the problem is better understood.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=71ae5a58fe72bc32dce030210a73ea5c9eeb4a1c commit 71ae5a58fe72bc32dce030210a73ea5c9eeb4a1c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:13:17 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:49 +0000 Revert "AsynchronousTask: add coroutine async_start method" This reverts commit d66e9ec0b10522528d62e18b83e012c1ec121787. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AsynchronousTask.py | 22 +++++----------------- lib/_emerge/CompositeTask.py | 17 +++-------------- lib/_emerge/TaskSequence.py | 7 +------ lib/portage/tests/ebuild/test_doebuild_fd_pipes.py | 6 ++---- lib/portage/tests/ebuild/test_doebuild_spawn.py | 6 +++--- lib/portage/tests/ebuild/test_fetch.py | 9 +++------ lib/portage/tests/ebuild/test_ipc_daemon.py | 4 ++-- lib/portage/tests/ebuild/test_spawn.py | 4 ++-- .../test_lazy_import_portage_baseline.py | 6 +++--- lib/portage/tests/locks/test_asynchronous_lock.py | 18 +++++++++--------- lib/portage/tests/process/test_PopenProcess.py | 6 +++--- .../tests/process/test_PopenProcessBlockingIO.py | 4 ++-- lib/portage/tests/process/test_poll.py | 4 ++-- .../util/futures/test_done_callback_after_exit.py | 2 +- .../tests/util/futures/test_iter_completed.py | 16 +++++----------- lib/portage/tests/util/test_file_copier.py | 6 +++--- 16 files changed, 49 insertions(+), 88 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=62ee9ec33821768388cf16a3a1e7b603c845f0c9 commit 62ee9ec33821768388cf16a3a1e7b603c845f0c9 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:09:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:49 +0000 Revert "AbstractEbuildProcess: add _async_start coroutine" This reverts commit a287c49f84ad3af7c8e20bebd116ea972f318e04. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 33 ++++++++++++++++++++++++--------- lib/_emerge/MiscFunctionsProcess.py | 8 +++----- 2 files changed, 27 insertions(+), 14 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=a0700108913e85b359ea7c579147f8f1d7825b9e commit a0700108913e85b359ea7c579147f8f1d7825b9e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:07:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:49 +0000 Revert "EbuildPhase: add _async_start coroutine" This reverts commit c7e52d0466211907d20cdbc04f1e90e7da626694. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildPhase.py | 21 ++++++++++----------- lib/_emerge/SequentialTaskQueue.py | 14 +++----------- 2 files changed, 13 insertions(+), 22 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=acd8f3a3841f3be50d4d649bb7180e3e0c77acfc commit acd8f3a3841f3be50d4d649bb7180e3e0c77acfc Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:03:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:49 +0000 Revert "AsynchronousTask: Call _start_hook after _start_async" This reverts commit 036c644712c36b2d67e818e6f60aa753ebd6acaf. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AsynchronousTask.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=bf1505777847aface5b4cac16b955071c2915ddd commit bf1505777847aface5b4cac16b955071c2915ddd Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:03:34 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:48 +0000 Revert "AsyncScheduler: use async_start method" This reverts commit 8f47d3fe1190d4476ae9eebfafcebdfb1794fc05. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/ebuild/test_doebuild_fd_pipes.py | 8 +++-- .../tests/util/futures/test_iter_completed.py | 2 -- lib/portage/util/_async/AsyncScheduler.py | 20 ++---------- lib/portage/util/futures/iter_completed.py | 38 +++++----------------- 4 files changed, 15 insertions(+), 53 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=5f0f1daa25e4eef751660909610ab247aff50f39 commit 5f0f1daa25e4eef751660909610ab247aff50f39 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:02:13 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:48 +0000 Revert "EbuildBuildDir: use async_start method" This reverts commit 2c596f49bab63c6c81dd4d68789823d45341264d. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildBuildDir.py | 122 +++++++++++++++++++++++++++--------------- 1 file changed, 78 insertions(+), 44 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=174b624d6a2d863d9779b0c2c4035ff33ba46a90 commit 174b624d6a2d863d9779b0c2c4035ff33ba46a90 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:01:31 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:48 +0000 Revert "ForkExecutor: use async_start method" This reverts commit 4df7a0a0c16c5ded65ad601d39840797b7704770. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AsynchronousTask.py | 15 ++------------- lib/portage/util/futures/executor/fork.py | 21 +++------------------ 2 files changed, 5 insertions(+), 31 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=28adac2b67f46e11b4cf9116761d888e91e55f3c commit 28adac2b67f46e11b4cf9116761d888e91e55f3c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:00:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:48 +0000 Revert "SpawnProcess: use async_start method (bug 709746)" This reverts commit 5c40c3e7ec180c9c7d1eea521d69487177c7f519. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 2 +- lib/_emerge/AsynchronousLock.py | 15 +++------------ lib/_emerge/BinpkgExtractorAsync.py | 9 ++------- lib/_emerge/BinpkgFetcher.py | 9 ++------- lib/_emerge/EbuildFetcher.py | 9 ++------- lib/_emerge/SpawnProcess.py | 8 ++------ lib/portage/dbapi/bintree.py | 4 ++-- lib/portage/tests/util/futures/test_iter_completed.py | 6 +----- lib/portage/util/_async/AsyncFunction.py | 9 ++------- lib/portage/util/_async/FileDigester.py | 9 ++------- 10 files changed, 19 insertions(+), 61 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=f1e9389d64b6ded41d0dac99979a049cfa27e75c commit f1e9389d64b6ded41d0dac99979a049cfa27e75c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 05:00:11 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:48 +0000 Revert "PipeLogger: non-blocking write to pipe (bug 709746)" This reverts commit 27712651aa7014a960b012dc89457df09677edc1. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/process/test_PopenProcess.py | 41 +--------------- lib/portage/util/_async/PipeLogger.py | 67 +++++--------------------- 2 files changed, 14 insertions(+), 94 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=d2443903643b8662ee5ac0bd03d3bbad964a9633 commit d2443903643b8662ee5ac0bd03d3bbad964a9633 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:55:37 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:47 +0000 Revert "SpawnProcess: add _main coroutine" This reverts commit 8074127bbc213fde75d51309c8fb4ee33ad278aa. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 33 +++++++++++---------------------- 1 file changed, 11 insertions(+), 22 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=971124a4cf7b0a578b7c8b87134743f8135481b6 commit 971124a4cf7b0a578b7c8b87134743f8135481b6 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:55:13 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:47 +0000 Revert "SpawnProcess: only cancel self._main_task if it's not done" This reverts commit 10cc6febb701cde8f7c099f5ecdf177486e94b3c. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=7e2918896d8dd84e07082bba734d3ce8429ba36d commit 7e2918896d8dd84e07082bba734d3ce8429ba36d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:54:27 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:47 +0000 Revert "SpawnProcess: make _main cancel pipe_logger when appropriate" This reverts commit 6eb6500435c80837f106d3f9f9f6bb246ff00c16. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=f7e36d1e9d468b5a7f502e477ba1971f8334a5d9 commit f7e36d1e9d468b5a7f502e477ba1971f8334a5d9 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:53:39 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:47 +0000 Revert "SpawnProcess: cancel pipe_logger after async_start CancelledError" This reverts commit d331f03ab25422e9afc2e752ea8e39c93a4b0bec. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=044012ffdbe8e946712fa63da9c03d31d8e1047d commit 044012ffdbe8e946712fa63da9c03d31d8e1047d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:53:06 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:46 +0000 Revert "CompositeTask: call self.cancel() for _start_task CancelledError" This reverts commit 1c681e0cc783ca241b1f2e1c782fccd8f1aac769. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/CompositeTask.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=4122a1e12884288e1d578a0cc8cfb400f00b8199 commit 4122a1e12884288e1d578a0cc8cfb400f00b8199 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:51:23 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:46 +0000 Revert "AsyncScheduler: cancel task after _task_coroutine CancelledError" This reverts commit 27a6ee3d09ff79b6addb2696a10fcc3f46e5ca91. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/AsyncScheduler.py | 1 - 1 file changed, 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=79a46f6ebdc9dc2b9bec611c8b88037b2ffe43d6 commit 79a46f6ebdc9dc2b9bec611c8b88037b2ffe43d6 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:50:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:46 +0000 Revert "EbuildBuildDir: cancel current tasks for CancelledError" This reverts commit 899d5e61010a85702e7c83506d2214ede179c964. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildBuildDir.py | 46 +++++++++++++++---------------------------- 1 file changed, 16 insertions(+), 30 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=a0148f998ad81700f68402ab3f58140e6d32f57d commit a0148f998ad81700f68402ab3f58140e6d32f57d Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:48:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:46 +0000 Revert "Support PORTAGE_LOG_FILTER_FILE (bug 709746)" This reverts commit 73f72f526a66b9953a46868cc1390fde2820997f. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 2 - lib/_emerge/EbuildPhase.py | 36 ++----- lib/_emerge/SpawnProcess.py | 32 ++---- .../package/ebuild/_config/special_env_vars.py | 8 +- lib/portage/util/_async/BuildLogger.py | 116 --------------------- lib/portage/util/_async/SchedulerInterface.py | 32 +----- man/make.conf.5 | 7 +- 7 files changed, 20 insertions(+), 213 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=c2ffa9413bbad9d56cbed6e1d779c204fafafc69 commit c2ffa9413bbad9d56cbed6e1d779c204fafafc69 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:48:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:45 +0000 Revert "_BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)" This reverts commit 1681309f252a4e91d7256b895a9af26ef82a9b30. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/BinpkgFetcher.py | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=66c66190724b6d58961827c5d642c59db10fcdb1 commit 66c66190724b6d58961827c5d642c59db10fcdb1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:47:12 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:45 +0000 Revert "SpawnProcess: cancel _main_task in _unregister (bug 711174)" This reverts commit 97587a47fcd00d070c081ad3933e4036cf0f8e81. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/SpawnProcess.py | 2 -- 1 file changed, 2 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=17df30bdfb72a157f9e34f3c2efa1e59389861d4 commit 17df30bdfb72a157f9e34f3c2efa1e59389861d4 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 04:34:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 05:29:45 +0000 Revert "Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILE" This reverts commit e24859eaa03ec86e10d842296f5570dd98bed4b7. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 2 +- lib/_emerge/EbuildPhase.py | 2 +- lib/portage/package/ebuild/_config/special_env_vars.py | 4 ++-- lib/portage/util/_async/BuildLogger.py | 2 +- man/make.conf.5 | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-)
Somebody just posted this which shows an exception ignored in _io_loop, which was reverted in f1e9389d64b6ded41d0dac99979a049cfa27e75c: > !!! existing preserved libs: > >>> package: dev-libs/icu-66.1 > * - /usr/lib64/libicudata.so.65 > * - /usr/lib64/libicudata.so.65.1 > * - /usr/lib64/libicui18n.so.65 > * - /usr/lib64/libicui18n.so.65.1 > * used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410) > * used by /usr/bin/upmendex (app-text/texlive-core-2019-r4) > * used by /usr/lib64/libedataserver-1.2.so.24.0.0 (gnome-extra/evolution-data-server-3.34.4) > * used by 5 other files > * - /usr/lib64/libicuuc.so.65 > * - /usr/lib64/libicuuc.so.65.1 > * used by /usr/bin/WebKitWebDriver (net-libs/webkit-gtk-2.24.4) > * used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410) > * used by /usr/bin/upmendex (app-text/texlive-core-2019-r4) > * used by 10 other files > Use emerge @preserved-rebuild to rebuild packages using these libraries > > * IMPORTANT: 4 news items need reading for repository 'gentoo'. > * Use eselect news read to view new items. > > * After world updates, it is important to remove obsolete packages with > * emerge --depclean. Refer to `man emerge` for more information. > Exception ignored in: <generator object _io_loop at 0x7f2ee9d90678> > Traceback (most recent call last): > File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop > future.done() or future.cancel() > File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon > self._check_closed() > File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed > raise RuntimeError('Event loop is closed') > RuntimeError: Event loop is closed > Exception ignored in: <generator object _io_loop at 0x7f2ee9d90a98> > Traceback (most recent call last): > File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop > future.done() or future.cancel() > File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon > self._check_closed() > File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed > raise RuntimeError('Event loop is closed') > RuntimeError: Event loop is closed
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b7ff1f6c1bf0e821e565c606d7df7d70bf575347 commit b7ff1f6c1bf0e821e565c606d7df7d70bf575347 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 06:52:08 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 06:55:33 +0000 sys-apps/portage: Bump to version 2.3.97 #709746 temporarily remove PORTAGE_LOG_FILTER_FILE_CMD support #715162 infer implicit soname from file basename, for musl #716636 emerge hangs in releases after 2.3.89-r1 Bug: https://bugs.gentoo.org/711148 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/715162 Closes: https://bugs.gentoo.org/716636 Package-Manager: Portage-2.3.97, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.97.ebuild | 263 +++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
As discussed on IRC, I, too, have had a hang occur a few times in recent times, one of which seemed to be re-producible [1]. The most recent ones happened during an 'emerge -ak1 $(qlist -Iq dev-qt/*)', moving from 5.15.9999 to 5.15.0_beta3 and back to 5.15.9999, with EMERGE_DEFAULT_OPTS="--quiet-build=n --tree --verbose --unordered-display --autounmask=n --verbose-conflicts --jobs=4 --misspell-suggestions=n". Each time there were at least some binary packages that had been merged last. Latest happened with 'sys-apps/portage-2.3.97' [2]. 1. https://gist.github.com/Chiitoo/0ac1e20e0f9d80a5485db30fa3fb3f9b 2. https://gist.github.com/Chiitoo/cfb3e99c4f40ce2b530e32dcea652a4e
I'll create another batch of reverts for exit listener call_soon changes related to bug 711322, which began with this commit: https://gitweb.gentoo.org/proj/portage.git/commit/?id=46903f3e5622bc479d4687c76c0e9fada8eb53db
(In reply to Zac Medico from comment #6) > I'll create another batch of reverts for exit listener call_soon changes > related to bug 711322, which began with this commit: > > https://gitweb.gentoo.org/proj/portage.git/commit/ > ?id=46903f3e5622bc479d4687c76c0e9fada8eb53db Actually, it appears that --load-average suppresses the issue, just like bug 711322. This points at an issue with callback scheduling related to SequentialTaskQueue, and I'd prefer to fix it rather than revert the exit listener call_soon changes.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=389429d798a186bdbeb11354d7f1299f628851fd commit 389429d798a186bdbeb11354d7f1299f628851fd Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-09 04:45:16 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-09 06:01:27 +0000 Scheduler: wakeup for empty merge queue (bug 711322) Add a wakeup callback to schedule a new merge when the merge queue becomes empty. This prevents the scheduler from hanging in cases where the order of _merge_exit callback invocation may cause the the merge queue to appear non-empty when it is about to become empty. Bug: https://bugs.gentoo.org/711322 Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/Scheduler.py | 23 +++++++++++++++++++++++ lib/_emerge/SequentialTaskQueue.py | 22 ++++++++++++---------- 2 files changed, 35 insertions(+), 10 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=273eac55223836285c42697268c22c925672835e commit 273eac55223836285c42697268c22c925672835e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-09 07:00:57 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-09 07:05:14 +0000 sys-apps/portage: Bump to version 2.3.98 #711322 always wakeup for empty merge queue Bug: https://bugs.gentoo.org/711148 Bug: https://bugs.gentoo.org/711322 Closes: https://bugs.gentoo.org/716636 Package-Manager: Portage-2.3.98, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.98.ebuild | 263 +++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
(In reply to Zac Medico from comment #3) > Somebody just posted this which shows an exception ignored in _io_loop, > which was reverted in f1e9389d64b6ded41d0dac99979a049cfa27e75c: > > > !!! existing preserved libs: > > >>> package: dev-libs/icu-66.1 > > * - /usr/lib64/libicudata.so.65 > > * - /usr/lib64/libicudata.so.65.1 > > * - /usr/lib64/libicui18n.so.65 > > * - /usr/lib64/libicui18n.so.65.1 > > * used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410) > > * used by /usr/bin/upmendex (app-text/texlive-core-2019-r4) > > * used by /usr/lib64/libedataserver-1.2.so.24.0.0 (gnome-extra/evolution-data-server-3.34.4) > > * used by 5 other files > > * - /usr/lib64/libicuuc.so.65 > > * - /usr/lib64/libicuuc.so.65.1 > > * used by /usr/bin/WebKitWebDriver (net-libs/webkit-gtk-2.24.4) > > * used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410) > > * used by /usr/bin/upmendex (app-text/texlive-core-2019-r4) > > * used by 10 other files > > Use emerge @preserved-rebuild to rebuild packages using these libraries > > > > * IMPORTANT: 4 news items need reading for repository 'gentoo'. > > * Use eselect news read to view new items. > > > > * After world updates, it is important to remove obsolete packages with > > * emerge --depclean. Refer to `man emerge` for more information. > > Exception ignored in: <generator object _io_loop at 0x7f2ee9d90678> > > Traceback (most recent call last): > > File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop > > future.done() or future.cancel() > > File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon > > self._check_closed() > > File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed > > raise RuntimeError('Event loop is closed') > > RuntimeError: Event loop is closed > > Exception ignored in: <generator object _io_loop at 0x7f2ee9d90a98> > > Traceback (most recent call last): > > File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop > > future.done() or future.cancel() > > File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon > > self._check_closed() > > File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed > > raise RuntimeError('Event loop is closed') > > RuntimeError: Event loop is closed The RuntimeError was raised by this code, due to interaction which a future inside of a finally block: > try: > yield future > finally: > self.scheduler.remove_reader(fd) > future.done() or future.cancel()
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=ad325eb10bc6e8ec2a8248f8e9173911f957c0da commit ad325eb10bc6e8ec2a8248f8e9173911f957c0da Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-09 20:27:36 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-09 20:47:08 +0000 _schedule_merge_wakeup: handle main loop exit Detect main loop exit and do not attempt to schedule in this case. Fixes: 389429d798a1 ("Scheduler: wakeup for empty merge queue (bug 711322)") Reported-by: Rick Farina <zerochaos@gentoo.org> Bug: https://bugs.gentoo.org/711322 Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/Scheduler.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7bbc98c13ec8eecf7ea291752cdbb72b60240fcf commit 7bbc98c13ec8eecf7ea291752cdbb72b60240fcf Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-09 20:57:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-09 21:05:13 +0000 sys-apps/portage: 2.3.98-r1 revbump Fix this error: Exception in callback Scheduler._schedule_merge_wakeup(<Future finished result=None>) handle: <Handle Scheduler._schedule_merge_wakeup(<Future finished result=None>)> Traceback (most recent call last): File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/lib/python3.7/site-packages/_emerge/Scheduler.py", line 1638, in _schedule_merge_wakeup self._schedule() File "/usr/lib/python3.7/site-packages/_emerge/PollScheduler.py", line 154, in _schedule self._schedule_tasks() File "/usr/lib/python3.7/site-packages/_emerge/Scheduler.py", line 1615, in _schedule_tasks self._keep_scheduling() or self._main_exit.done()): AttributeError: 'NoneType' object has no attribute 'done' Reported-by: Rick Farina <zerochaos@gentoo.org> Bug: https://bugs.gentoo.org/711148 Bug: https://bugs.gentoo.org/711322 Bug: https://bugs.gentoo.org/716636 Package-Manager: Portage-2.3.98, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/{portage-2.3.98.ebuild => portage-2.3.98-r1.ebuild} | 3 +++ 1 file changed, 3 insertions(+)
I think bug 709746 comment 30 explains the reason for the emerge hangs. It's consistent with what we've observed when using pdb to debug a hung emerge instance.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=66da001e4a552b4794bd102d47a43a938583275a commit 66da001e4a552b4794bd102d47a43a938583275a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-23 02:33:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-23 02:34:45 +0000 sys-apps/portage: Bump to version 2.3.102 #709746 Support PORTAGE_LOG_FILTER_FILE_CMD once again, now that bug 716636 is fixed #716636 Fix emerge hang triggered by unsafe remove_reader and remove_writer calls related to bug 709746 Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/716636 Package-Manager: Portage-2.3.102, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.102.ebuild | 263 ++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
