Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 716636 - sys-apps/portage: emerge hangs in releases after 2.3.89-r1
Summary: sys-apps/portage: emerge hangs in releases after 2.3.89-r1
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, REGRESSION
Depends on:
Blocks: 709746 721152
  Show dependency tree
 
Reported: 2020-04-08 03:37 UTC by Zac Medico
Modified: 2020-06-23 02:52 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2020-04-08 03:37:49 UTC
I've had multiple reports in #gentoo-portage of emerge hanging, and the issue appears to be both difficult to reproduce and debug (I've never experience it myself). In one pdb debug session with Zero_Chaos, it seemed as though there was a missing read event for a PipeReader instance. When we faked the read event, we saw that all of the expected data was sitting in the pipe:

> (Pdb) p list(self._running_tasks.values())[0]._current_task._current_task._current_task._digest_pipe_reader.input_files
> {'input': 40}
> (Pdb) p list(self._running_tasks.values())[0]._current_task._current_task._current_task._digest_pipe_reader._output_handler(40)
> None
> (Pdb) p p list(self._running_tasks.values())[0]._current_task._current_task._current_task._digest_pipe_reader._read_data
> [b'SHA1=25e55233a750937eb6f74c1b1bc2b584e9ee92ef\nMD5=3700a73435636e13cabd202088e35f5f\n']

Apparently the changes that have triggered this issue are all related to bug 709746, beginning with this commit which added the AsynchronousTask async_start method:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d66e9ec0b10522528d62e18b83e012c1ec121787

Note that there's a special type of hang reported in bug 711322 which can be suppressed by the emerge --load-average option, and that's an entirely different type of hang which is more easily understood.
Comment 1 Zac Medico gentoo-dev 2020-04-08 03:51:28 UTC
Due to the elusive nature of this bug, I think the best course of action is to revert all changes related to bug 709746 until the problem is better understood.
Comment 2 Larry the Git Cow gentoo-dev 2020-04-08 05:56:45 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=71ae5a58fe72bc32dce030210a73ea5c9eeb4a1c

commit 71ae5a58fe72bc32dce030210a73ea5c9eeb4a1c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:13:17 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:49 +0000

    Revert "AsynchronousTask: add coroutine async_start method"
    
    This reverts commit d66e9ec0b10522528d62e18b83e012c1ec121787.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AsynchronousTask.py                    | 22 +++++-----------------
 lib/_emerge/CompositeTask.py                       | 17 +++--------------
 lib/_emerge/TaskSequence.py                        |  7 +------
 lib/portage/tests/ebuild/test_doebuild_fd_pipes.py |  6 ++----
 lib/portage/tests/ebuild/test_doebuild_spawn.py    |  6 +++---
 lib/portage/tests/ebuild/test_fetch.py             |  9 +++------
 lib/portage/tests/ebuild/test_ipc_daemon.py        |  4 ++--
 lib/portage/tests/ebuild/test_spawn.py             |  4 ++--
 .../test_lazy_import_portage_baseline.py           |  6 +++---
 lib/portage/tests/locks/test_asynchronous_lock.py  | 18 +++++++++---------
 lib/portage/tests/process/test_PopenProcess.py     |  6 +++---
 .../tests/process/test_PopenProcessBlockingIO.py   |  4 ++--
 lib/portage/tests/process/test_poll.py             |  4 ++--
 .../util/futures/test_done_callback_after_exit.py  |  2 +-
 .../tests/util/futures/test_iter_completed.py      | 16 +++++-----------
 lib/portage/tests/util/test_file_copier.py         |  6 +++---
 16 files changed, 49 insertions(+), 88 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=62ee9ec33821768388cf16a3a1e7b603c845f0c9

commit 62ee9ec33821768388cf16a3a1e7b603c845f0c9
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:09:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:49 +0000

    Revert "AbstractEbuildProcess: add _async_start coroutine"
    
    This reverts commit a287c49f84ad3af7c8e20bebd116ea972f318e04.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py | 33 ++++++++++++++++++++++++---------
 lib/_emerge/MiscFunctionsProcess.py  |  8 +++-----
 2 files changed, 27 insertions(+), 14 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=a0700108913e85b359ea7c579147f8f1d7825b9e

commit a0700108913e85b359ea7c579147f8f1d7825b9e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:07:25 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:49 +0000

    Revert "EbuildPhase: add _async_start coroutine"
    
    This reverts commit c7e52d0466211907d20cdbc04f1e90e7da626694.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/EbuildPhase.py         | 21 ++++++++++-----------
 lib/_emerge/SequentialTaskQueue.py | 14 +++-----------
 2 files changed, 13 insertions(+), 22 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=acd8f3a3841f3be50d4d649bb7180e3e0c77acfc

commit acd8f3a3841f3be50d4d649bb7180e3e0c77acfc
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:03:56 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:49 +0000

    Revert "AsynchronousTask: Call _start_hook after _start_async"
    
    This reverts commit 036c644712c36b2d67e818e6f60aa753ebd6acaf.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AsynchronousTask.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=bf1505777847aface5b4cac16b955071c2915ddd

commit bf1505777847aface5b4cac16b955071c2915ddd
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:03:34 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:48 +0000

    Revert "AsyncScheduler: use async_start method"
    
    This reverts commit 8f47d3fe1190d4476ae9eebfafcebdfb1794fc05.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/ebuild/test_doebuild_fd_pipes.py |  8 +++--
 .../tests/util/futures/test_iter_completed.py      |  2 --
 lib/portage/util/_async/AsyncScheduler.py          | 20 ++----------
 lib/portage/util/futures/iter_completed.py         | 38 +++++-----------------
 4 files changed, 15 insertions(+), 53 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=5f0f1daa25e4eef751660909610ab247aff50f39

commit 5f0f1daa25e4eef751660909610ab247aff50f39
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:02:13 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:48 +0000

    Revert "EbuildBuildDir: use async_start method"
    
    This reverts commit 2c596f49bab63c6c81dd4d68789823d45341264d.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/EbuildBuildDir.py | 122 +++++++++++++++++++++++++++---------------
 1 file changed, 78 insertions(+), 44 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=174b624d6a2d863d9779b0c2c4035ff33ba46a90

commit 174b624d6a2d863d9779b0c2c4035ff33ba46a90
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:01:31 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:48 +0000

    Revert "ForkExecutor: use async_start method"
    
    This reverts commit 4df7a0a0c16c5ded65ad601d39840797b7704770.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AsynchronousTask.py           | 15 ++-------------
 lib/portage/util/futures/executor/fork.py | 21 +++------------------
 2 files changed, 5 insertions(+), 31 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=28adac2b67f46e11b4cf9116761d888e91e55f3c

commit 28adac2b67f46e11b4cf9116761d888e91e55f3c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:00:56 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:48 +0000

    Revert "SpawnProcess: use async_start method (bug 709746)"
    
    This reverts commit 5c40c3e7ec180c9c7d1eea521d69487177c7f519.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py                  |  2 +-
 lib/_emerge/AsynchronousLock.py                       | 15 +++------------
 lib/_emerge/BinpkgExtractorAsync.py                   |  9 ++-------
 lib/_emerge/BinpkgFetcher.py                          |  9 ++-------
 lib/_emerge/EbuildFetcher.py                          |  9 ++-------
 lib/_emerge/SpawnProcess.py                           |  8 ++------
 lib/portage/dbapi/bintree.py                          |  4 ++--
 lib/portage/tests/util/futures/test_iter_completed.py |  6 +-----
 lib/portage/util/_async/AsyncFunction.py              |  9 ++-------
 lib/portage/util/_async/FileDigester.py               |  9 ++-------
 10 files changed, 19 insertions(+), 61 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f1e9389d64b6ded41d0dac99979a049cfa27e75c

commit f1e9389d64b6ded41d0dac99979a049cfa27e75c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 05:00:11 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:48 +0000

    Revert "PipeLogger: non-blocking write to pipe (bug 709746)"
    
    This reverts commit 27712651aa7014a960b012dc89457df09677edc1.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/process/test_PopenProcess.py | 41 +---------------
 lib/portage/util/_async/PipeLogger.py          | 67 +++++---------------------
 2 files changed, 14 insertions(+), 94 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d2443903643b8662ee5ac0bd03d3bbad964a9633

commit d2443903643b8662ee5ac0bd03d3bbad964a9633
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:55:37 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:47 +0000

    Revert "SpawnProcess: add _main coroutine"
    
    This reverts commit 8074127bbc213fde75d51309c8fb4ee33ad278aa.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py | 33 +++++++++++----------------------
 1 file changed, 11 insertions(+), 22 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=971124a4cf7b0a578b7c8b87134743f8135481b6

commit 971124a4cf7b0a578b7c8b87134743f8135481b6
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:55:13 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:47 +0000

    Revert "SpawnProcess: only cancel self._main_task if it's not done"
    
    This reverts commit 10cc6febb701cde8f7c099f5ecdf177486e94b3c.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=7e2918896d8dd84e07082bba734d3ce8429ba36d

commit 7e2918896d8dd84e07082bba734d3ce8429ba36d
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:54:27 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:47 +0000

    Revert "SpawnProcess: make _main cancel pipe_logger when appropriate"
    
    This reverts commit 6eb6500435c80837f106d3f9f9f6bb246ff00c16.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f7e36d1e9d468b5a7f502e477ba1971f8334a5d9

commit f7e36d1e9d468b5a7f502e477ba1971f8334a5d9
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:53:39 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:47 +0000

    Revert "SpawnProcess: cancel pipe_logger after async_start CancelledError"
    
    This reverts commit d331f03ab25422e9afc2e752ea8e39c93a4b0bec.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=044012ffdbe8e946712fa63da9c03d31d8e1047d

commit 044012ffdbe8e946712fa63da9c03d31d8e1047d
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:53:06 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:46 +0000

    Revert "CompositeTask: call self.cancel() for _start_task CancelledError"
    
    This reverts commit 1c681e0cc783ca241b1f2e1c782fccd8f1aac769.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/CompositeTask.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=4122a1e12884288e1d578a0cc8cfb400f00b8199

commit 4122a1e12884288e1d578a0cc8cfb400f00b8199
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:51:23 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:46 +0000

    Revert "AsyncScheduler: cancel task after _task_coroutine CancelledError"
    
    This reverts commit 27a6ee3d09ff79b6addb2696a10fcc3f46e5ca91.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/util/_async/AsyncScheduler.py | 1 -
 1 file changed, 1 deletion(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=79a46f6ebdc9dc2b9bec611c8b88037b2ffe43d6

commit 79a46f6ebdc9dc2b9bec611c8b88037b2ffe43d6
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:50:21 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:46 +0000

    Revert "EbuildBuildDir: cancel current tasks for CancelledError"
    
    This reverts commit 899d5e61010a85702e7c83506d2214ede179c964.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/EbuildBuildDir.py | 46 +++++++++++++++----------------------------
 1 file changed, 16 insertions(+), 30 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=a0148f998ad81700f68402ab3f58140e6d32f57d

commit a0148f998ad81700f68402ab3f58140e6d32f57d
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:48:56 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:46 +0000

    Revert "Support PORTAGE_LOG_FILTER_FILE (bug 709746)"
    
    This reverts commit 73f72f526a66b9953a46868cc1390fde2820997f.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py               |   2 -
 lib/_emerge/EbuildPhase.py                         |  36 ++-----
 lib/_emerge/SpawnProcess.py                        |  32 ++----
 .../package/ebuild/_config/special_env_vars.py     |   8 +-
 lib/portage/util/_async/BuildLogger.py             | 116 ---------------------
 lib/portage/util/_async/SchedulerInterface.py      |  32 +-----
 man/make.conf.5                                    |   7 +-
 7 files changed, 20 insertions(+), 213 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c2ffa9413bbad9d56cbed6e1d779c204fafafc69

commit c2ffa9413bbad9d56cbed6e1d779c204fafafc69
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:48:05 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:45 +0000

    Revert "_BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)"
    
    This reverts commit 1681309f252a4e91d7256b895a9af26ef82a9b30.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/BinpkgFetcher.py | 32 +++++++++++++++-----------------
 1 file changed, 15 insertions(+), 17 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=66c66190724b6d58961827c5d642c59db10fcdb1

commit 66c66190724b6d58961827c5d642c59db10fcdb1
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:47:12 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:45 +0000

    Revert "SpawnProcess: cancel _main_task in _unregister (bug 711174)"
    
    This reverts commit 97587a47fcd00d070c081ad3933e4036cf0f8e81.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 --
 1 file changed, 2 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=17df30bdfb72a157f9e34f3c2efa1e59389861d4

commit 17df30bdfb72a157f9e34f3c2efa1e59389861d4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 04:34:56 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 05:29:45 +0000

    Revert "Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILE"
    
    This reverts commit e24859eaa03ec86e10d842296f5570dd98bed4b7.
    
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py                   | 2 +-
 lib/_emerge/EbuildPhase.py                             | 2 +-
 lib/portage/package/ebuild/_config/special_env_vars.py | 4 ++--
 lib/portage/util/_async/BuildLogger.py                 | 2 +-
 man/make.conf.5                                        | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)
Comment 3 Zac Medico gentoo-dev 2020-04-08 06:32:32 UTC
Somebody just posted this which shows an exception ignored in _io_loop, which was reverted in f1e9389d64b6ded41d0dac99979a049cfa27e75c:

> !!! existing preserved libs:
> >>> package: dev-libs/icu-66.1
>  *  - /usr/lib64/libicudata.so.65
>  *  - /usr/lib64/libicudata.so.65.1
>  *  - /usr/lib64/libicui18n.so.65
>  *  - /usr/lib64/libicui18n.so.65.1
>  *      used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410)
>  *      used by /usr/bin/upmendex (app-text/texlive-core-2019-r4)
>  *      used by /usr/lib64/libedataserver-1.2.so.24.0.0 (gnome-extra/evolution-data-server-3.34.4)
>  *      used by 5 other files
>  *  - /usr/lib64/libicuuc.so.65
>  *  - /usr/lib64/libicuuc.so.65.1
>  *      used by /usr/bin/WebKitWebDriver (net-libs/webkit-gtk-2.24.4)
>  *      used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410)
>  *      used by /usr/bin/upmendex (app-text/texlive-core-2019-r4)
>  *      used by 10 other files
> Use emerge @preserved-rebuild to rebuild packages using these libraries
> 
>  * IMPORTANT: 4 news items need reading for repository 'gentoo'.
>  * Use eselect news read to view new items.
> 
>  * After world updates, it is important to remove obsolete packages with
>  * emerge --depclean. Refer to `man emerge` for more information.
> Exception ignored in: <generator object _io_loop at 0x7f2ee9d90678>
> Traceback (most recent call last):
>   File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop
>     future.done() or future.cancel()
>   File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon
>     self._check_closed()
>   File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed
>     raise RuntimeError('Event loop is closed')
> RuntimeError: Event loop is closed
> Exception ignored in: <generator object _io_loop at 0x7f2ee9d90a98>
> Traceback (most recent call last):
>   File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop
>     future.done() or future.cancel()
>   File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon
>     self._check_closed()
>   File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed
>     raise RuntimeError('Event loop is closed')
> RuntimeError: Event loop is closed
Comment 4 Larry the Git Cow gentoo-dev 2020-04-08 06:55:41 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b7ff1f6c1bf0e821e565c606d7df7d70bf575347

commit b7ff1f6c1bf0e821e565c606d7df7d70bf575347
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-08 06:52:08 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-08 06:55:33 +0000

    sys-apps/portage: Bump to version 2.3.97
    
     #709746 temporarily remove PORTAGE_LOG_FILTER_FILE_CMD support
     #715162 infer implicit soname from file basename, for musl
     #716636 emerge hangs in releases after 2.3.89-r1
    
    Bug: https://bugs.gentoo.org/711148
    Bug: https://bugs.gentoo.org/709746
    Bug: https://bugs.gentoo.org/715162
    Closes: https://bugs.gentoo.org/716636
    Package-Manager: Portage-2.3.97, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.97.ebuild | 263 +++++++++++++++++++++++++++++++++
 2 files changed, 264 insertions(+)
Comment 5 Chiitoo gentoo-dev 2020-04-08 17:48:33 UTC
As discussed on IRC, I, too, have had a hang occur a few times in recent times, one of which seemed to be re-producible [1].

The most recent ones happened during an 'emerge -ak1 $(qlist -Iq dev-qt/*)', moving from 5.15.9999 to 5.15.0_beta3 and back to 5.15.9999, with EMERGE_DEFAULT_OPTS="--quiet-build=n --tree --verbose --unordered-display --autounmask=n --verbose-conflicts --jobs=4 --misspell-suggestions=n".

Each time there were at least some binary packages that had been merged last.

Latest happened with 'sys-apps/portage-2.3.97' [2].

1. https://gist.github.com/Chiitoo/0ac1e20e0f9d80a5485db30fa3fb3f9b
2. https://gist.github.com/Chiitoo/cfb3e99c4f40ce2b530e32dcea652a4e
Comment 6 Zac Medico gentoo-dev 2020-04-08 19:10:02 UTC
I'll create another batch of reverts for exit listener call_soon changes related to bug 711322, which began with this commit:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=46903f3e5622bc479d4687c76c0e9fada8eb53db
Comment 7 Zac Medico gentoo-dev 2020-04-08 20:01:51 UTC
(In reply to Zac Medico from comment #6)
> I'll create another batch of reverts for exit listener call_soon changes
> related to bug 711322, which began with this commit:
> 
> https://gitweb.gentoo.org/proj/portage.git/commit/
> ?id=46903f3e5622bc479d4687c76c0e9fada8eb53db

Actually, it appears that --load-average suppresses the issue, just like bug 711322. This points at an issue with callback scheduling related to SequentialTaskQueue, and I'd prefer to fix it rather than revert the exit listener call_soon changes.
Comment 8 Larry the Git Cow gentoo-dev 2020-04-09 06:48:28 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=389429d798a186bdbeb11354d7f1299f628851fd

commit 389429d798a186bdbeb11354d7f1299f628851fd
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-09 04:45:16 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-09 06:01:27 +0000

    Scheduler: wakeup for empty merge queue (bug 711322)
    
    Add a wakeup callback to schedule a new merge when the merge queue
    becomes empty. This prevents the scheduler from hanging in cases
    where the order of _merge_exit callback invocation may cause the
    the merge queue to appear non-empty when it is about to become
    empty.
    
    Bug: https://bugs.gentoo.org/711322
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/Scheduler.py           | 23 +++++++++++++++++++++++
 lib/_emerge/SequentialTaskQueue.py | 22 ++++++++++++----------
 2 files changed, 35 insertions(+), 10 deletions(-)
Comment 9 Larry the Git Cow gentoo-dev 2020-04-09 07:05:22 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=273eac55223836285c42697268c22c925672835e

commit 273eac55223836285c42697268c22c925672835e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-09 07:00:57 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-09 07:05:14 +0000

    sys-apps/portage: Bump to version 2.3.98
    
     #711322 always wakeup for empty merge queue
    
    Bug: https://bugs.gentoo.org/711148
    Bug: https://bugs.gentoo.org/711322
    Closes: https://bugs.gentoo.org/716636
    Package-Manager: Portage-2.3.98, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.98.ebuild | 263 +++++++++++++++++++++++++++++++++
 2 files changed, 264 insertions(+)
Comment 10 Zac Medico gentoo-dev 2020-04-09 19:57:11 UTC
(In reply to Zac Medico from comment #3)
> Somebody just posted this which shows an exception ignored in _io_loop,
> which was reverted in f1e9389d64b6ded41d0dac99979a049cfa27e75c:
> 
> > !!! existing preserved libs:
> > >>> package: dev-libs/icu-66.1
> >  *  - /usr/lib64/libicudata.so.65
> >  *  - /usr/lib64/libicudata.so.65.1
> >  *  - /usr/lib64/libicui18n.so.65
> >  *  - /usr/lib64/libicui18n.so.65.1
> >  *      used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410)
> >  *      used by /usr/bin/upmendex (app-text/texlive-core-2019-r4)
> >  *      used by /usr/lib64/libedataserver-1.2.so.24.0.0 (gnome-extra/evolution-data-server-3.34.4)
> >  *      used by 5 other files
> >  *  - /usr/lib64/libicuuc.so.65
> >  *  - /usr/lib64/libicuuc.so.65.1
> >  *      used by /usr/bin/WebKitWebDriver (net-libs/webkit-gtk-2.24.4)
> >  *      used by /usr/bin/bibtexu (dev-tex/bibtexu-3.71_p20190410)
> >  *      used by /usr/bin/upmendex (app-text/texlive-core-2019-r4)
> >  *      used by 10 other files
> > Use emerge @preserved-rebuild to rebuild packages using these libraries
> > 
> >  * IMPORTANT: 4 news items need reading for repository 'gentoo'.
> >  * Use eselect news read to view new items.
> > 
> >  * After world updates, it is important to remove obsolete packages with
> >  * emerge --depclean. Refer to `man emerge` for more information.
> > Exception ignored in: <generator object _io_loop at 0x7f2ee9d90678>
> > Traceback (most recent call last):
> >   File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop
> >     future.done() or future.cancel()
> >   File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon
> >     self._check_closed()
> >   File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed
> >     raise RuntimeError('Event loop is closed')
> > RuntimeError: Event loop is closed
> > Exception ignored in: <generator object _io_loop at 0x7f2ee9d90a98>
> > Traceback (most recent call last):
> >   File "/usr/lib64/python3.6/site-packages/portage/util/_async/PipeLogger.py", line 91, in _io_loop
> >     future.done() or future.cancel()
> >   File "/usr/lib64/python3.6/asyncio/base_events.py", line 595, in call_soon
> >     self._check_closed()
> >   File "/usr/lib64/python3.6/asyncio/base_events.py", line 381, in _check_closed
> >     raise RuntimeError('Event loop is closed')
> > RuntimeError: Event loop is closed

The RuntimeError was raised by this code, due to interaction which a future inside of a finally block:

> 	try:
> 		yield future
> 	finally:
> 		self.scheduler.remove_reader(fd)
> 		future.done() or future.cancel()
Comment 11 Larry the Git Cow gentoo-dev 2020-04-09 20:48:45 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ad325eb10bc6e8ec2a8248f8e9173911f957c0da

commit ad325eb10bc6e8ec2a8248f8e9173911f957c0da
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-09 20:27:36 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-09 20:47:08 +0000

    _schedule_merge_wakeup: handle main loop exit
    
    Detect main loop exit and do not attempt to schedule in this case.
    
    Fixes: 389429d798a1 ("Scheduler: wakeup for empty merge queue (bug 711322)")
    Reported-by: Rick Farina <zerochaos@gentoo.org>
    Bug: https://bugs.gentoo.org/711322
    Bug: https://bugs.gentoo.org/716636
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/Scheduler.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Comment 12 Larry the Git Cow gentoo-dev 2020-04-09 21:05:27 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7bbc98c13ec8eecf7ea291752cdbb72b60240fcf

commit 7bbc98c13ec8eecf7ea291752cdbb72b60240fcf
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-09 20:57:43 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-04-09 21:05:13 +0000

    sys-apps/portage: 2.3.98-r1 revbump
    
    Fix this error:
    
    Exception in callback Scheduler._schedule_merge_wakeup(<Future finished result=None>)
    handle: <Handle Scheduler._schedule_merge_wakeup(<Future finished result=None>)>
    Traceback (most recent call last):
      File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
        self._context.run(self._callback, *self._args)
      File "/usr/lib/python3.7/site-packages/_emerge/Scheduler.py", line 1638, in _schedule_merge_wakeup
        self._schedule()
      File "/usr/lib/python3.7/site-packages/_emerge/PollScheduler.py", line 154, in _schedule
        self._schedule_tasks()
      File "/usr/lib/python3.7/site-packages/_emerge/Scheduler.py", line 1615, in _schedule_tasks
        self._keep_scheduling() or self._main_exit.done()):
    AttributeError: 'NoneType' object has no attribute 'done'
    
    Reported-by: Rick Farina <zerochaos@gentoo.org>
    Bug: https://bugs.gentoo.org/711148
    Bug: https://bugs.gentoo.org/711322
    Bug: https://bugs.gentoo.org/716636
    Package-Manager: Portage-2.3.98, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/{portage-2.3.98.ebuild => portage-2.3.98-r1.ebuild} | 3 +++
 1 file changed, 3 insertions(+)
Comment 13 Zac Medico gentoo-dev 2020-06-17 08:22:47 UTC
I think bug 709746 comment 30 explains the reason for the emerge hangs. It's consistent with what we've observed when using pdb to debug a hung emerge instance.
Comment 14 Larry the Git Cow gentoo-dev 2020-06-23 02:38:17 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=66da001e4a552b4794bd102d47a43a938583275a

commit 66da001e4a552b4794bd102d47a43a938583275a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-06-23 02:33:05 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-06-23 02:34:45 +0000

    sys-apps/portage: Bump to version 2.3.102
    
     #709746 Support PORTAGE_LOG_FILTER_FILE_CMD once again, now
             that bug 716636 is fixed
     #716636 Fix emerge hang triggered by unsafe remove_reader and
             remove_writer calls related to bug 709746
    
    Bug: https://bugs.gentoo.org/721152
    Bug: https://bugs.gentoo.org/709746
    Bug: https://bugs.gentoo.org/716636
    Package-Manager: Portage-2.3.102, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest               |   1 +
 sys-apps/portage/portage-2.3.102.ebuild | 263 ++++++++++++++++++++++++++++++++
 2 files changed, 264 insertions(+)