Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 737698 - sys-apps/portage: doebuild function hangs for external API consumers
Summary: sys-apps/portage: doebuild function hangs for external API consumers
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, REGRESSION
Depends on:
Blocks: 733180
  Show dependency tree
 
Reported: 2020-08-17 21:15 UTC by Zac Medico
Modified: 2021-06-26 18:57 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
example script that hangs (kernel-config-check.py,910 bytes, text/x-python)
2020-08-17 21:16 UTC, Zac Medico
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2020-08-17 21:15:02 UTC
The problem is that doebuild calls _spawn_phase which uses asyncio._safe_loop(), and then the coroutine code in EbuildPhase tries to use the global event loop instead of the local loop from asyncio._safe_loop().
Comment 1 Zac Medico gentoo-dev 2020-08-17 21:16:12 UTC
Created attachment 655180 [details]
example script that hangs
Comment 2 Zac Medico gentoo-dev 2020-08-17 21:23:39 UTC
There are multiple coroutines that need to be fixed, and this fixes the first one:

> diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
> index 4bc2749bd..07209f976 100644
> --- a/lib/_emerge/EbuildPhase.py
> +++ b/lib/_emerge/EbuildPhase.py
> @@ -70,11 +70,11 @@ class EbuildPhase(CompositeTask):
>         _locked_phases = ("setup", "preinst", "postinst", "prerm", "postrm")
>  
>         def _start(self):
> -               future = asyncio.ensure_future(self._async_start(), loop=self.scheduler)
> +               future = asyncio.ensure_future(self._async_start(loop=self.scheduler), loop=self.scheduler)
>                 self._start_task(AsyncTaskFuture(future=future), self._async_start_exit)
>  
>         @coroutine
> -       def _async_start(self):
> +       def _async_start(self, loop=None):
>  
>                 need_builddir = self.phase not in EbuildProcess._phases_without_builddir
Comment 3 Zac Medico gentoo-dev 2020-08-17 22:11:45 UTC
This patch makes it easy to find the missing loop arguments:

> diff --git a/lib/portage/util/futures/compat_coroutine.py b/lib/portage/util/futures/compat_coroutine.py
> index 79bd0da68..741fbe296 100644
> --- a/lib/portage/util/futures/compat_coroutine.py
> +++ b/lib/portage/util/futures/compat_coroutine.py
> @@ -67,6 +67,8 @@ def _generator_future(generator_func, *args, **kwargs):
>         keyword argument named 'loop' is given, then it is used instead of
>         the default event loop.
>         """
> +       if kwargs.get('loop') is None:
> +               raise AssertionError("Missing required argument 'loop'")
>         loop = asyncio._wrap_loop(kwargs.get('loop'))
>         result = loop.create_future()
>         _GeneratorTask(generator_func(*args, **kwargs), result, loop=loop)
Comment 4 Michael 'veremitz' Everitt 2020-08-18 17:23:22 UTC
What on earth commit does that reference, Zac?! And if its 2012, is it still relevant?!? :)
Comment 5 Michael 'veremitz' Everitt 2020-08-18 18:47:07 UTC
(In reply to Michael 'veremitz' Everitt from comment #4)
> What on earth commit does that reference, Zac?! And if its 2012, is it still
> relevant?!? :)

For completeness, here is a reply Zac posted to my via IRC [#gentoo-portage] -
[Tue  6:32 pm]<zmedico> veremitz: bug 737698 is triggered by the local event loop support that solves https://bugs.sabayon.org/show_bug.cgi?id=3305 (since local event loops can run in non-main threads)
Comment 6 Zac Medico gentoo-dev 2020-08-18 18:47:54 UTC
The error shown in https://bugs.sabayon.org/show_bug.cgi?id=3305 is still relevant, but today with the asyncio event loop it would manifest as RuntimeError('This event loop is already running') if two threads try to run the main loop at once.
Comment 7 Larry the Git Cow gentoo-dev 2020-08-19 04:47:01 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=dc7919541712d846574e6b7d672a3bed0ca7ef1a

commit dc7919541712d846574e6b7d672a3bed0ca7ef1a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-08-18 06:31:54 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-08-19 04:01:46 +0000

    coroutine: use explicit loop parameter (bug 737698)
    
    In order to support local event loops within API functions
    like doebuild, use an explicit loop parameter when calling a
    coroutine. Internal code will now raise an AssertionError if
    the loop parameter is omitted for a coroutine, but API
    consumers may omit it.
    
    Bug: https://bugs.gentoo.org/737698
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/Binpkg.py                              |  8 ++--
 lib/_emerge/EbuildPhase.py                         | 16 ++++----
 lib/_emerge/Scheduler.py                           |  4 +-
 lib/_emerge/SequentialTaskQueue.py                 |  4 +-
 lib/_emerge/SpawnProcess.py                        |  5 ++-
 lib/portage/dbapi/bintree.py                       | 12 +++---
 lib/portage/dbapi/vartree.py                       |  8 ++--
 .../repository/storage/hardlink_quarantine.py      | 26 ++++++-------
 lib/portage/repository/storage/hardlink_rcu.py     | 34 ++++++++--------
 lib/portage/repository/storage/inplace.py          | 10 ++---
 lib/portage/repository/storage/interface.py        | 10 ++---
 lib/portage/sync/syncbase.py                       |  2 +-
 lib/portage/tests/dbapi/test_auxdb.py              |  9 +++--
 lib/portage/tests/emerge/test_simple.py            |  6 +--
 lib/portage/tests/process/test_AsyncFunction.py    |  4 +-
 lib/portage/tests/process/test_PipeLogger.py       |  2 +-
 .../util/futures/asyncio/test_child_watcher.py     |  4 +-
 .../tests/util/futures/test_compat_coroutine.py    | 45 ++++++++++++----------
 lib/portage/tests/util/test_socks5.py              |  2 +-
 lib/portage/util/_async/BuildLogger.py             |  4 +-
 lib/portage/util/_async/ForkProcess.py             |  6 +--
 lib/portage/util/_async/PipeLogger.py              |  4 +-
 lib/portage/util/_async/SchedulerInterface.py      |  4 +-
 lib/portage/util/futures/_asyncio/process.py       | 16 ++++----
 lib/portage/util/futures/_sync_decorator.py        |  3 +-
 lib/portage/util/futures/compat_coroutine.py       |  7 +++-
 lib/portage/util/socks5.py                         |  4 +-
 repoman/lib/repoman/modules/scan/depend/profile.py |  4 +-
 28 files changed, 138 insertions(+), 125 deletions(-)
Comment 8 Larry the Git Cow gentoo-dev 2020-08-19 05:43:26 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ed032cc07b8877e32f3c5d4a9f6e95096d6419ea

commit ed032cc07b8877e32f3c5d4a9f6e95096d6419ea
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-08-19 05:40:22 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-08-19 05:43:17 +0000

    app-portage/repoman: Bump to version 3.0.1
    
     #737698 Update repoman for compatibility with portage-3.0.4
    
    Bug: https://bugs.gentoo.org/737698
    Package-Manager: Portage-3.0.4, Repoman-3.0.1
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 app-portage/repoman/Manifest             |  1 +
 app-portage/repoman/repoman-3.0.1.ebuild | 64 ++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8304a26f4c4d5ef446cac60caa41df5529700929

commit 8304a26f4c4d5ef446cac60caa41df5529700929
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-08-19 05:13:39 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-08-19 05:43:16 +0000

    sys-apps/portage: Bump to version 3.0.4
    
     #737698 fix hang for API consumers, involving support for
             local event loops which may run in threads
    
    Bug: https://bugs.gentoo.org/733180
    Bug: https://bugs.gentoo.org/737698
    Package-Manager: Portage-3.0.3, Repoman-3.0.0
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest             |   1 +
 sys-apps/portage/portage-3.0.4.ebuild | 264 ++++++++++++++++++++++++++++++++++
 2 files changed, 265 insertions(+)
Comment 9 Larry the Git Cow gentoo-dev 2021-01-18 12:20:15 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=8d962cb5cc97a5092ff45446c0f8da55b27d2434

commit 8d962cb5cc97a5092ff45446c0f8da55b27d2434
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-01-18 10:09:05 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-01-18 11:18:43 +0000

    coroutine: do not require loop parameter
    
    The loop parameter is not needed since global_event_loop now
    returns the running event loop for the current thread.
    
    Bug: https://bugs.gentoo.org/737698
    Bug: https://bugs.gentoo.org/763339
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/util/futures/compat_coroutine.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)