There are multiple reports of intermittent key refresh failures, and gemato has no retry support (see bug 649254): bug 647696 bug 648586 https://forums.gentoo.org/viewtopic-p-8179992.html#8179992 https://forums.gentoo.org/viewtopic-p-8185252.html#8185252 https://forums.gentoo.org/viewtopic-p-8188376.html#8188376 https://forums.gentoo.org/viewtopic-p-8190196.html#8190196 There's an example backoff library here which supports both synchronous calls and coroutines: https://pypi.python.org/pypi/backoff https://github.com/litl/backoff
The sync code runs in a fork since bug 557426, and the asyncio currently does not support running the event loop in a fork, see https://bugs.python.org/issue21998. Therefore, the backoff code should not rely on an event loop if it is to be compatible with asyncio (bug 591760).
Since portage has a lot of code already running in forks that relies on event loops, I suppose it's fine for the sync backoff code to rely on an event loop. In fact, I've found this workaround for asyncio that I've posted on the upstream issue: > # https://bugs.python.org/issue21998#msg313196 > > _default_policy = asyncio.get_event_loop_policy() > _pid_loop = {} > > class MultiprocessingPolicy(asyncio.AbstractEventLoopPolicy): > def get_event_loop(self): > pid = os.getpid() > loop = _pid_loop.get(pid) > if loop is None: > loop = self.new_event_loop() > _pid_loop[pid] = loop > return loop > > def new_event_loop(self): > return _default_policy.new_event_loop() > > asyncio.set_event_loop_policy(MultiprocessingPolicy())
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=369f75c043173531d52a4aa6c7ba55e5a8d5b1ac commit 369f75c043173531d52a4aa6c7ba55e5a8d5b1ac Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-09 23:23:01 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-03-09 23:27:56 +0000 sys-apps/portage: rsync-verify off by default (bug 649276) Since portage-2.3.24 is due for stabilization, and rsync-verify support is not really stable yet, turn it off by default. Bug: https://bugs.gentoo.org/649276 See: https://bugs.gentoo.org/647964 Package-Manager: Portage-2.3.24, Repoman-2.3.6 sys-apps/portage/portage-2.3.24-r1.ebuild | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9c860bd224915e7a7e94915a36629174a783f749 commit 9c860bd224915e7a7e94915a36629174a783f749 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-09 23:34:16 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-03-09 23:36:58 +0000 profiles/base: Stable-mask sys-apps/portage[rsync-verify] Bug: https://bugs.gentoo.org/649276 profiles/base/package.use.stable.mask | 5 +++++ 1 file changed, 5 insertions(+)}
I have a working retry decorator, using futures, with API similar to tenacity: https://github.com/zmedico/portage/tree/asyncio-bug_649276 Now I only need to put this to use for the gpg key refresh, and add a few repos.conf configuration options to control it.
Patch series posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/523fb001fa88c5548199966f550f2a5e https://github.com/gentoo/portage/pull/289
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5f29f03d0d9714082fabbbae5cead8857fbc9093 commit 5f29f03d0d9714082fabbbae5cead8857fbc9093 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-30 08:12:19 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-02 16:53:24 +0000 rsync: add key refresh retry (bug 649276) Since key refresh is prone to failure, retry using exponential backoff with random jitter. This adds the following sync-openpgp-* configuration settings: sync-openpgp-key-refresh-retry-count = 40 Maximum number of times to retry key refresh if it fails. Between each key refresh attempt, there is an exponential delay with a constant multiplier and a uniform random multiplier between 0 and 1. sync-openpgp-key-refresh-retry-delay-exp-base = 2 The base of the exponential expression. The exponent is the number of previous refresh attempts. sync-openpgp-key-refresh-retry-delay-max = 60 Maximum delay between each retry attempt, in units of seconds. This places a limit on the length of the exponential delay. sync-openpgp-key-refresh-retry-delay-mult = 4 Multiplier for the exponential delay. sync-openpgp-key-refresh-retry-overall-timeout = 1200 Combined time limit for all refresh attempts, in units of seconds. Bug: https://bugs.gentoo.org/649276 cnf/repos.conf | 5 ++ man/portage.5 | 21 +++++++- pym/portage/repository/config.py | 22 +++++++++ pym/portage/sync/modules/rsync/rsync.py | 16 +++++- pym/portage/sync/syncbase.py | 87 ++++++++++++++++++++++++++++++++- 5 files changed, 146 insertions(+), 5 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=1aa9a71731f3ab05e10190493956c70998abf12a commit 1aa9a71731f3ab05e10190493956c70998abf12a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-16 19:37:54 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-02 16:53:24 +0000 Add retry decorator (API inspired by tenacity) This decorator will be useful for retrying asynchronous operations, such as gpg key refresh (bug 649276). The API is inspired by tenacity, but is simpler. Only asynchronous functions (like @asyncio.coroutine functions) are supported. In order to retry a synchronous function, first convert it to an asynchronous function as follows: asynchronous_func = functools.partial( loop.run_in_executor, None, synchronous_func) Bug: https://bugs.gentoo.org/649276 See: https://github.com/jd/tenacity Reviewed-by: Alec Warner <antarus@gentoo.org> pym/portage/tests/util/futures/test_retry.py | 147 +++++++++++++++++++++ pym/portage/util/futures/futures.py | 6 + pym/portage/util/futures/retry.py | 183 +++++++++++++++++++++++++++ 3 files changed, 336 insertions(+)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2edaa0b0321a8378d88f8852507648c05f85fcc2 commit 2edaa0b0321a8378d88f8852507648c05f85fcc2 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-04-04 18:52:00 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-04-04 18:53:26 +0000 profiles/base: Stable-mask <sys-apps/portage-2.3.28[rsync-verify] Retry support for key refresh is implemented in portage-2.3.28. Bug: https://bugs.gentoo.org/649276 profiles/base/package.use.stable.mask | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)}
Fixed in portage-2.3.40-r1.
