The code I've written is good enough for the initial testing and the transition period. However, before we remove the flat layout it needs two improvements: 1. FETCHCOMMAND support. People can put all kinds of weird stuff in FETCHCOMMAND. Unless we respect it, we may get the wrong result (e.g. fetch failure when FETCHCOMMAND would work). 2. Distinction between 404 and connection failure. We want to assume flat layout and cache that result if layout.conf isn't there but we don't want to wrongly cache flat layout if one connection to the mirror failed. However, I don't see how we could combine the two, i.e. distinguish the two results from FETCHCOMMAND. One option would be to cache positive results long-term (i.e. for 24 hours), and negative results only for one emerge run. So if the mirror failed once, Portage would retry on next run.
Disabling cache for negative results sounds good.
(In reply to Michał Górny from comment #0) > 1. FETCHCOMMAND support. People can put all kinds of weird stuff in > FETCHCOMMAND. Unless we respect it, we may get the wrong result (e.g. fetch > failure when FETCHCOMMAND would work). This one becomes tricky within the context of bug 220533.
(In reply to Zac Medico from comment #2) > (In reply to Michał Górny from comment #0) > > 1. FETCHCOMMAND support. People can put all kinds of weird stuff in > > FETCHCOMMAND. Unless we respect it, we may get the wrong result (e.g. fetch > > failure when FETCHCOMMAND would work). > > This one becomes tricky within the context of bug 220533. Shouldn't the blocker be the other way around? The current code shouldn't block that. Though TBH I fail to see why this would be a problem. If you can't write into DISTDIR, you obviously can't fetch distfiles.
(In reply to Michał Górny from comment #3) > (In reply to Zac Medico from comment #2) > > (In reply to Michał Górny from comment #0) > > > 1. FETCHCOMMAND support. People can put all kinds of weird stuff in > > > FETCHCOMMAND. Unless we respect it, we may get the wrong result (e.g. fetch > > > failure when FETCHCOMMAND would work). > > > > This one becomes tricky within the context of bug 220533. > > Shouldn't the blocker be the other way around? The current code shouldn't > block that. Well, if we have conflicting concerns then the blocker goes both ways. > Though TBH I fail to see why this would be a problem. If you can't write > into DISTDIR, you obviously can't fetch distfiles. The idea behind bug 220533 is that we can't write to DISTDIR locally because it's mounted read-only, but FETCHCOMMAND is set to trigger fetch on a remote system where the same DISTDIR happens to be writable.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=41ae830027648e65f867476a4039e7f51db64673 commit 41ae830027648e65f867476a4039e7f51db64673 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-14 20:34:37 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-14 20:35:10 +0000 get_mirror_url: do not cache negative results Bug: https://bugs.gentoo.org/697566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/fetch.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=76fd5593d8ddf446f445ba3dd9da18a6e44dc5b3 commit 76fd5593d8ddf446f445ba3dd9da18a6e44dc5b3 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-14 20:37:25 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-14 20:38:29 +0000 get_mirror_url: do not cache after ConfigParserError Bug: https://bugs.gentoo.org/697566 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/fetch.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=52bc75a60b84d709712e91c68782f2f207bfce4e commit 52bc75a60b84d709712e91c68782f2f207bfce4e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-20 00:55:09 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-20 08:33:09 +0000 fetch: add force parameter (bug 697566) Add a force parameter which forces download even when a file already exists in DISTDIR (and no digests are available to verify it). This avoids the need to remove the existing file in advance, which makes it possible to atomically replace the file and avoid interference with concurrent processes. This is useful when using FETCHCOMMAND to fetch a mirror's layout.conf file, for the purposes of bug 697566. Bug: https://bugs.gentoo.org/697566 Reviewed-by: Michał Górny <mgorny@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/fetch.py | 22 +++++++++++++++---- lib/portage/tests/ebuild/test_fetch.py | 40 +++++++++++++++++++++++++++++++--- 2 files changed, 55 insertions(+), 7 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=9115e1f6ba35cdcd85f2292dc293f0696caa8f12 commit 9115e1f6ba35cdcd85f2292dc293f0696caa8f12 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2019-10-18 07:23:02 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-10-20 09:25:10 +0000 fetch: Use distfile fetching method to get layout.conf Rewrite the layout.conf getter to reuse the standard fetch() method rather than using urlopen(). While at it, fix negative cache elision to apply to memory cache as well (and not get written to disk if next mirror was fine). Most importantly, this ensures that we respect FETCHCOMMAND while fetching layout.conf, and so layout.conf is fetched the same way normal distfiles are. With some uncommon configurations, the previous disjoint logic might have resulted in one of the fetches failing while the other succeeded. This also adds some nice verbosity. If mirror connection takes a while, the user sees that rather than having Portage wait silently. Bug: https://bugs.gentoo.org/697566 Reviewed-by: Zac Medico <zmedico@gentoo.org> Signed-off-by: Michał Górny <mgorny@gentoo.org> lib/portage/package/ebuild/fetch.py | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ecbfeae408f1bdcfa4241a3f4001d57cf30c3405 commit ecbfeae408f1bdcfa4241a3f4001d57cf30c3405 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-21 19:12:06 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-21 19:16:01 +0000 sys-apps/portage: Bump to version 2.3.78 #697566 fetch: Use FETCHCOMMAND to fetch mirror layout.conf #697890 emirrordist: Fix DeletionTask layout assumptions #697906 emirrordist: Delete potential symlinks for all layouts #698046 make.globals: Change FETCHCOMMAND_RSYNC to --copy-links Bug: https://bugs.gentoo.org/697734 Bug: https://bugs.gentoo.org/697566 Bug: https://bugs.gentoo.org/697890 Bug: https://bugs.gentoo.org/697906 Bug: https://bugs.gentoo.org/698046 Package-Manager: Portage-2.3.78, Repoman-2.3.17 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.78.ebuild | 261 +++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+)
