Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 646898 - sys-apps/portage: Please start implementing GLEP75 (split mirror structure)
Summary: sys-apps/portage: Please start implementing GLEP75 (split mirror structure)
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 646068
Blocks: 534528 697734
  Show dependency tree
 
Reported: 2018-02-07 14:31 UTC by Michał Górny
Modified: 2020-01-17 22:22 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2018-02-07 14:31:48 UTC
I think there's a reasonable agreement on the new GLEP, so it'd be useful to start implementing Portage support. I think the actual implementation can be split into multiple parts, that is:


1. emirrordist support
======================

1.1. implement class to read 'layout.conf' from a mirror.

1.2. have classes to represent different layouts with functions such as 'get distfile path', 'find distfiles' and whatever else Portage needs.

1.3. emirrordist reads 'layout.conf' from DISTDIR it is operating on, with extra option to specify another config file.

1.4. core emirrordist functionality:

1.4.1. new distfiles are fetched to layout [0] and then hardlinked to remaining layouts,

1.4.2. existing distfiles that are not present in all layouts are hardlinked from the existing layout,

1.4.3. old distfiles are removed from all layouts in which they are present.


At this point, we could start deploying it on Infra, that is creating the extra directory structure needed to accomplish this.


2. fetching support
===================

2.1. before using any of GENTOO_MIRRORS, Portage checks local cache for layout.conf of that mirror and fetches/updates it as necessary.

2.1.1. note: we should somehow indicate whenever mirror doesn't have that file to avoid continuously attempting to fetch it.

2.2. Portage uses appropriate layout (first that is supported) to get remote path for fetching.


At this point, our users can start taking advantage of the new layout.


3. Local DISTDIR split
======================

Technically, it's not necessary but I guess some users will appreciate having a split of their DISTDIR as well. Details up for Portage team to decide.
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-10-02 13:51:57 UTC
Ping.  Is Portage going to implement anything new?
Comment 2 Zac Medico gentoo-dev 2019-10-02 17:48:50 UTC
I don't see myself having time to implement this any time soon, but I'd be happy to review patches.
Comment 3 Larry the Git Cow gentoo-dev 2019-10-04 21:25:51 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6a539b7c5163899db1d58cf152aeab1b2b4f9be4

commit 6a539b7c5163899db1d58cf152aeab1b2b4f9be4
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2019-10-03 14:19:54 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2019-10-04 21:25:00 +0000

    fetch: Support GLEP 75 mirror structure
    
    Add a support for the subset of GLEP 75 needed by Gentoo Infra.  This
    includes fetching and parsing layout.conf, and support for flat layout
    and filename-hash layout with cutoffs being multiplies of 4.
    
    Bug: https://bugs.gentoo.org/646898
    Closes: https://github.com/gentoo/portage/pull/462
    Reviewed-by: Zac Medico <zmedico@gentoo.org>
    Signed-off-by: Michał Górny <mgorny@gentoo.org>

 lib/portage/package/ebuild/fetch.py    | 160 ++++++++++++++++++++++++++++++++-
 lib/portage/tests/ebuild/test_fetch.py |  94 ++++++++++++++++++-
 2 files changed, 250 insertions(+), 4 deletions(-)
Comment 4 Larry the Git Cow gentoo-dev 2019-10-14 23:01:22 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e18292c54af86bee8299b6a925d25eb6c55fd77b

commit e18292c54af86bee8299b6a925d25eb6c55fd77b
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-10-14 22:53:35 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-10-14 23:01:06 +0000

    sys-apps/portage: Bump to version 2.3.77
    
     #220533 Fix FEATURES=skiprocheck read-only DISTDIR support
     #646898 Support GLEP 75 mirror structure
     #658648 Disable emerge --autounmask by default, except for package.use
             and package.license changes
     #695870 Improvements to ebuild(5) man page
    
    Bug: https://bugs.gentoo.org/697734
    Bug: https://bugs.gentoo.org/220533
    Bug: https://bugs.gentoo.org/646898
    Bug: https://bugs.gentoo.org/658648
    Bug: https://bugs.gentoo.org/695870
    Package-Manager: Portage-2.3.77, Repoman-2.3.17
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.77.ebuild | 261 +++++++++++++++++++++++++++++++++
 2 files changed, 262 insertions(+)
Comment 5 Zac Medico gentoo-dev 2019-10-14 23:07:03 UTC
(In reply to Michał Górny from comment #0)
> 3. Local DISTDIR split
> ======================
> 
> Technically, it's not necessary but I guess some users will appreciate
> having a split of their DISTDIR as well. Details up for Portage team to
> decide.

We can open a separate bug for that.