Bug 794487

Summary:	sys-apps/portage: support GLEP 75 mirror layout.conf in DISTDIR
Product:	Portage Development	Reporter:	Zac Medico <zmedico>
Component:	Core - Ebuild Support	Assignee:	Portage team <dev-portage>
Status:	CONFIRMED ---
Severity:	enhancement	CC:	dan
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	All
See Also:	https://bugs.gentoo.org/show_bug.cgi?id=646898 https://bugs.gentoo.org/show_bug.cgi?id=629048 https://bugs.gentoo.org/show_bug.cgi?id=645810
Whiteboard:
Package list:		Runtime testing required:	---
Bug Depends on:	646068
Bug Blocks:	377365

Description Zac Medico gentoo-dev

2021-06-06 00:04:55 UTC

It seems like it must be very easy to support layout.conf in DISTDIR since the PORTAGE_RO_DISTDIRS layout.conf support in this commit:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e284848704cb5e189eec35a93a54a8e91aa42d64

> From e284848704cb5e189eec35a93a54a8e91aa42d64 Mon Sep 17 00:00:00 2001
> From: "Daniel M. Weeks" <dan@danweeks.net>
> Date: Mon, 31 May 2021 11:21:45 -0400
> Subject: [PATCH] Support GLEP 75 layout in PORTAGE_RO_DISTDIRS
> 
> Signed-off-by: Daniel M. Weeks <dan@danweeks.net>
> Signed-off-by: Zac Medico <zmedico@gentoo.org>
> ---
>  lib/portage/package/ebuild/fetch.py | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/portage/package/ebuild/fetch.py b/lib/portage/package/ebuild/fetch.py
> index f687a01a2..c2bfca61c 100644
> --- a/lib/portage/package/ebuild/fetch.py
> +++ b/lib/portage/package/ebuild/fetch.py
> @@ -663,8 +663,15 @@ def get_mirror_url(mirror_url, filename, mysettings, cache_path=None):
>  	else:
>  		tmpfile = '.layout.conf.%s' % urlparse(mirror_url).hostname
>  		try:
> -			if fetch({tmpfile: (mirror_url + '/distfiles/layout.conf',)},
> -					mysettings, force=1, try_mirrors=0):
> +			if mirror_url[:1] == "/":
> +				tmpfile = os.path.join(mirror_url, "layout.conf")
> +				mirror_conf.read_from_file(tmpfile)
> +			elif fetch(
> +				{tmpfile: (mirror_url + "/distfiles/layout.conf",)},
> +				mysettings,
> +				force=1,
> +				try_mirrors=0,
> +			):
>  				tmpfile = os.path.join(mysettings['DISTDIR'], tmpfile)
>  				mirror_conf.read_from_file(tmpfile)
>  			else:
> @@ -683,8 +690,10 @@ def get_mirror_url(mirror_url, filename, mysettings, cache_path=None):
>  	path = mirror_conf.get_best_supported_layout(filename=filename).get_path(filename)
>  	if urlparse(mirror_url).scheme in ('ftp', 'http', 'https'):
>  		path = urlquote(path)
> -	return mirror_url + "/distfiles/" + path
> -
> +	if mirror_url[:1] == "/":
> +		return os.path.join(mirror_url, path)
> +	else:
> +		return mirror_url + "/distfiles/" + path
>  
>  def fetch(myuris, mysettings, listonly=0, fetchonly=0,
>  	locks_in_subdir=".locks", use_locks=1, try_mirrors=1, digests=None,
> @@ -1212,7 +1221,7 @@ def fetch(myuris, mysettings, listonly=0, fetchonly=0,
>  				if distdir_writable and ro_distdirs:
>  					readonly_file = None
>  					for x in ro_distdirs:
> -						filename = os.path.join(x, myfile)
> +						filename = get_mirror_url(x, myfile, mysettings)
>  						match, mystat = _check_distfile(
>  							filename, pruned_digests, eout, hash_filter=hash_filter)
>  						if match:
> -- 
> 2.26.2
>

Comment 1 Zac Medico gentoo-dev

2021-06-06 17:48:31 UTC

We'll need some helper command(s) for working with the mirror layout. For example, we need a way to locate the distfiles associated with a particular ebuild, and those distfiles can originate from multiple local mirror with different layouts. I imagine something like the portageq metadata command that outputs a mapping of distfiles to paths will work. A versatile format for the output is newline delimited json, and in addition we can support alternative form(s) of columnar output if there's demand.