Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 380295

Summary: [patch] sys-apps/portage: Enhanced output for emerge --pretend --fetchonly
Product: Portage Development Reporter: Miroslav Los <milamby>
Component: Conceptual/Abstract IdeasAssignee: Portage team <dev-portage>
Status: UNCONFIRMED ---    
Severity: enhancement CC: kingjon3377
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 377365    
Attachments: portage-fetch_listheader.patch
Sample downloader script

Description Miroslav Los 2011-08-22 22:55:01 UTC
The current output of emerge --pretend --fetchonly (also applies to --fetch-all-uri to a lesser degree) is only of limited use to people who wish to use it for alternative source downloading (usually on a different computer).

Firstly, it lists a source even if it is successfully downloaded. Secondly, and more importantly, it prints neither the target file name, nor its size and digest(s). This further complicates successful fetching.

I previously used a rather complicated script to alleviate these problems, but now I found a better solution, presented in my patch.

With this patch, fetch does not list any URIs for files that are present in DISTDIR and their checksums are verified.

It also adds a new settings variable, PORTAGE_FETCH_LISTHEADER, that can be specified, like FETCHCOMMAND, either as an environment variable in the shell, or in /etc/make.conf. The contents of this variable are prepended to the space-separated list of URIs for the file.

In this string, the following variables get expanded:
- file: the target file name (may be different from the url name)
- size: the size in bytes
- RMD160, SHA256, SHA1: all digests from the manifest
- nl, tab: convenience variables for inserting a newline or tab

The variable defaults to "${nl}", matching current emerge output separating uli lists by blank lines (but still just listing missing files).

I have also written a sample download script, utilising this functionality.

Reproducible: Always





Additionally, I looked into a few bug reports regarding these matters, which are all tracked by bug #377365 (among others). Some of them may be closed, either by this patch or a simple WONTFIX. I do not wish to pimpspam this patch on all of them, so I comment on them here, hoping the competent developers will sort them out:

Bugs #293827 and #293830 from the same reporter, also bug #358923 only wish to list one URI per file. This can be readily solved by standard utilities, ex. :
emerge -pfq ... | cut -f 1 (easy)
emerge -pfq ... | sed '/^\s*$/d;s/\s.*//' (better)
emerge -pfq ... | tr ' ' '\n' | sed '/^\s*$/d' | wget -nc -i - (best, note -nc)

This does not even require my patch (although my solution presented in a later attachment/comment is superior). 

Bug #109304, especially comment 6, were an inspiration for skipping verifiably fetched files without any more options to portage.
Comment 1 Miroslav Los 2011-08-22 23:08:50 UTC
Created attachment 284269 [details, diff]
portage-fetch_listheader.patch

This patch does not address listing the same file for multiple packages. This can be solved bu simply piping through sort -u, with a caveat that the URI list may be AFAIR in a randomized order. Alternatively, one may use something like this:

PORTAGE_FETCH_LISTHEADER="${file}${tab}${size}${tab}" emerge -pfq ... | sort -uk 1,2

This patch does not handle documentation. Also, it does not register PORTAGE_FETCH_LISTHEADER anywhere (which may be a requirement in portage).
Comment 2 Miroslav Los 2011-08-22 23:19:40 UTC
Created attachment 284271 [details]
Sample downloader script

For lack of a better venue, I present a sample downloader script utilising the patch. It requires a custom setting for PORTAGE_FETCH_LISTHEADER, either in make.conf or on the commandline.

Simplest usage:
Target machine:
PORTAGE_FETCH_LISTHEADER="${file}${tab}${size}${tab}${SHA256}${tab}" emerge -pfq -uDn --with-bdeps=y @world | sort -uk 1,1 >/media/drive/wishlist

Fetcher machine:
cd /media/drive; KEEP_GOING=y bash getem
Comment 3 Miroslav Los 2011-08-22 23:41:43 UTC
Bad mistake in the examples I gave: the value of PORTAGE_FETCH_LISTHEADER must use escaped \$ signs both in make.conf and on the command line of course. That is:
PORTAGE_FETCH_LISTHEADER="\${file}\${tab}\${size}\${tab}\${SHA256}\${tab}"