| Summary: | [patch] sys-apps/portage: Enhanced output for emerge --pretend --fetchonly | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Miroslav Los <milamby> |
| Component: | Conceptual/Abstract Ideas | Assignee: | Portage team <dev-portage> |
| Status: | UNCONFIRMED --- | ||
| Severity: | enhancement | CC: | kingjon3377 |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 377365 | ||
| Attachments: |
portage-fetch_listheader.patch
Sample downloader script |
||
Created attachment 284269 [details, diff]
portage-fetch_listheader.patch
This patch does not address listing the same file for multiple packages. This can be solved bu simply piping through sort -u, with a caveat that the URI list may be AFAIR in a randomized order. Alternatively, one may use something like this:
PORTAGE_FETCH_LISTHEADER="${file}${tab}${size}${tab}" emerge -pfq ... | sort -uk 1,2
This patch does not handle documentation. Also, it does not register PORTAGE_FETCH_LISTHEADER anywhere (which may be a requirement in portage).
Created attachment 284271 [details]
Sample downloader script
For lack of a better venue, I present a sample downloader script utilising the patch. It requires a custom setting for PORTAGE_FETCH_LISTHEADER, either in make.conf or on the commandline.
Simplest usage:
Target machine:
PORTAGE_FETCH_LISTHEADER="${file}${tab}${size}${tab}${SHA256}${tab}" emerge -pfq -uDn --with-bdeps=y @world | sort -uk 1,1 >/media/drive/wishlist
Fetcher machine:
cd /media/drive; KEEP_GOING=y bash getem
Bad mistake in the examples I gave: the value of PORTAGE_FETCH_LISTHEADER must use escaped \$ signs both in make.conf and on the command line of course. That is:
PORTAGE_FETCH_LISTHEADER="\${file}\${tab}\${size}\${tab}\${SHA256}\${tab}"
|
The current output of emerge --pretend --fetchonly (also applies to --fetch-all-uri to a lesser degree) is only of limited use to people who wish to use it for alternative source downloading (usually on a different computer). Firstly, it lists a source even if it is successfully downloaded. Secondly, and more importantly, it prints neither the target file name, nor its size and digest(s). This further complicates successful fetching. I previously used a rather complicated script to alleviate these problems, but now I found a better solution, presented in my patch. With this patch, fetch does not list any URIs for files that are present in DISTDIR and their checksums are verified. It also adds a new settings variable, PORTAGE_FETCH_LISTHEADER, that can be specified, like FETCHCOMMAND, either as an environment variable in the shell, or in /etc/make.conf. The contents of this variable are prepended to the space-separated list of URIs for the file. In this string, the following variables get expanded: - file: the target file name (may be different from the url name) - size: the size in bytes - RMD160, SHA256, SHA1: all digests from the manifest - nl, tab: convenience variables for inserting a newline or tab The variable defaults to "${nl}", matching current emerge output separating uli lists by blank lines (but still just listing missing files). I have also written a sample download script, utilising this functionality. Reproducible: Always Additionally, I looked into a few bug reports regarding these matters, which are all tracked by bug #377365 (among others). Some of them may be closed, either by this patch or a simple WONTFIX. I do not wish to pimpspam this patch on all of them, so I comment on them here, hoping the competent developers will sort them out: Bugs #293827 and #293830 from the same reporter, also bug #358923 only wish to list one URI per file. This can be readily solved by standard utilities, ex. : emerge -pfq ... | cut -f 1 (easy) emerge -pfq ... | sed '/^\s*$/d;s/\s.*//' (better) emerge -pfq ... | tr ' ' '\n' | sed '/^\s*$/d' | wget -nc -i - (best, note -nc) This does not even require my patch (although my solution presented in a later attachment/comment is superior). Bug #109304, especially comment 6, were an inspiration for skipping verifiably fetched files without any more options to portage.