| Summary: | [PATCH] emerge pretend fetch: no option to print urls of the missing required files | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Darragh Bailey <daragh.bailey> |
| Component: | Core - Interface (emerge) | Assignee: | Portage team <dev-portage> |
| Status: | CONFIRMED --- | ||
| Severity: | enhancement | CC: | dan, navid.zamani |
| Priority: | High | Keywords: | Inclusion |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=358923 https://bugs.gentoo.org/show_bug.cgi?id=293830 https://bugs.gentoo.org/show_bug.cgi?id=293827 https://bugs.gentoo.org/show_bug.cgi?id=629762 https://bugs.gentoo.org/show_bug.cgi?id=629048 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 377365, 120474 | ||
| Attachments: |
Adds --fetch-missing option to emerge and portage.py
adds patch to files and updates portage-2.0.51.22-r3.ebuild to apply it |
||
Actually I made a mistake with the comment "The behaviour that occurs with emerge is that -fp and -Fp return the url list of all files for the package." Which is not quite true, -F will also get files in the package even if you don't have the required use options enabled. Given this extra detail that I've worked out, I've changed this to an enhancement request to include an extra option instead to limit the list to just missing files to be printed. Updated summary to reflect this Created attachment 70692 [details, diff]
Adds --fetch-missing option to emerge and portage.py
This patch provides support for a new option that allows the user to request
emerge to only disable the urls of the missing files when it is used in
conjunction with --pretend and ( --fetchonly or --fetch-all-uris )
This should be useful for those people that generate a list of files to be
downloaded in order to retreive them using systems with better internet
connections.
Created attachment 70728 [details, diff]
adds patch to files and updates portage-2.0.51.22-r3.ebuild to apply it
This patch can be applied to the portage area to add in patch 70692 and have
the 2.0.51.22-r3 ebuild automatically apply it. Other than that its pretty much
the same as the previous patch.
At least with this applied when I do an "emerge -pfuD --fetch-missing world
2>1& | awk '/^[fh].*/ { print $1 }'" I get a list of urls of just the files I'm
missing and need to retreive. Instead of all the files for each package that
have to be updated.
If you're still after this please merge the patches and check them against a current portage version. Since moving to DSL, this is no longer of importance to me. Can be closed WONTFIX unless the developers see a benefit. I think we could safely exclude files that are already fetched without adding a separate option. Please see bug #358923 for a suggestion of a more advanced solution. (Basically emerge would generate a script, using FETCHCOMMAND, that you could run on another host.) |
from emerge man page --fetchonly (-f) Instead of doing any package building, just perform fetches for all packages (the main package as well as all dependencies). --fetch-all-uri (-F) Instead of doing any package building, just perform fetches for all packages (the main package as well as all dependencies), grabbing all potential files. This means that -f just gets the missing files and -F gets all files anyway. When combining -p, pretend with these commands the expected behaviour is -fp just print the missing file urls, and -Fp print all urls for files in the packages. The behaviour that occurs with emerge is that -fp and -Fp return the url list of all files for the package. If you to use an alternative system to download files based on the urls generated by these options you can end up with downloading an entire package when your only missing the latest patchset and all the other files are already in your local distfiles area. This can result in downloading updates using alternative system (i.e. one will high bandwidth) resulting in 300MB+ of downloads where the total new data is 100MB Reproducible: Always Steps to Reproduce: 1. emerge -pfuD world 2>&1 | awk '/^[fh].*/ {print $1}' > ~/urllist 2. 3. Actual Results: Contents of ~/urllist ftp://ftp.heanet.ie/pub/gentoo/distfiles/portage-2.0.51.22.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/shadow-4.0.7.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12i-cryptoapi-losetup.patch.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/loop-AES-v3.1b.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12i.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12r.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/info-2.4.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/html-2.4.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/Python-2.4.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/perl-5.8.6.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/automake-1.9.6.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-libidn-2.3.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-2.3.5-patches-1.13.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-linuxthreads-2.3.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-fedora-20041219T2331.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-manpages-2.3.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-2.3.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-infopages-2.3.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/libIDL-0.8.6.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/ORBit2-2.12.3.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonobo-2.10.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/qt-x11-immodule-unified-qt3.3.4-20041203.diff.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/qt-x11-free-3.3.4.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18-utf8.diff.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/linux-2.6.13.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.base.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.extras.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/ruby-1.8.3.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/onigd2_4_2.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/gnome-vfs-2.10.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonoboui-2.10.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtk-engines-2.6.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/texinfo-4.8.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/xscreensaver-4.22.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/dia-0.94.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/mime-types-4.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/pine-4.63-r3-chappa-all.patch.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/pine4.63.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/ipc-1.1.3.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/moz_libart_lgpl-2.3.10p1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/enigmail-0.92.0.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-source.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/xv-3.10a.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/xv-3.10a-jumbo-patches-20050501.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gedit-2.10.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/xine-lib-1.0.1.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/xine-lib-patches-5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gcalctool-5.5.42.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gdm-2.8.0.3.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gentoo-gdm-theme-r2.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gdm-2.8-gentoo-files-r2.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gnome-backgrounds-2.10.2.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/moz_libart_lgpl-2.3.10p1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.2.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/firefox-1.0.7-source.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtkspell-2.0.11.tar.gz Expected Results: The following files are actually all that are required ftp://ftp.heanet.ie/pub/gentoo/distfiles/loop-AES-v3.1b.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12r.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/html-2.4.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/Python-2.4.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-2.3.5-patches-1.13.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/libIDL-0.8.6.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/ORBit2-2.12.3.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonobo-2.10.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/qt-x11-immodule-unified-qt3.3.4-20041203.diff.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18-utf8.diff.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/linux-2.6.13.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.base.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.extras.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/ruby-1.8.3.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/onigd2_4_2.tar.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonoboui-2.10.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtk-engines-2.6.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/mime-types-4.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/pine-4.63-r3-chappa-all.patch.gz ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-source.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/xv-3.10a-jumbo-patches-20050501.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gedit-2.10.5.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gdm-2.8.0.3.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gnome-backgrounds-2.10.2.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/moz_libart_lgpl-2.3.10p1.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.2.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/firefox-1.0.7-source.tar.bz2 ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtkspell-2.0.11.tar.gz In this case there is 67MB less in file sizes to download. In other cases depending on the original size it can be much higher.