Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 109304 - [PATCH] emerge pretend fetch: no option to print urls of the missing required files
Summary: [PATCH] emerge pretend fetch: no option to print urls of the missing required...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Interface (emerge) (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: Inclusion
Depends on:
Blocks: 377365 120474
  Show dependency tree
 
Reported: 2005-10-14 14:25 UTC by Darragh Bailey
Modified: 2017-12-28 04:59 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Adds --fetch-missing option to emerge and portage.py (fetch_only_required.patch,3.32 KB, patch)
2005-10-14 14:57 UTC, Darragh Bailey
Details | Diff
adds patch to files and updates portage-2.0.51.22-r3.ebuild to apply it (fetch-missing-ebuild.patch,4.05 KB, patch)
2005-10-15 06:17 UTC, Darragh Bailey
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Darragh Bailey 2005-10-14 14:25:30 UTC
from emerge man page
        --fetchonly (-f)
                Instead of doing any package building, just perform fetches for
all packages (the main
                package as well as all  dependencies).

        --fetch-all-uri (-F)
                Instead  of doing any package building, just perform fetches for
all packages (the main
                package as well as all dependencies), grabbing all potential files.

This means that -f just gets the missing files and -F gets all files anyway.
When combining -p, pretend
with these commands the expected behaviour is -fp just print the missing file
urls, and -Fp print all urls
for files in the packages.

The behaviour that occurs with emerge is that -fp and -Fp return the url list of
all files for the package. 

If you to use an alternative system to download files based on the urls
generated by these options you can end up with downloading an entire package
when your only missing the latest patchset and all the other files are already
in your local distfiles area.

This can result in downloading updates using alternative system (i.e. one will
high bandwidth) resulting in 300MB+ of downloads where the total new data is 100MB 

Reproducible: Always
Steps to Reproduce:
1. emerge -pfuD world 2>&1 | awk '/^[fh].*/ {print $1}' > ~/urllist
2.
3.

Actual Results:  
Contents of ~/urllist
ftp://ftp.heanet.ie/pub/gentoo/distfiles/portage-2.0.51.22.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/shadow-4.0.7.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12i-cryptoapi-losetup.patch.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/loop-AES-v3.1b.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12i.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12r.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/info-2.4.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/html-2.4.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/Python-2.4.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/perl-5.8.6.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/automake-1.9.6.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-libidn-2.3.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-2.3.5-patches-1.13.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-linuxthreads-2.3.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-fedora-20041219T2331.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-manpages-2.3.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-2.3.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-infopages-2.3.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/libIDL-0.8.6.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/ORBit2-2.12.3.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonobo-2.10.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/qt-x11-immodule-unified-qt3.3.4-20041203.diff.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/qt-x11-free-3.3.4.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18-utf8.diff.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/linux-2.6.13.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.base.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.extras.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/ruby-1.8.3.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/onigd2_4_2.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gnome-vfs-2.10.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonoboui-2.10.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtk-engines-2.6.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/texinfo-4.8.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/xscreensaver-4.22.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/dia-0.94.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mime-types-4.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/pine-4.63-r3-chappa-all.patch.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/pine4.63.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/ipc-1.1.3.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/moz_libart_lgpl-2.3.10p1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/enigmail-0.92.0.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-source.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/xv-3.10a.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/xv-3.10a-jumbo-patches-20050501.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gedit-2.10.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/xine-lib-1.0.1.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/xine-lib-patches-5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gcalctool-5.5.42.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gdm-2.8.0.3.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gentoo-gdm-theme-r2.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gdm-2.8-gentoo-files-r2.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gnome-backgrounds-2.10.2.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/moz_libart_lgpl-2.3.10p1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.2.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/firefox-1.0.7-source.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtkspell-2.0.11.tar.gz

Expected Results:  
The following files are actually all that are required
ftp://ftp.heanet.ie/pub/gentoo/distfiles/loop-AES-v3.1b.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/util-linux-2.12r.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/html-2.4.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/Python-2.4.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/glibc-2.3.5-patches-1.13.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/libIDL-0.8.6.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/ORBit2-2.12.3.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonobo-2.10.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/qt-x11-immodule-unified-qt3.3.4-20041203.diff.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18-utf8.diff.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/links-2.1pre18.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/linux-2.6.13.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.base.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/genpatches-2.6.13-5.extras.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/ruby-1.8.3.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/onigd2_4_2.tar.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/libbonoboui-2.10.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtk-engines-2.6.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mime-types-4.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/pine-4.63-r3-chappa-all.patch.gz
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-source.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/xv-3.10a-jumbo-patches-20050501.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gedit-2.10.5.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gdm-2.8.0.3.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gnome-backgrounds-2.10.2.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/moz_libart_lgpl-2.3.10p1.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/mozilla-1.7.12-patches-1.2.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/firefox-1.0.7-source.tar.bz2
ftp://ftp.heanet.ie/pub/gentoo/distfiles/gtkspell-2.0.11.tar.gz

In this case there is 67MB less in file sizes to download. In other cases
depending on the original size it can be much higher.
Comment 1 Darragh Bailey 2005-10-14 14:38:17 UTC
Actually I made a mistake with the comment
"The behaviour that occurs with emerge is that -fp and -Fp return the url list of
all files for the package."

Which is not quite true, -F will also get files in the package even if you don't
have the required use options enabled.

Given this extra detail that I've worked out, I've changed this to an
enhancement request to include an extra option instead to limit the list to just
missing files to be printed. 

Updated summary to reflect this
Comment 2 Darragh Bailey 2005-10-14 14:57:41 UTC
Created attachment 70692 [details, diff]
Adds --fetch-missing option to emerge and portage.py


This patch provides support for a new option that allows the user to request
emerge to only disable the urls of the missing files when it is used in
conjunction with --pretend and ( --fetchonly or --fetch-all-uris )

This should be useful for those people that generate a list of files to be
downloaded in order to retreive them using systems with better internet
connections.
Comment 3 Darragh Bailey 2005-10-15 06:17:33 UTC
Created attachment 70728 [details, diff]
adds patch to files and updates portage-2.0.51.22-r3.ebuild to apply it

This patch can be applied to the portage area to add in patch 70692 and have
the 2.0.51.22-r3 ebuild automatically apply it. Other than that its pretty much
the same as the previous patch.

At least with this applied when I do an "emerge -pfuD --fetch-missing world
2>1& | awk '/^[fh].*/ { print $1 }'" I get a list of urls of just the files I'm
missing and need to retreive. Instead of all the files for each package that
have to be updated.
Comment 4 Marius Mauch (RETIRED) gentoo-dev 2007-02-17 09:26:33 UTC
If you're still after this please merge the patches and check them against a current portage version.
Comment 5 Darragh Bailey 2009-12-01 23:38:04 UTC
Since moving to DSL, this is no longer of importance to me. Can be closed WONTFIX unless the developers see a benefit.
Comment 6 Zac Medico gentoo-dev 2010-02-27 03:13:59 UTC
I think we could safely exclude files that are already fetched without adding a separate option.