Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 17049 - Support batched downloads
Summary: Support batched downloads
Status: RESOLVED WONTFIX
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: High enhancement (vote)
Assignee: Nicholas Jones (RETIRED)
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2003-03-07 18:14 UTC by Sven Vermeulen (RETIRED)
Modified: 2011-10-30 22:37 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sven Vermeulen (RETIRED) gentoo-dev 2003-03-07 18:14:56 UTC
In some cases (fi when all downloads are done through non-anonymous ftp or even
ssh - sftp) it is more interesting to download all distfiles in one time (ie
batched) instead of one at a time (which is done now).

This is especially true when the password should be entered by the admin...
entering the password once and downloading all files (batched) compared to
entering the password for each file (currently) is in this case preferable.

This doesn't seem hard to do... some small changes in portage.py:fetch() and a
new option. Has anyone already done such a thing? If not, is it adviseable to
put some small effort in it or do you rather not see this feature in Portage?

Reproducible: Always
Steps to Reproduce:
Comment 1 SpanKY gentoo-dev 2003-03-08 02:05:09 UTC
`emerge world -fup` gives you a list of all files to be d/l-ed ...
or you could do `emerge world -uf` to only fetch the packages ...
Comment 2 Sven Vermeulen (RETIRED) gentoo-dev 2003-03-08 02:30:18 UTC
I didn't say something works or not...

emerge -fu world will download all packages one by one, not batched. It does the equivalent of - in case wget is used:

wget uri1
wget uri2
wget uri3
...
wget uriN

And what I was talking about is a batched download:

cat > filelist << EOF
uri1
uri2
uri3
...
uriN
EOF
wget -i filelist
Comment 3 Sven Vermeulen (RETIRED) gentoo-dev 2003-03-08 02:46:43 UTC
Reopened due to misidentification of problem
Comment 4 SpanKY gentoo-dev 2003-03-08 23:57:25 UTC
as i said in comment #1 then: 
`emerge world -fup` gives you a list of all files to be d/l-ed ... 
 
so you could conceivably do `emerge world -fup > filelist` 
Comment 5 Sven Vermeulen (RETIRED) gentoo-dev 2003-03-09 05:37:46 UTC
Yes, however then you would still have to parse that output so that only 1 URI for each package is used, then download it batched, then for all packages that were not available try to do a batched download with a different URI etc...

This isn't a one-liner anymore, therefor I thought it could be handy to import it into Portage. But if not, no sweat, I'll create a script that does the same... just thought some other ppl could find it handy.
Comment 6 SpanKY gentoo-dev 2003-03-09 10:04:17 UTC
well, depending on the client, cant you tell it to not d/l a file if it already exists ?  and/or tell it to attempt a resume ?
Comment 7 Sven Vermeulen (RETIRED) gentoo-dev 2003-03-09 10:43:44 UTC
Yes it does, kewl, then this could actually be an improvement for Gentoo's portage: if Portage would use batched downloads per default when "emerge -f" is used, then it would not have the overhead of (re)starting 'fetchcommand' and (re)initializing the connection for each package.
Comment 8 Sven Vermeulen (RETIRED) gentoo-dev 2003-05-08 12:05:25 UTC
seems like this is an enhancement which doesn't have much fallback from the user community. Ppl who want such a batched download can write a script based on the "emerge -fp <package>" output.