Bug 17049 – Support batched downloads

• Product: Portage Development
• Status: RESOLVED WONTFIX

[Description Sven Vermeulen (RETIRED) 2003-03-07 18:14:56 UTC]

In some cases (fi when all downloads are done through non-anonymous ftp or even
ssh - sftp) it is more interesting to download all distfiles in one time (ie
batched) instead of one at a time (which is done now).

This is especially true when the password should be entered by the admin...
entering the password once and downloading all files (batched) compared to
entering the password for each file (currently) is in this case preferable.

This doesn't seem hard to do... some small changes in portage.py:fetch() and a
new option. Has anyone already done such a thing? If not, is it adviseable to
put some small effort in it or do you rather not see this feature in Portage?

Reproducible: Always
Steps to Reproduce:

[Comment 1 SpanKY 2003-03-08 02:05:09 UTC]

`emerge world -fup` gives you a list of all files to be d/l-ed ...
or you could do `emerge world -uf` to only fetch the packages ...

[Comment 2 Sven Vermeulen (RETIRED) 2003-03-08 02:30:18 UTC]

I didn't say something works or not...

emerge -fu world will download all packages one by one, not batched. It does the equivalent of - in case wget is used:

wget uri1
wget uri2
wget uri3
...
wget uriN

And what I was talking about is a batched download:

cat > filelist << EOF
uri1
uri2
uri3
...
uriN
EOF
wget -i filelist

[Comment 3 Sven Vermeulen (RETIRED) 2003-03-08 02:46:43 UTC]

Reopened due to misidentification of problem

[Comment 4 SpanKY 2003-03-08 23:57:25 UTC]

as i said in comment #1 then: 
`emerge world -fup` gives you a list of all files to be d/l-ed ... 
 
so you could conceivably do `emerge world -fup > filelist` 

[Comment 5 Sven Vermeulen (RETIRED) 2003-03-09 05:37:46 UTC]

Yes, however then you would still have to parse that output so that only 1 URI for each package is used, then download it batched, then for all packages that were not available try to do a batched download with a different URI etc...

This isn't a one-liner anymore, therefor I thought it could be handy to import it into Portage. But if not, no sweat, I'll create a script that does the same... just thought some other ppl could find it handy.

[Comment 6 SpanKY 2003-03-09 10:04:17 UTC]

well, depending on the client, cant you tell it to not d/l a file if it already exists ?  and/or tell it to attempt a resume ?

[Comment 7 Sven Vermeulen (RETIRED) 2003-03-09 10:43:44 UTC]

Yes it does, kewl, then this could actually be an improvement for Gentoo's portage: if Portage would use batched downloads per default when "emerge -f" is used, then it would not have the overhead of (re)starting 'fetchcommand' and (re)initializing the connection for each package.

[Comment 8 Sven Vermeulen (RETIRED) 2003-05-08 12:05:25 UTC]

seems like this is an enhancement which doesn't have much fallback from the user community. Ppl who want such a batched download can write a script based on the "emerge -fp <package>" output.