Summary: | sys-apps/portage-2.2.0_alpha45: With FEATURES="parallel-fetch", Portage should not download more than one package at a time | ||
---|---|---|---|
Product: | Portage Development | Reporter: | dwfreed <dwfreed> |
Component: | Unclassified | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | enhancement | Keywords: | InVCS |
Priority: | Normal | ||
Version: | 2.1 | ||
Hardware: | x86 | ||
OS: | All | ||
URL: | http://i.imgur.com/OOTmV.png | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 377365, 373933 | ||
Attachments: |
use fetch queue to cap fetch threads
enable prefetch for first package |
Description
dwfreed
2011-07-16 05:25:43 UTC
Created attachment 280161 [details, diff]
use fetch queue to cap fetch threads
Save as /tmp/fetch.patch and apply as follows:
patch /usr/lib/portage/pym/_emerge/EbuildBuild.py /tmp/fetch.patch
(In reply to comment #1) > Created attachment 280161 [details, diff] > use fetch queue to cap fetch threads This patch can cause waiting on the fetch queue even when everything is already fetched. We could optimize it to check if the files are already fetched and skip the fetch queue in that case. (In reply to comment #1) > Created attachment 280161 [details, diff] > use fetch queue to cap fetch threads Oddly, no matter how many packages are to be merged, this patch grabs the second package first. In the mean time, Portage is doing nothing because it hasn't grabbed the first package yet. Created attachment 280167 [details, diff] enable prefetch for first package Save as /tmp/first_fetch.patch and apply as follows: patch /usr/lib/portage/pym/_emerge/Scheduler.py /tmp/first_fetch.patch This patch applies on top of the previous patch. (In reply to comment #3) > Oddly, no matter how many packages are to be merged, this patch grabs the > second package first. In the mean time, Portage is doing nothing because it > hasn't grabbed the first package yet. This additional patch will make it fetch the first package first. The output will go to /var/log/emerge-fetch.log though, so you'll have to do something like this if you want to monitor the progress: tail -f /var/log/emerge-fetch.log We can tweak it to show the first fetch in the foreground, though the code will be a little more complex that way. (In reply to comment #2) > (In reply to comment #1) > > Created attachment 280161 [details, diff] > > use fetch queue to cap fetch threads > > This patch can cause waiting on the fetch queue even when everything is already > fetched. We could optimize it to check if the files are already fetched and > skip the fetch queue in that case. This above issue is fixed in git: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=785fe0b42a66b1d137955a4b816b3c254a1b785d (In reply to comment #4) > This additional patch will make it fetch the first package first. The output > will go to /var/log/emerge-fetch.log though, so you'll have to do something > like this if you want to monitor the progress: > > tail -f /var/log/emerge-fetch.log > > We can tweak it to show the first fetch in the foreground, though the code will > be a little more complex that way. Since I don't see a clean and easy way to make the first fetch show in the foreground, I'm not going to bother with it. People can always disable FEATURES=parallel-fetch if they really want to see the output and they can't be bothered to run tail -f. Here's a patch for fetching binary packages (PORTAGE_BINHOST): http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=25d00e7bc361a659a729d5165fe580afcbd39664 For now, I've make --jobs > 1 enable concurrent fetch: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=ef58bc7573ddce5e3a5466eea50160b81de8edf4 We can add a --fetch-jobs option later, but if we enable that for parallel-fetch then that's going to require logging changes since parallel-fetch shares emerge-fetch.log. This is fixed in 2.1.10.7 and 2.2.0_alpha46. |