Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 1661 - Parallelising the download > compile > download > compile cycle (FEATURES=parallel-fetch)
Summary: Parallelising the download > compile > download > compile cycle (FEATURES=par...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: x86 Linux
: Low enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 2832 3944 4553 5868 5917 6413 8535 8953 9288 9441 9851 10142 11769 12595 13362 14802 15790 18332 18408 18656 19367 21192 27153 28465 29402 29763 36532 36659 38760 40500 47124 71619 75367 85324 87296 89412 96554 97861 129955 132490 186047 (view as bug list)
Depends on: 1184
Blocks: 377365 2765 7127
  Show dependency tree
 
Reported: 2002-04-10 17:52 UTC by Aniruddha Shankar
Modified: 2012-01-07 08:38 UTC (History)
63 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
patch for /usr/lib/python2.2/site-packages/portage.py (portage 2.0.47-r10) (portage.py.patch,1.44 KB, patch)
2003-05-07 04:23 UTC, Josep Sanjuas
Details | Diff
updated patch for /usr/lib/python2.2/site-packages/portage.py (portage 2.0.48-r1) (portage.py-2.0.48-r1.diff,1.42 KB, patch)
2003-06-16 08:19 UTC, Josep Sanjuas
Details | Diff
Patch for portage.py that uses environment variables for the lockdir (portage.py-2.0.48-r1-patch.diff,1.42 KB, patch)
2003-06-27 16:11 UTC, Chris Case
Details | Diff
updated patch for /usr/lib/portage/pym/portage.py (portage 2.0.50-r8) (portage.py-patch-2.0.50-r8,995 bytes, patch)
2004-07-09 12:59 UTC, Josep Sanjuas
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aniruddha Shankar 2002-04-10 17:52:34 UTC
Compiling and downloading of software takes a very long time in Gentoo when the hardware (e.g. 
Celeron 850 / 128 mb ) and the connection speeds (256 - 64 kbps link from India) are limited. Instead 
of having to wait for a particular compile to finish before the download of the next package, a 
download for packages that will be needed if all goes well with the compile of the current package 
(as most do) should start and be "backgrounded". 

To illustrate, gcc, gettext and glibc are 
downloaded while binutils is being built.
Comment 1 Brandon Low (RETIRED) gentoo-dev 2002-07-18 20:41:03 UTC
I'm going to pass this one to carpaski as he has already written the code once
and is working on a rewrite to synch up with drobbins' latest code... it worked
and various versions are available at
http://gentoo.twobit.net/scripts/portage/threaded
Nick, get to work! :-D
Comment 2 Nicholas Jones (RETIRED) gentoo-dev 2002-07-18 21:40:36 UTC
*** Bug 4553 has been marked as a duplicate of this bug. ***
Comment 3 Nicholas Jones (RETIRED) gentoo-dev 2002-07-18 21:42:06 UTC
*** Bug 2832 has been marked as a duplicate of this bug. ***
Comment 4 Nicholas Jones (RETIRED) gentoo-dev 2002-08-04 01:34:55 UTC
*** Bug 3944 has been marked as a duplicate of this bug. ***
Comment 5 Nicholas Jones (RETIRED) gentoo-dev 2002-08-04 01:38:38 UTC
*** Bug 5868 has been marked as a duplicate of this bug. ***
Comment 6 Nicholas Jones (RETIRED) gentoo-dev 2002-08-04 01:43:35 UTC
From my [gentoo-dev] mail:
http://lists.gentoo.org/pipermail/gentoo-dev/2002-August/014047.html

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
I've been working on code to make downloading occur continuously
during the emerge process. I have new, clean, and working code
that needs a bit more testing that I can give it. I'd appreciate
some testing. It includes a couple subtle enhancements also. Try
'emerge -ef world' to see the "(X of Y)" counts.

At present, the output is a little noisy because wget and builds
are happening at the same time. I HIGHLY recommend adding "-q"
to wget's FETCH_COMMAND in make.globals/make.conf.

Please CC me (carpaski@gentoo.org) with any bugs that you think
you come across. PLEASE follow the directions in the README.txt.

http://gentoo.twobit.net/portage/threaded/

It does as follows:

1. Downloads begin and do not pause until they are completed.
2. Merges occur in order as the required files finish downloading.
3. Failed merges will stop the merging of packages at that point,
   but fetching will continue until all files are downloaded.
4. Failed fetches will be announced, but WILL NOT stop the fetching.
   Merges will continue until the failed fetch is reached. Fetches
   will continue until all fetches have completed.
5. Failures in the fetching and/or failure in the merge will be
   noted when both the fetching and the merging are completed.
   Only one notice for the fetches is given. Only one notice
   of the failed merge is given.

Thanks,
-- Nicholas Jones
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Comment 7 Nicholas Jones (RETIRED) gentoo-dev 2002-08-04 08:59:57 UTC
Another note... After trolling the forums I saw that the use of
prozilla across mirrors was improving download rates. You might
consider trying this as well.
Comment 8 Nicholas Jones (RETIRED) gentoo-dev 2002-08-04 21:23:29 UTC
Harmless --fetchonly traceback fixed in -r2
Comment 9 Nicholas Jones (RETIRED) gentoo-dev 2002-08-05 04:53:13 UTC
Update to 2.0.24 available
Comment 10 Nicholas Jones (RETIRED) gentoo-dev 2002-08-05 23:13:45 UTC
2.0.25 update available. Any problems? Goodness/Badness reports?
http://gentoo.twobit.net/portage/threaded/
Comment 11 Nicholas Jones (RETIRED) gentoo-dev 2002-08-06 17:35:29 UTC
2.0.26 available.

Any commentary on functionality... Things that might be nice...
Anything you can think of would be appreciated.
Comment 12 Nicholas Jones (RETIRED) gentoo-dev 2002-08-10 02:01:33 UTC
2.0.27 is up.
Comment 13 Bruce A. Locke (RETIRED) gentoo-dev 2002-08-11 20:06:34 UTC
It appears to work fine though wget and compile output being spit out together
doesn't make for good reading :)
Comment 14 Maurizio Disimino 2002-08-12 00:47:53 UTC
worked fine for me, too.
Tried with gentoo1.4b stage1.

sed '22,23s:"$: -a /tmp/port_log":' /etc/make.global
and the output is ok.
Comment 15 delta407 2002-08-12 17:08:41 UTC
It works under Portage 2.0.27, except that one package failed to download which
killed the rest of the install process.
Comment 16 Francisco León 2002-08-12 20:59:40 UTC
huh? how does this work? i am emerging world and i dont see it downloading 
something as it is compiling.
Comment 17 SpanKY gentoo-dev 2002-08-13 10:32:23 UTC
*** Bug 6413 has been marked as a duplicate of this bug. ***
Comment 18 Paul de Vrieze (RETIRED) gentoo-dev 2002-08-13 14:14:48 UTC
Is there any estimation as when we can see this functionality merged into the
official portage?
Comment 19 Maurizio Disimino 2002-08-14 01:34:42 UTC
Ximian bind # emerge bind-9.2.2_rc1.ebuild
Calculating dependencies ...done!
<<< Fetching daemon started (1 packages)
<<< Fetching (1 of 1) net-dns/bind-9.2.2_rc1.
!!! doebuild: /usr/portage/net-dns/bind/bind-9.2.2_rc1.ebuild not found.
!!! Fetching failed on net-dns/bind-9.2.2_rc1.
<<< Fetching ended.

!!! Fetching failed.

Ximian bind # pwd
/usr/myportage/net-dns/bind
Ximian bind # grep PORTDIR_OVERLAY /etc/make.conf
PORTDIR_OVERLAY="/usr/myportage"
Comment 20 Nicholas Jones (RETIRED) gentoo-dev 2002-09-04 02:41:57 UTC
Posted 2.0.35 ... I still need to add in PORTAGE_OVERLAY support to it.
Comment 21 SpanKY gentoo-dev 2002-09-05 14:19:22 UTC
*** Bug 5917 has been marked as a duplicate of this bug. ***
Comment 22 SpanKY gentoo-dev 2002-09-30 00:11:24 UTC
*** Bug 8535 has been marked as a duplicate of this bug. ***
Comment 23 robert longhausen 2002-10-02 19:57:24 UTC
Is this stable?  Is it planned to be integrated into the "official" version
soon?  What can we do to help?
Comment 24 Nicholas Jones (RETIRED) gentoo-dev 2002-10-13 01:05:19 UTC
Well... This isn't exactly stable. There's a shared memory issue
caused by globals in portage, and I'll have to find some time
to make a fair large change to portage before I cna get this
working properly.
Comment 25 Nicholas Jones (RETIRED) gentoo-dev 2002-10-18 12:56:25 UTC
*** Bug 9288 has been marked as a duplicate of this bug. ***
Comment 26 SpanKY gentoo-dev 2002-10-20 23:19:49 UTC
*** Bug 8953 has been marked as a duplicate of this bug. ***
Comment 27 SpanKY gentoo-dev 2002-10-21 13:51:31 UTC
*** Bug 9441 has been marked as a duplicate of this bug. ***
Comment 28 SpanKY gentoo-dev 2002-10-29 01:37:48 UTC
*** Bug 9851 has been marked as a duplicate of this bug. ***
Comment 29 SpanKY gentoo-dev 2002-11-04 11:17:35 UTC
*** Bug 10142 has been marked as a duplicate of this bug. ***
Comment 30 Nicholas Jones (RETIRED) gentoo-dev 2002-12-08 12:40:18 UTC
*** Bug 11769 has been marked as a duplicate of this bug. ***
Comment 31 Miguel Sousa Filipe 2002-12-17 22:35:06 UTC
Are there any still any development being done to add this feature to portage?
what is the status if so? 
Comment 32 Martin Holzer (RETIRED) gentoo-dev 2002-12-22 17:48:05 UTC
*** Bug 12595 has been marked as a duplicate of this bug. ***
Comment 33 Martin Holzer (RETIRED) gentoo-dev 2003-01-06 07:59:53 UTC
*** Bug 13362 has been marked as a duplicate of this bug. ***
Comment 34 Robert Coie (RETIRED) gentoo-dev 2003-01-31 15:10:47 UTC
If this functionality is not going to be implemented in standard Portage any time soon, might it be a good idea to at least have Portage implement advisory locking, so that multiple instances of emerge will refuse to run?  This would be especially helpful on machines administerered by several people.
Comment 35 Tobias Sager 2003-02-07 07:55:44 UTC
Is this going to be added soon?
I very much would appreciate that.
Comment 36 Nicholas Jones (RETIRED) gentoo-dev 2003-02-17 06:04:15 UTC
*** Bug 14802 has been marked as a duplicate of this bug. ***
Comment 37 Robert Cole 2003-03-20 15:32:17 UTC
If this can't be implemented anytime soon how about at least a fetch before compile option?

ie. emerge --fetch_only kde fetches all packages and deps then exit. 

and another option to emerge --fetchall_then_compile kde

?? It's a good compromise I think.
Comment 38 Maik Schreiber 2003-03-20 15:38:59 UTC
Try emerge -f kde.
Comment 39 Bjarke Istrup Pedersen (RETIRED) gentoo-dev 2003-03-27 15:03:10 UTC
Maybe it could be done this way. 
 
User runs: emerge kde -something. 
 
portage downloads the first package, and starts compiling it, while it downloads the second 
package. 
Lets say it finishes compiling package #12 before downloading of package #13 is complete. 
Normally this would mess up portage, but how about this: 
portage waits until downloading of package #13 is complete, and begins compiling it once 
done, while downloading package #14. 
If this is done from the same emerge command, it wouldn't mess up portage. 
 
I have no idea on how to switch between looking at the compiling and looking at the 
downloading, but that should't be difficult to fix. 
Comment 40 Martin Holzer (RETIRED) gentoo-dev 2003-03-28 18:38:17 UTC
*** Bug 18332 has been marked as a duplicate of this bug. ***
Comment 41 Nicholas Jones (RETIRED) gentoo-dev 2003-04-02 02:34:53 UTC
*** Bug 18408 has been marked as a duplicate of this bug. ***
Comment 42 Nicholas Jones (RETIRED) gentoo-dev 2003-04-02 18:16:59 UTC
*** Bug 18656 has been marked as a duplicate of this bug. ***
Comment 43 Josep Sanjuas 2003-04-16 15:28:24 UTC
I have written a little patch that seems to work fine for me. You can emerge -fu world in one tty then emerge -u world on another one. Before compiling, the second emerge will wait until files are downloaded (if they are being downloaded). And obviously, the first emerge will keep downloading while the other compiles. Given its simplicity, I'd suggest adding this little funtionality into portage. Use the patch at your own risk, though.

Patch for /usr/lib/python2.2/site-packages/portage.py, tested only on portage 2.0.47-r10 follows: (please back up your original portage.py, just in case)

13a14,38
> #/kl4rk #TODO: check permissions on the locks directory and on locked files?
> 
> lockdir="var/tmp/portage/"
> def lockfile(filename,verbose=1):
>       "Locks on /lockdir/filename, creates the lock file if necessary"
>       #fd=open(root+lockdir+md5.new(filename).hexdigest(),'w')
>       fd=open(root+lockdir+filename,'w')
>       if verbose:
>               try:
>                       fcntl.lockf(fd,fcntl.LOCK_EX|fcntl.LOCK_NB)
>               except IOError:
>                       print green(" * ")+"Another process is holding a lock on '"+filename+"'"
>                       print green(" * ")+"Waiting.. ",
>                       sys.stdout.flush()
>                       fcntl.lockf(fd,fcntl.LOCK_EX)
>                       print "Resuming."
>       else:
>               fcntl.lockf(fd,fcntl.LOCK_EX)
>       return fd
> 
> def unlockfile(fd):
>       fcntl.lockf(fd,fcntl.LOCK_UN)
>       fd.close()
> #kl4rk/
> 
999a1025
>       
1062a1089,1093
>                       
>               #/kl4rk
>               myfilefd=lockfile(myfile)
>               #kl4rk/
>               
1122a1154,1156
>               #/kl4rk
>               unlockfile(myfilefd)
>               #kl4rk/
1235c1269
< 
---
>       
Comment 44 Martin Holzer (RETIRED) gentoo-dev 2003-04-26 11:17:50 UTC
*** Bug 19999 has been marked as a duplicate of this bug. ***
Comment 45 Stuart Herbert (RETIRED) gentoo-dev 2003-04-26 11:31:57 UTC
Josep,

That patch of yours works for me.  It'd be great to see this added to the official portage asap ;-)

Thanks,
Stu
Comment 46 Josep Sanjuas 2003-05-07 04:23:11 UTC
Created attachment 11635 [details, diff]
patch for /usr/lib/python2.2/site-packages/portage.py (portage 2.0.47-r10)

This is a new version of the patch i posted before, but with better coding
style and without stupid comments. Just to make it more serious.
Comment 47 Martin Holzer (RETIRED) gentoo-dev 2003-05-22 13:45:04 UTC
*** Bug 15790 has been marked as a duplicate of this bug. ***
Comment 48 Josep Sanjuas 2003-06-16 08:19:55 UTC
Created attachment 13367 [details, diff]
updated patch for /usr/lib/python2.2/site-packages/portage.py (portage 2.0.48-r1)

Update of the patch I submitted.

Should I contact anyone to ask them to merge this into the official portage?
Did anyone else try this? I loved to read Stuart's reply :-)
Comment 49 Chris Case 2003-06-27 16:11:08 UTC
Created attachment 13944 [details, diff]
Patch for portage.py that uses environment variables for the lockdir

The location of the "lockdir" in the original patchfile relies on
"/var/log/portage".  This causes it not to work on systems who have changed the
"PORTAGE_TMPDIR" in their "make.conf".	I have updated the "lockdir" to take
from "settings["BUILD_PREFIX"]" which is "PORTAGE_TMPDIR/portage".  This allows
continued functionality if "PORTAGE_TMPDIR" or the "portage" directories change
names.	The variable "lockdir" has also been placed inside the "def lockfile"
since the variable "settings" isn't available until after a call to "doebuild"
and it doesn't really need a global scope as far as I can tell.
Comment 50 Chris Case 2003-06-27 16:15:04 UTC
Eh, my comment looks horrible, bare with me...my first patch ever ;) 

-Chris
Comment 51 Tobias Sager 2003-06-28 03:10:00 UTC
Yes, Josep Sanjuas patch works for me nicely.
Although it's not very userfriendly...
Is there no possibility to integrate the calling of "emerge -fU @args" into the normal "emerge -U @args"?

Would love to test this patch.
Comment 52 SpanKY gentoo-dev 2003-07-09 12:19:48 UTC
*** Bug 21192 has been marked as a duplicate of this bug. ***
Comment 53 SpanKY gentoo-dev 2003-08-22 17:42:18 UTC
*** Bug 27153 has been marked as a duplicate of this bug. ***
Comment 54 Michael Pyne 2003-08-26 20:34:16 UTC
I have made a multi-threaded C program that takes care of this idea.  See (http://forums.gentoo.org/viewtopic.php?p=486763#486763).  This would perhaps be a good workaround until this is finally integrated into Portage.  It depends on being able to run emerge to install a package while using emerge to download one.  I'm not sure of whether or not that will b0rk Portage, but it seems to work OK for me, at least until Portage can do it by itself.
Comment 55 SpanKY gentoo-dev 2003-09-11 18:06:21 UTC
*** Bug 28465 has been marked as a duplicate of this bug. ***
Comment 56 Marius Mauch (RETIRED) gentoo-dev 2003-09-21 09:27:35 UTC
*** Bug 19367 has been marked as a duplicate of this bug. ***
Comment 57 Benjamin Coles 2003-09-22 16:35:56 UTC
*** Bug 18408 has been marked as a duplicate of this bug. ***
Comment 58 SpanKY gentoo-dev 2003-09-22 21:46:19 UTC
*** Bug 29402 has been marked as a duplicate of this bug. ***
Comment 59 Marius Mauch (RETIRED) gentoo-dev 2003-09-27 13:03:18 UTC
*** Bug 29763 has been marked as a duplicate of this bug. ***
Comment 60 SpanKY gentoo-dev 2003-12-26 11:42:29 UTC
*** Bug 36532 has been marked as a duplicate of this bug. ***
Comment 61 Nicholas Jones (RETIRED) gentoo-dev 2003-12-31 01:52:53 UTC
*** Bug 36659 has been marked as a duplicate of this bug. ***
Comment 62 Nicholas Jones (RETIRED) gentoo-dev 2003-12-31 01:55:39 UTC
This will be coming about in the not-too-distant future.
Lockfiles are in place for the DB already and the globals
are disappearing rapidly.
Comment 63 Marius Mauch (RETIRED) gentoo-dev 2004-01-19 15:02:44 UTC
*** Bug 38760 has been marked as a duplicate of this bug. ***
Comment 64 Nicholas Jones (RETIRED) gentoo-dev 2004-02-05 09:36:03 UTC
*** Bug 40500 has been marked as a duplicate of this bug. ***
Comment 65 Marius Mauch (RETIRED) gentoo-dev 2004-04-07 12:00:01 UTC
*** Bug 47124 has been marked as a duplicate of this bug. ***
Comment 66 david.antliff 2004-07-02 19:40:43 UTC
Is this enhancement dead or maintained and managed elsewhere?
Comment 67 Disenchanted (RETIRED) gentoo-dev 2004-07-06 15:47:20 UTC
Can this be implemented? everything i read sounds like portage is really needing this sort of duality of merging/fetching to make one's life easier.

At this time i can not start a emerge -ef world and then after a few package have downloaded safely start a emerge -e world, as it happens your internet connection slows down the day you run a emerge -ef and emerge -e at the same time and eventually your PC gets fast enough and yay the -e world caught up to -ef world and both touch the same tarball, md5 bad, both fail and to make it all better emerge --resume decided to resume the -ef world to be what needs to be done.

i will be trying the supplied patches once this emerge -e world finishes

the last important comment was from december 31st last year, what is the current official stance on file locking/parallel merging/fetching (threaded or not)

it would even help for this case to have the option to tell emerge to --retry-fetch and the other to --skip-fetch to signal what to do in the case a bad md5 (from a dual accessed file on download or otherwise)
one would happily skip over it and keep on fetching, the other would retry until it gets a good md5, i guess --retry-fetch=max_retries should be specified for those times the md5 is just bad, period

maybe this approach is simpler/easier to implement?

now i have more reason to hope this -e world finishes soon

Comment 68 Josep Sanjuas 2004-07-09 12:59:42 UTC
Created attachment 35074 [details, diff]
updated patch for /usr/lib/portage/pym/portage.py (portage 2.0.50-r8)

This is almost a one-line-patch because newer versions already have the
locking/unlocking functions defined. Works for me
Comment 69 Brian Harring gentoo-dev 2004-08-01 03:42:32 UTC
Josep- that's a helluva lot cleaner then what I worked up.
You're right, it pretty much comes down to a lock and unlock call.
Also, need to lock /var/log/emerge.log to prevent messages getting mixed during writing.

Either way, commiting fetch locking shortly- it shouldn't eat your fs, but if it does tough cookies (and file a comment here) :)

Note this isn't a final solution- a final solution would be jstubb's mergemanager.  I'm adding the locking to try and prevent corruption of distfiles (and the log file when running multiple emerges).  If an emerge instance hits an active lock in fetch, it waits for the lock- this is kind of ugly behaviour, since there is no indication of what's happening.  I'd rather not dirty up fetch w/ comments along the lines of 'acquiring lock on %s', 'releasing lock on %s'.  I'll leave that to a correct solution.  Either way, been testing it w/ (emerge -f blah &); emerge blah, locks are working fine.

InCVS, should show in pre14.
Comment 70 SpanKY gentoo-dev 2004-11-17 18:14:24 UTC
*** Bug 71619 has been marked as a duplicate of this bug. ***
Comment 71 Brian Harring gentoo-dev 2004-12-05 14:41:52 UTC
Update from cvs, atm, parallelized fetching/compiling is commited under FEATURES="parallel-fetch", and requires FEATURES="distlocks" to be on.
Comment 72 SpanKY gentoo-dev 2004-12-23 06:04:56 UTC
*** Bug 75367 has been marked as a duplicate of this bug. ***
Comment 73 SpanKY gentoo-dev 2005-03-15 05:46:42 UTC
*** Bug 85324 has been marked as a duplicate of this bug. ***
Comment 74 SpanKY gentoo-dev 2005-03-30 10:15:42 UTC
*** Bug 87296 has been marked as a duplicate of this bug. ***
Comment 75 Jason Stubbs (RETIRED) gentoo-dev 2005-04-17 07:04:05 UTC
*** Bug 89412 has been marked as a duplicate of this bug. ***
Comment 76 Thomas Bettler 2005-04-17 07:12:35 UTC
Portage should have a better conception. (poormans proposal)

The few actions for each ebuild should consist of atomic actions.
- Fetch
- Unpack
- Compile
- Install
- Collision-protect (optional)
- Merge
- Config
(or whatever it might look like)

Merging multiple packages meens playing with dependencies. The dependencies should be calculated on the actions.
examples to explain:
- Fetching: always allowed
- Unpacking: unpack x packages to the queue
- Compiling: compile x packages parallel - if dependencies are merged
- Install: always allowed
- Merge: always / or maybe restriction: not parallel
- Config: maybe on a seperate terminal, that admins easily track all the important messages...

These atomic actions could be handled in a continuously modular way on multiple queues. Locking and Unlocking needs to be handled clearly.
examples:
- qt and xorg-x11 could not be compiled together (but fetched, unpacked)
- openoffice and gnomemeeting could be comipled parallel

Advantages:
- distcc would behave far better (even incl. casas of -j1)
- installing packages on a computerfarm would be a picknick
- parallel fetching / merging resolved
- maybe better load balance of diskaccess and cpu
- would make gentoo a favorite distro for compiling reasons

Disadvantages:
- Modular rewrite of portage with great effor needed (so no earlier than in portage 4.0)
- The dependencies of the packages need to be handled while running

New features
Pro and Con: (would be great but needs lot of work and conception)
- emerge.log needs new format to idetify the atomic actions, 
but parallel merges could be better tracked than today
- a new feature "load logger" could maybe be integrated, calculating the GUs used to compile
serving as a base for stats on packages compiling (might be tricky to integrate it into distcc)
- based on stats the progress could be estimated.
Comment 77 Jason Stubbs (RETIRED) gentoo-dev 2005-04-17 07:26:35 UTC
The plan is to essentially have two "atoms". One is fetch and the other is build. A separate upper limit will be specifiable for both and portage will do as many in parallel as it can based on dependencies.
Comment 78 Thomas Bettler 2005-04-17 07:56:55 UTC
Theese plans sound cool, even if there are only two atomic steps.

When might it be integrated, is there already some testing version around, not mentioned in this bug?
Comment 79 Jason Stubbs (RETIRED) gentoo-dev 2005-04-17 08:13:20 UTC
I did have it working at one stage, but the code went in a completely different direction to where portage _was_ going. CVS HEAD currently has a single parallel fetch thread which will be available in the next major portage version. The major version following that will contain many fixes for dependency calculation and will enable proper parallelization of all tasks. 

Timelines? Next major version will probably go stable in around 6 months. Can't really make any reasonable estimate on the following version at this stage.
Comment 80 Thomas Bettler 2005-04-17 08:22:08 UTC
Thanks for your info.

Of course it's estimated 6months, 
but I'd be lucky to see this fixed once in future.
Comment 81 Jakub Moc (RETIRED) gentoo-dev 2005-06-19 15:28:17 UTC
*** Bug 96554 has been marked as a duplicate of this bug. ***
Comment 82 Jakub Moc (RETIRED) gentoo-dev 2005-07-03 23:36:36 UTC
*** Bug 97861 has been marked as a duplicate of this bug. ***
Comment 83 Jason Stubbs (RETIRED) gentoo-dev 2005-07-28 07:25:12 UTC
Putting a hold on feature requests for portage as they are drowning out the 
bugs. Most of these features should be available in the next major version of 
portage. But for the time being, they are just drowning out the major bugs and 
delaying the next version's progress. 
 
Any bugs that contain patches and any bugs for etc-update or dispatch-conf can 
be reopened. Sorry, I'm just not good enough with bugzilla. ;) 
Comment 84 Brian Harring gentoo-dev 2006-02-09 14:57:05 UTC
reopening...
Comment 85 Brian Harring gentoo-dev 2006-02-09 14:57:46 UTC
released in 2.1_pre*, enabled via FEATURES="distlocks parallel-fetch"
Comment 86 Jakub Moc (RETIRED) gentoo-dev 2006-04-14 09:00:19 UTC
*** Bug 129955 has been marked as a duplicate of this bug. ***
Comment 87 Sean Crago 2006-04-14 12:40:13 UTC
Why isn't parralel-fetch on by default yet?
Comment 88 Zac Medico gentoo-dev 2006-04-14 12:54:39 UTC
(In reply to comment #87)
> Why isn't parralel-fetch on by default yet?

Portage currenly has no memory of which distfiles it has checksummed and their timestamps.  Due to this issue, if you're emerging a large list of packages and you need to restart it (for example if one of the packages fails to build), the parallel-fetch causes all the files to be re-checksummed needlessly.  The extra io overhead could be annoying in some circumstances, so it's disabled by default.

Comment 89 Jakub Moc (RETIRED) gentoo-dev 2006-05-06 12:54:55 UTC
*** Bug 132490 has been marked as a duplicate of this bug. ***
Comment 90 Zac Medico gentoo-dev 2007-07-21 01:29:01 UTC
*** Bug 186047 has been marked as a duplicate of this bug. ***
Comment 91 Cristian Tarsoaga 2009-01-16 21:18:01 UTC
Hello,
I have a simple question: is it possible to resume a failed fetch in the background?
I start an emerge world, it downloads x packages then inet connection fails. The background fetch stops while the compilation continues. At one point, inet is back while emerge still compiles. Can I restart the fetchonly in the background? (without interrupting the compilation)

 Thanks,
    Chris
Comment 92 Zac Medico gentoo-dev 2009-01-16 21:33:47 UTC
(In reply to comment #91)

You can't restart it "in the background" of running emerge instance, but you can do this:

  emerge --resume --fetchonly

(In reply to comment #87)
> Why isn't parralel-fetch on by default yet?

It's enabled by default now, in >=portage-2.1.5.
Comment 93 Vu Tran Kien 2012-01-07 08:30:19 UTC
Just something that comes out of my mind:
Why don't emerge uses many mirror as advantages in downloading? 
For example assume emerge is downloading a huge package from 1st mirror, during that time (simultaneously) , It starts download the 2nd package from 2nd mirror, and so on (depending on some variable configured like MAX_PARALLEL_FETCH=3)

Just some of my oppinion, what do you think?
Comment 94 Zac Medico gentoo-dev 2012-01-07 08:38:13 UTC
(In reply to comment #93)
> Just something that comes out of my mind:
> Why don't emerge uses many mirror as advantages in downloading? 
> For example assume emerge is downloading a huge package from 1st mirror, during
> that time (simultaneously) , It starts download the 2nd package from 2nd
> mirror, and so on (depending on some variable configured like
> MAX_PARALLEL_FETCH=3)
> 
> Just some of my oppinion, what do you think?

I've thought of adding an emerge --fetch-jobs option, which is similar to --jobs but controls the number of parallel-fetch threads instead of build threads. You'd be able to configure it as a default setting in the EMERGE_DEFAULT_OPTS variable.