Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 4551 - Parallel emerges gives problems
Summary: Parallel emerges gives problems
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: x86 Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 13359 emerge_locking 16115 49063 (view as bug list)
Depends on:
Blocks:
 
Reported: 2002-07-04 17:35 UTC by Ole Tange
Modified: 2011-10-30 22:19 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ole Tange 2002-07-04 17:35:39 UTC
If package X is compiling and another emerge depends on X then the compile breaks.

Examples:

emerge xfree &
(wait for until compile starts)
emerge openoffice
Comment 1 Seemant Kulleen (RETIRED) gentoo-dev 2002-07-04 17:40:01 UTC
Is this really a bug?
Comment 2 Ole Tange 2002-07-04 18:48:19 UTC
I come from Sorcery-linux. In Sorcery-linux you can install any packages
simultaniously. If any of the packages depend on eachother they wait for the
others to complete.

If you have multiple CPU's you can decrease compile time if it is possible to
install packages in parallel.

You *can* install some packages in parallel today, but you cannot install any
packages in parallel. You do not get any sensible error messages saying: You
can/should not install packages in parallel.

My definition of a bug: If a program does not do what I expect it to do, then it
is a bug - especially if the program does not tell me what went wrong and what I
can do to fix it. This is clearly a bug in my definition.
Comment 3 Martin Holzer (RETIRED) gentoo-dev 2003-01-06 08:04:40 UTC
*** Bug 13359 has been marked as a duplicate of this bug. ***
Comment 4 Charles Goodwin 2003-01-06 10:32:52 UTC
It's not really a bug, but it's something portage should deal with.  It's more
like a feature request.

I would disagree with Ole that it is a bug in that it's obvious to the
half-sensible linux user that compiling 2 version of the same package
concurrently will create problems.  Just because you can do it doesn't mean it's
a bug.  For example, I can delete all of /etc with rm, but that's not a bug in
rm, even though it will break my system.

You do need to be able to run concurrent emerges, for example, where you are
emerging a large package and you want to emerge a small package without
interrupting the compilation of the larger package.
Comment 5 Ole Tange 2003-01-06 11:27:35 UTC
> I would disagree with Ole that it is a bug in that it's obvious to the 
> half-sensible linux user that compiling 2 version of the same package 
> concurrently will create problems. 
 
If you look at my initial report I was not compiling 2 versions of the same 
package. I was compiling xfree and openoffice (which happened to depend on 
xfree). If we can agree that it is not fair to assume that the half-sensible 
linux user knows the whole dependency graph for everything then then we should 
also agree that this behaviour is not good. In my example it may be obvious 
that both depend on xfree, but is it obvious to you that Python depends on 
xfree? 
 
As mentioned a solution (that is used in Sorcery) is to wait untill the first 
first compile is done. It is fairly simple and works great in Sorcery. 
 
Comment 6 Mikael A 2003-01-23 11:28:38 UTC
The comments about rm is quite unecessary. Because thats what rm is 
intended to do. emerge is intended to install packages. When it doesn't 
do that correctly, it's a bug. If it's an implementation bug or an documentation 
bug is another question. 
 
 
Comment 7 SpanKY gentoo-dev 2003-02-02 15:30:09 UTC
*** Bug 14969 has been marked as a duplicate of this bug. ***
Comment 8 SpanKY gentoo-dev 2003-02-20 21:29:21 UTC
*** Bug 16115 has been marked as a duplicate of this bug. ***
Comment 9 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2003-02-20 22:05:47 UTC
would anybody object to the fix I suggested 
in bug #16115 ?

One question tho, what should be the correct 
behaviour for depancies? 
I think that while an emerge depends on 
something that is still being merged, it 
should just delay those items.
Comment 10 Charles Goodwin 2003-02-21 04:56:22 UTC
I agree.
Comment 11 Ole Tange 2003-03-30 14:04:31 UTC
Seems like a good solution, robbat2. 
 
Comment 12 James 2003-08-10 12:11:52 UTC
In my view this is a feature request, but I do think that it is a necessary requirement for happy emerging.
E.g. emerge -uf xfree
at the same time as 
emerge -uf kde

These will both start downloading xfree. If two appliactions download the same file to the same destination at once, the file becomes corrupt. The MD5 check fails, and the download starts again...thus deadlock, and both application continually re-download the same file forever.

For this alone, we definitly need a locking machanism on a per portage package basis.
With emerge doing all downloads before it compiles anything, and letting the download function skip a package if it is locked, and then come back to it after downloading other packages. If it is the last package, just print a message "Waiting for lock free" and then MD5 check it and download again if needed.

It is a well know fact that downloading 2 files in parralel is much quicker than downloading the 2 files sequencially.

So, in my view, this is a much needed feature. "package locking." and "download all before build"

Comment 13 SpanKY gentoo-dev 2004-05-05 17:33:13 UTC
*** Bug 49063 has been marked as a duplicate of this bug. ***
Comment 14 Jacob Joseph 2004-06-21 14:26:01 UTC
I am the poster of bug 49063, which was marked a duplicate of this bug.  While my issue is with multiple emerges across a cluster of machines with a common portage dir, the locking issue remains the same here.  I'm unclear why there is resistance to using file locks, whether over NFS or not.  Two years is a long time for any bug to be unresolved.

With respect to SpanKY's comment that file locking over nfs is a bad idea, a quick read of 'man open' will indicate: "The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g.,  incorporating  host-name  and  pid),  use  link(2) to make a link to the lockfile."

-Jacob
Comment 15 Nicholas Jones (RETIRED) gentoo-dev 2005-02-25 21:14:45 UTC
This should be handled as of a stable release a few months ago.