Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 519098 - sys-devel/make: builds fail randomly with: make: INTERNAL: Exiting with 1 jobserver tokens available; should be 3!
Summary: sys-devel/make: builds fail randomly with: make: INTERNAL: Exiting with 1 job...
Status: RESOLVED OBSOLETE
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: AMD64 OS X
: Normal normal (vote)
Assignee: Gentoo's Team for Core System packages
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-08-05 04:33 UTC by iloveosxzfs
Modified: 2020-12-14 20:26 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build log (ncurses-5.9-r3-build.log,109.63 KB, text/plain)
2014-08-05 04:34 UTC, iloveosxzfs
Details
emerge info (emerge-info-ncurses-5.9-r3.txt,15.82 KB, text/plain)
2014-08-05 04:37 UTC, iloveosxzfs
Details
emerge pqv (emerge-pqv-ncurses-5.9-r3.txt,245 bytes, text/plain)
2014-08-05 04:37 UTC, iloveosxzfs
Details

Note You need to log in before you can comment on or make changes to this bug.
Description iloveosxzfs 2014-08-05 04:33:46 UTC
Building sys-libs/ncurses-5.9-r3 can result in "make: INTERNAL: Exiting with 1 jobserver tokens available; should be 3!" which can be worked around by de-parallelizing the build:

cd ${EPREFIX}
mkdir -p etc/portage/env
echo 'MAKEOPTS="-j1"' > etc/portage/env/deparallelize
echo '=sys-libs/ncurses-5.9-r3 deparallelize' >> etc/portage/package.env
Comment 1 iloveosxzfs 2014-08-05 04:34:07 UTC
Created attachment 382294 [details]
build log
Comment 2 iloveosxzfs 2014-08-05 04:37:30 UTC
Created attachment 382296 [details]
emerge info
Comment 3 iloveosxzfs 2014-08-05 04:37:42 UTC
Created attachment 382298 [details]
emerge pqv
Comment 4 Christoph Junghans (RETIRED) gentoo-dev 2014-08-05 14:40:56 UTC
Doesn't look like a prefix problem to me.
Comment 5 Michael Haubenwallner (RETIRED) gentoo-dev 2014-08-06 16:47:32 UTC
Could be a make problem as well.
Comment 6 Fabian Groffen gentoo-dev 2014-08-06 18:54:09 UTC
make[1]: *** read jobs pipe: No such file or directory.  Stop.
make[1]: *** Waiting for unfinished jobs....
Makefile:109: recipe for target 'all' failed

Same error as in all your previous parallel make bugs.  Please stop filing new ones.  Something's wrong with your make or system.
Comment 7 Michael Haubenwallner (RETIRED) gentoo-dev 2014-08-07 07:50:39 UTC
Comment on attachment 382294 [details]
build log

FWIW: Similar problems were seen on Interix. For AIX, I had to mask gmake-4.0 for stability problems for now.
But even on Linux sometimes I do see parallel build issues - which feel related to portage's stream-redirection to some degree...


Also, there's this one as the first output of configure:
configure: line 1109: 33371 Segmentation fault: 11  ( PATH=".;."; conftest.sh ) 2>&5
Comment 8 iloveosxzfs 2014-08-07 09:06:26 UTC
@Fabian and @Michael, so it looks like the theory that this is a make version 4  issue may be partially but not completely correct.

Core i7
hw.ncpu: 8
MAKEOPTS="-j5"
CFLAGS="${CFLAGS} -O2 -pipe"

1. qlop -l | awk '{print $NF;}' | sort | uniq | wc -l
453

2. I've needed to deparallelize
app-portage/eix-0.30.1
dev-lang/ruby-2.1.1
net-misc/rsync-3.1.0
app-arch/libarchive-3.1.2-r1
dev-vcs/git-2.0.0-r2
dev-libs/gmp-5.1.3-r1
sys-devel/gettext-0.18.3.2

3.
qlop -l | awk '{print $NF;}' | sort | uniq | while read atom; do for cp in app-portage/eix dev-lang/ruby net-misc/rsync app-arch/libarchive dev-vcs/git dev-libs/gmp ; do [[ "$atom" =~ .*"$cp".* ]] && echo "$atom" ; done ; done
app-arch/libarchive-3.1.2-r1
app-portage/eix-0.30.1
app-portage/eix-0.30.2
dev-lang/ruby-1.9.3_p448-r1
dev-lang/ruby-1.9.3_p545
dev-lang/ruby-2.0.0_p247-r1
dev-lang/ruby-2.0.0_p451
dev-lang/ruby-2.1.1
dev-libs/gmp-5.1.3-r1
dev-vcs/git-1.9.1
dev-vcs/git-1.9.3
dev-vcs/git-2.0.0
dev-vcs/git-2.0.0-r1
dev-vcs/git-2.0.0-r2
dev-vcs/git-2.0.2
dev-vcs/git-2.0.4
net-misc/rsync-3.1.0
net-misc/rsync-3.1.0-r1
net-misc/rsync-3.1.1

So out of 453, that would be about 19 affected (about 4%), assuming every version of each package was affected.

4. I just downgraded to sys-devel/make-3.82-r4 from sys-devel/make-4.0-r1 and attempted to emerge gmp, rsync, gettext, ruby, libarchive, git, and eix.

Results:
dev-libs/gmp-5.1.3-r1 failed
net-misc/rsync-3.1.1 succeeded
app-arch/libarchive-3.1.2-r1 succeeded
sys-devel/gettext-0.18.3.2 succeeded
dev-vcs/git-2.0.4 succeeded
app-portage/eix-0.30.2 succeeded
dev-lang/ruby-2.1.1 failed

5. So it looks like make 4 may be the problem for rsync, libarchive, gettext, git, and six, and that gmp and ruby have parallelization issues unrelated to make 4.

@Fabian, while it appears that you are correct that some of my problems have actually been make specific, almost a third of the problems were not specific to make, or at least not specific to make version 4. Though I suppose you cannot rule out that there may be some other make problem affecting both make version 3 and make version 4, or that there may be some problem affecting my particular system, it seems more likely that the gmp and ruby issues are indeed specific to those packages. Therefore, assuming that the downgrade to make 3.8 is sufficient to work around the make-specific issues, it seems I should actually continue reporting parallelization issues I encounter going forward, or do you disagree?

6. I suppose someone needs to create a bug report about the make 4 problem.
Comment 9 iloveosxzfs 2014-08-07 12:47:29 UTC
Further analysis seems to indicate the problem may have nothing to do with make versions.

I tried build 5 times with make4 and 5 times with make3. In no case did rsync or git fail, so let's ignore those.

(make4 failures, make3 failures)
eix (3, 2)
gettext (1, 1)
gmp (2, 3)
libarchive (4, 2)
ruby (3, 2)
total (13, 10)

and of course
(make4 successes, make3 successes)
eix (2, 3)
gettext (4, 4)
gmp (3, 2)
libarchive (1, 3)
ruby (2, 3)
total (12, 15)

make4 13 failures, 12 successes
make3 10 failures, 15 successes

Note that no package shows 0 failures with make3.

So not sure where this leaves us.
Comment 10 Michael Haubenwallner (RETIRED) gentoo-dev 2014-08-08 11:34:10 UTC
(In reply to iloveosxzfs from comment #9)
> make4 13 failures, 12 successes
> make3 10 failures, 15 successes

Does it make a difference with make-3.81 eventually?

AFAIR, there's one problem with Solaris and NFS we don't carry the patch for, as it is fixed in 3.82 (https://savannah.gnu.org/bugs/?18680).
Comment 11 iloveosxzfs 2014-08-08 13:21:06 UTC
I was comparing sys-devel/make-3.82-r4 and sys-devel/make-4.0-r1
Comment 12 SpanKY gentoo-dev 2015-08-12 08:37:44 UTC
is this still an issue w/make-4.1 ?
Comment 13 Fabian Groffen gentoo-dev 2020-12-14 20:26:51 UTC
we've a bit further now, haven't been able to reproduce the problem