Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 31794 - large enviroment during emerges and other actils causes xargs to fail
Summary: large enviroment during emerges and other actils causes xargs to fail
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Highest critical (vote)
Assignee: Gentoo Linux bug wranglers
Depends on:
Blocks: 21438 23569 25535 30900 31378
  Show dependency tree
Reported: 2003-10-22 23:02 UTC by Robin Johnson
Modified: 2003-12-24 09:44 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2003-10-22 23:02:19 UTC
ok, firstly a quick description of the problem:
a large number of packages either utilize xargs in their ebuilds, or inside their own build systems.
with the ever-growing environment, we are starting to hit the arbitrary 20kb limit that GNU xargs imposes.

The problem is coming up because _every_ function and variable that is defined in the shell takes up space in the environment, and when you have large eclasses and ebuilds, things start to go beyond the 20kb limit. The OpenOffice ebuild has this problem presently (bug 31378). I was working on some major changes to my php eclass today, and afterwards, the xargs invocation I have in mod_php was causing weird compile failures.

124 ebuilds currently use xargs directly.

Baselayout was previously effected with this problem, and the invocation:
( find /var/lock -type f -print0 | xargs -0 rm -f -- 1>&2 )
got changed to:
( find /var/lock -type f -exec rm -f {} \; )

While similiar variations on this can be made in all our ebuilds, we would still get the problems from the package's build systems.
I don't believe this is the answer then, also as my mod_php case required a long sed expression to replace what was being done very simply before.

either needs to reduce the size of the enviroment it's passing in (dangerous i think, and won't nessicarily solve the problem), or we need to adjust the 20kb limit in xargs. see lines 303-304 of findutils-4.1.20/xargs/xargs.c for their limit.
Comment 1 Paul de Vrieze (RETIRED) gentoo-dev 2003-10-23 02:48:50 UTC
I have made a patch and a new ebuild and am ready to commit it. I want to
do this soon as I get tons of bugs related to this. As I now know the cause
I have no problem with fixing xargs for this.
Comment 2 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2003-10-23 07:25:39 UTC
no objections to it here.
i've moved the limit to 64kb for xargs on my personal machines, but i'm wondering
where we should put the proper limit.
Comment 3 Paul de Vrieze (RETIRED) gentoo-dev 2003-10-23 13:07:36 UTC
I committed 4.1.7-r5 (stable) and 4.1.20-r1 (unstable) with limits of 50k.
This indeed is quite arbitrary, but it solves all current problems. However
we might want to consider to look at removing fully the broken behaviour
from xargs, and make it allocate the buffer it needs.
Comment 4 SpanKY gentoo-dev 2003-10-23 21:51:50 UTC
does debian fix this ?  or perhaps redhat ?
Comment 5 Paul de Vrieze (RETIRED) gentoo-dev 2003-10-24 00:52:27 UTC
I don't know. I needed a temporary fix rather fast as the stable openoffice-1.1.0
was failing. (When I marked it stable it wasn't an issue yet)
Comment 6 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2003-10-24 01:25:24 UTC
I checked Debian, Redhat, Mandrake, and none of them presently include a
patch for it. Both Debian and Redhat included patches for it previously,
as they seem to have had some other tool that had problems just like portage
for us, but they since deprecated those tools and removed the patches (more
than a year ago).
Comment 7 Martin Schlemmer (RETIRED) gentoo-dev 2003-10-26 09:32:47 UTC
An big problem was from baslayout which due to misbehaving bash
did a 'set -a'.  I removed this in CVS again - not sure though if my issues
will be fixed, but should help here (and one or two other bugs).
Comment 8 Nicholas Jones (RETIRED) gentoo-dev 2003-12-23 21:24:05 UTC
Not much that portage can do about this... and I'd imagine it's handled.
Someone want to determine if it is, and close this?
Comment 9 SpanKY gentoo-dev 2003-12-23 21:46:43 UTC
paul, we all set on this now ?
Comment 10 Paul de Vrieze (RETIRED) gentoo-dev 2003-12-24 09:44:15 UTC
There seem to be no problems anymore. I don't know however if that will stay this way. I guess it depends on the environment size increases of portage. I'll mark it later so that we can look into actual fundamental fixes when problems arise.