Bug 438112 – eselect-1.3.2: detection of GNU sed is too fragile (was: sys-apps/portage-2.2.0_alpha136 -- PATH that contains ._portage_reinstall_. in all packages are set when run emerge -e world.)

• Status: RESOLVED FIXED
• Product: Gentoo Hosted Projects

[Description Yuta SATOH 2012-10-12 13:53:36 UTC]

Usually, when you run emerge, this will be set to PATH.

PATH="/usr/lib/portage/bin/ebuild-helpers/bsd:/usr/lib/portage/bin/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-gentoo-freebsd9.1/gcc-bin/4.6.3"

However, if you target sys-apps/portage itself as emerge -e world and emerge portage freebsd-lib, a different path is set.

PATH="/var/tmp/portage/._portage_reinstall_.qr7enn/bin/ebuild-helpers/bsd:/var/tmp/portage/._portage_reinstall_.qr7enn/bin/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-gentoo-freebsd9.1/gcc-bin/4.6.3"


I discovered this problem that the operation of app-admin/eselect-1.3.2-r1 is not correct.
eselect to find a sed from the PATH at compile time.
And, writes to /usr/share/eselect/libs/core.bash .

# eselect profile list
/usr/share/eselect/libs/core.bash: line 108: /var/tmp/portage/._portage_reinstall_.staz86/bin/ebuild-helpers/bsd/sed: No such file or directory



Reproducible: Always

Steps to Reproduce:
1. FEATURES=noclean emerge -qe world
>>> Emerging (1 of 315) sys-freebsd/freebsd-lib-9.1_rc2

2. Ctrl + C
3. grep PATH= /var/tmp/portage/sys-freebsd/freebsd-lib-9.1_rc2/temp/environment
<snip>
declare -x PATH="/var/tmp/portage/._portage_reinstall_.qr7enn/bin/ebuild-helpers/bsd:/var/tmp/portage/._portage_reinstall_.qr7enn/bin/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-gentoo-freebsd9.1/gcc-bin/4.6.3"
declare -x PYTHONPATH="/usr/lib/portage/pym"
                export LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:${libdir}"
                export LD_LIBRARY_PATH="${libdir}"
</snip>



Expected Results:  
Packages other than sys-apps/portage set ordinary PATH.

[Comment 1 Zac Medico 2012-10-12 16:31:12 UTC]

@eselect: Is it really necessary to hardcode the path to sed at build time?

@bsd: Should we continue to assume that the /usr/lib/portage/bin/ebuild-helpers/bsd/sed wrapper is needed? If the system sed is not GNU sed, then I guess so.

[Comment 2 Zac Medico 2012-10-12 16:34:06 UTC]

As a workaround, do this:

  emerge -e @world --exclude=sys-apps/portage

[Comment 3 Zac Medico 2012-10-12 16:41:43 UTC]

(In reply to comment #0)
> Expected Results:  
> Packages other than sys-apps/portage set ordinary PATH.

As noted, you can get that behavior by using `emerge -e @world --exclude=sys-apps/portage`. Since bug 390819, portage does not restart itself after it updates itself, so it runs from a temporary copy of itself so all packages built in the same run. Having portage restart itself is too error prone, so I don't want to support that.

[Comment 4 Ulrich Müller 2012-10-12 18:11:13 UTC]

(In reply to comment #1)
> @eselect: Is it really necessary to hardcode the path to sed at build time?

We need to make sure that we have GNU sed, and it is done by looking up sed and gsed in PATH at configure time. Standard autoconf tests (like AC_PROG_SED) do the same traversing of PATH.

This code in eselect hasn't changed since 2005, BTW.

[Comment 5 Zac Medico 2012-10-12 18:22:48 UTC]

(In reply to comment #4)
> (In reply to comment #1)
> > @eselect: Is it really necessary to hardcode the path to sed at build time?
> 
> We need to make sure that we have GNU sed, and it is done by looking up sed
> and gsed in PATH at configure time. Standard autoconf tests (like
> AC_PROG_SED) do the same traversing of PATH.
> 
> This code in eselect hasn't changed since 2005, BTW.

If you can make it prefer gsed over sed, then that should fix it as long as the FreeBSD system has gsed.

[Comment 6 Ulrich Müller 2012-10-12 19:09:23 UTC]

(In reply to comment #5)
> If you can make it prefer gsed over sed, then that should fix it as long as
> the FreeBSD system has gsed.

Too fragile, for my taste. I'd rather traverse $prefix/bin:/usr/bin:/bin:$PATH (instead of just $PATH) and check for sed and gsed in each directory.

Where is gsed located on FreeBSD systems? /usr/bin/gsed?

The bug would have also been prevented if the Portage helper script wasn't impersonating as GNU sed:
/usr/lib/portage/bin/ebuild-helpers/bsd/sed --version
GNU sed version 4.2.1

[Comment 7 Zac Medico 2012-10-12 19:17:23 UTC]

(In reply to comment #6)
> The bug would have also been prevented if the Portage helper script wasn't
> impersonating as GNU sed:
> /usr/lib/portage/bin/ebuild-helpers/bsd/sed --version
> GNU sed version 4.2.1

It's whole purpose is to impersonate GNU sed though, because BSD sed is incompatible with most build scripts which expect GNU tools.

[Comment 8 Ulrich Müller 2012-10-12 19:48:50 UTC]

(In reply to comment #6)
> I'd rather traverse $prefix/bin:/usr/bin:/bin:$PATH (instead of just $PATH)
> and check for sed and gsed in each directory.

Thinking about it again, this would probably break things on prefix systems.

[Comment 9 Fabian Groffen 2012-10-13 06:15:31 UTC]

for sure, it would find host-provided sed before the prefix one

in Prefix Portage's sed script is killed (e.g. never installed), because it harms more than it fixes for us (we always provide GNU sed as 'sed')

[Comment 10 Yuta SATOH 2012-10-13 09:06:44 UTC]

(In reply to comment #2)
> As a workaround, do this:
> 
>   emerge -e @world --exclude=sys-apps/portage

Thank you for your comment.
I was able to work around the problem in your way.


BSD Team,
BTW, seems to hard-code sed as well sys-devel/libtool-2.4.2 .
Should we announce how to update properly　?

# grep SED= /usr/bin/libtool*
/usr/bin/libtool:SED="/var/tmp/portage/._portage_reinstall_.s2pu3x/bin/ebuild-helpers/bsd/sed"
/usr/bin/libtoolize:: ${SED="/var/tmp/portage/._portage_reinstall_.s2pu3x/bin/ebuild-helpers/bsd/sed"}

[Comment 11 Ulrich Müller 2012-10-13 09:57:53 UTC]

(In reply to comment #0)
> Usually, when you run emerge, this will be set to PATH.
> 
> PATH="/usr/lib/portage/bin/ebuild-helpers/bsd:/usr/lib/portage/bin/ebuild-
> helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/
> bin:/usr/x86_64-gentoo-freebsd9.1/gcc-bin/4.6.3"

And eselect-1.3.2 will pick up /usr/lib/portage/bin/ebuild-helpers/bsd/sed which already isn't right. It should use /usr/bin/gsed instead.

The code for sed detection is here:
<http://git.overlays.gentoo.org/gitweb/?p=proj/eselect.git;a=blob;f=acinclude.m4;h=bde52c04b531f7bf20b7b16693784f8934f0e1d6;hb=fe1de0b73d86de33ffadb7c0d0b8aa37c89c0887>
It was looping over PATH components and testing for sed or gsed in each. The _last_ sed found in PATH would be returned, or if none found, the first gsed. I've no idea why it was done in such a strange way.


(In reply to comment #4)
> This code in eselect hasn't changed since 2005, BTW.

I've replaced this old custom code by a straightforward
   AC_PATH_PROGS(SED, [gsed sed])
followed by a check for GNU sed, version 4.0 or later. See this commit:
<http://git.overlays.gentoo.org/gitweb/?p=proj/eselect.git;a=commit;h=89efdeb043758ed074c58f389b98bc23b9241dfa>

Note that autoconf's standard AC_PROG_SED also doesn't work: It checks for both sed and gsed indeed, but it gets confused by the presence of Portage's wrapper script in PATH (and returns its location instead of the real sed).

[Comment 12 Ulrich Müller 2012-10-23 18:13:55 UTC]

Should be fixed in eselect-1.3.3.
Thank you for reporting.