Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 279915 - Shebang line of /usr/bin/emerge is interpreted incorrectly when app-admin/eselect-python-20090801 is installed and Linux <2.6.28 or non-Linux kernel is used
Summary: Shebang line of /usr/bin/emerge is interpreted incorrectly when app-admin/ese...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords: InVCS
: 280101 280229 (view as bug list)
Depends on:
Blocks:
 
Reported: 2009-08-01 15:19 UTC by Greg
Modified: 2010-04-06 14:21 UTC (History)
26 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
output emerge --info --verbose (emerge-info.txt,9.95 KB, text/plain)
2009-08-02 14:42 UTC, Vadim Efimov
Details
emerge --info --verbose (emerge-info.txt,11.33 KB, text/plain)
2009-08-02 14:47 UTC, Mike Doty (RETIRED)
Details
emerge --info --verbose (emerge-info-verbose,8.08 KB, text/plain)
2009-08-02 15:33 UTC, David Klann
Details
emerge --info --verbose (g/fbsd-6.2) (emerge-info,8.22 KB, text/plain)
2009-08-03 12:59 UTC, Dmitri Bogomolov
Details
python.eselect.patch (python.eselect.patch,3.31 KB, patch)
2009-08-03 15:40 UTC, Arfrever Frehtes Taifersar Arahesis (RETIRED)
Details | Diff
python.eselect.patch (python.eselect.patch,3.61 KB, patch)
2009-08-03 22:18 UTC, Arfrever Frehtes Taifersar Arahesis (RETIRED)
Details | Diff
python.eselect.patch (python.eselect.diff,986 bytes, patch)
2009-08-04 01:40 UTC, Jonathan Callen (RETIRED)
Details | Diff
wrapper for python (python-wrapper.c,1.95 KB, text/plain)
2009-08-04 01:42 UTC, Jonathan Callen (RETIRED)
Details
new wrapper for python (python-wrapper.c,3.24 KB, text/plain)
2009-08-04 03:08 UTC, Jonathan Callen (RETIRED)
Details
wrapper for python (python-wrapper.c,4.10 KB, text/plain)
2009-08-04 10:27 UTC, Jonathan Callen (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Greg 2009-08-01 15:19:02 UTC
During updating 'portage system world', it failed right after eselect-python finished updating and now portage won't emerge anything.

Reproducibility: Everytime
1. Run emerge <anything>
Actual Results: Python error.
Expected Results: emerge should work

Error:
romulus ~ # emerge --info
/usr/bin/emerge: line 6: import: command not found
/usr/bin/emerge: line 8: try:: command not found
/usr/bin/emerge: line 9: import: command not found
/usr/bin/emerge: line 11: syntax error near unexpected token `('
/usr/bin/emerge: line 11: `     def exithandler(signum,frame):

Only thing I've been able get to work so far is this (emerging a package like this still fails):
romulus ~ #  /usr/bin/python /usr/bin/emerge --info
Portage 2.1.6.13 (default/linux/amd64/2008.0, gcc-4.3.3, glibc-2.10.1-r0, 2.6.26-gentoo-r2 x86_64)
=================================================================
System uname: Linux-2.6.26-gentoo-r2-x86_64-Dual_Core_AMD_Opteron-tm-_Processor_875-with-gentoo-2.0.1
Timestamp of tree: Sat, 01 Aug 2009 08:45:01 +0000
ccache version 2.4 [enabled]
app-shells/bash:     4.0_p28
dev-java/java-config: 2.1.8-r1
dev-lang/python:     2.6.2-r1
dev-util/ccache:     2.4-r8
dev-util/cmake:      2.6.4-r1
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.4.3-r3
sys-apps/sandbox:    2.0
sys-devel/autoconf:  2.13, 2.63-r1
sys-devel/automake:  1.5, 1.9.6-r2, 1.10.2, 1.11
sys-devel/binutils:  2.19.1-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6a
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="amd64 ~amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-mtune=opteron -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-mtune=opteron -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="ccache distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.mirrors.hoobly.com/ http://gentoo.arcticnetwork.ca/source/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1"
LINGUAS="en en_US ja ja_JP"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/layman/zugaina /usr/local/portage/layman/lisp /usr/local/portage/layman/nx /usr/local/portage/layman/java-overlay /usr/local/portage/layman/x11 /usr/local/myoverlay"
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="acl alsa amd64 berkdb bzip2 cjk cli cracklib crypt doc dri fortran gdbm gpm iconv immqt-bc ipv6 isdnlog midi mmx mudflap multilib ncurses nls nptl nptlonly nsplugin offensive ogg openmp pam pcre perl pppd python qt3support readline reflection session spl sse sse2 ssl sysfs tcpd unicode vim-syntax vorbis xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US ja ja_JP" USERLAND="GNU" VIDEO_CARDS="nvidia"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-01 16:37:29 UTC
Post the output of:
# readlink /bin/sh
# eselect python set python2.6
# cat /usr/bin/python
Comment 2 Lorenz Quack 2009-08-01 16:57:37 UTC
I had the same problem and I believe there is something wrong with the 
"Gentoo Python wrapper script" in /usr/bin/python
here is the content:
> cat /usr/bin/python
#!/bin/sh
# Gentoo Python wrapper script

[[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.6"
"${0%/*}/${EPYTHON:-python2.6}" "$@"


I replacing the script with a link to /usr/bin/python2.6 worked.
Then you can reverted to eselect-python-20090606 for now.
Comment 3 Greg 2009-08-01 17:05:18 UTC
romulus ~ # readlink /bin/sh
bash
romulus ~ # eselect python set python2.6
romulus ~ # cat /usr/bin/python
#!/bin/sh
# Gentoo Python wrapper script

[[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.6"
"${0%/*}/${EPYTHON:-python2.6}" "$@"

I dunno if that helps.  I'll try the link fix in the meantime.
Comment 4 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-01 17:09:35 UTC
Please add the following line before the last line:
echo "${0%/*}/${EPYTHON:-python2.6}" "$@"

Next post the output of e.g. `emerge --info`.
Comment 5 Greg 2009-08-01 17:17:39 UTC
Already did, but I can again:

(01:17:05) [gerglion@romulus ~]$ emerge --info
Portage 2.1.6.13 (default/linux/amd64/2008.0, gcc-4.3.3, glibc-2.10.1-r0, 2.6.26-gentoo-r2 x86_64)
=================================================================
System uname: Linux-2.6.26-gentoo-r2-x86_64-Dual_Core_AMD_Opteron-tm-_Processor_875-with-gentoo-2.0.1
Timestamp of tree: Sat, 01 Aug 2009 08:45:01 +0000
ccache version 2.4 [enabled]
app-shells/bash:     4.0_p28
dev-java/java-config: 2.1.8-r1
dev-lang/python:     2.6.2-r1
dev-util/ccache:     2.4-r8
dev-util/cmake:      2.6.4-r1
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.4.3-r3
sys-apps/sandbox:    2.0
sys-devel/autoconf:  2.13, 2.63-r1
sys-devel/automake:  1.5, 1.9.6-r2, 1.10.2, 1.11
sys-devel/binutils:  2.19.1-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6a
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="amd64 ~amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-mtune=opteron -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-mtune=opteron -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="ccache distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.mirrors.hoobly.com/ http://gentoo.arcticnetwork.ca/source/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1"
LINGUAS="en en_US ja ja_JP"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/layman/zugaina /usr/local/portage/layman/lisp /usr/local/portage/layman/nx /usr/local/portage/layman/java-overlay /usr/local/portage/layman/x11 /usr/local/myoverlay"
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="acl alsa amd64 berkdb bzip2 cjk cli cracklib crypt doc dri fortran gdbm gpm iconv immqt-bc ipv6 isdnlog midi mmx mudflap multilib ncurses nls nptl nptlonly nsplugin offensive ogg openmp pam pcre perl pppd python qt3support readline reflection session spl sse sse2 ssl sysfs tcpd unicode vim-syntax vorbis xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US ja ja_JP" USERLAND="GNU" VIDEO_CARDS="nvidia"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 6 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-01 17:31:55 UTC
(In reply to comment #5)

I don't need the Portage-part of output of `emerge --info`.

Previously you posted:
# emerge --info
/usr/bin/emerge: line 6: import: command not found
/usr/bin/emerge: line 8: try:: command not found
/usr/bin/emerge: line 9: import: command not found
/usr/bin/emerge: line 11: syntax error near unexpected token `('
/usr/bin/emerge: line 11: `     def exithandler(signum,frame):

I want to specifically know what is printed by that call to echo().
I'm only interested in output from when app-admin/eselect-python-20090801 is installed and /usr/bin/python is generated by app-admin/eselect-python-20090801.
Also post the output of:
$ head -n4 /usr/bin/emerge
Comment 7 Lorenz Quack 2009-08-01 18:01:11 UTC
I emerged eselect-python-20090801 again to help debugging, so here it goes:
I added the echo line and my `emerge --info` looks like this:

# emerge --info
Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

import: unable to open X server `:0' @ import.c/ImportImageCommand/367.
/usr/bin/emerge: line 8: try:: command not found
Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

import: unable to open X server `:0' @ import.c/ImportImageCommand/367.
/usr/bin/emerge: line 11: syntax error near unexpected token `('
/usr/bin/emerge: line 11: `     def exithandler(signum,frame):'


I figure the shebang somehow isn't recognised or something similar. so I directly ran python to get your echo line:

# python
/usr/bin/python2.6
Python 2.6.2 (r262:71600, Jun 11 2009, 18:25:30)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()



and here is the head of emerge:

# head -n4 /usr/bin/emerge
#!/usr/bin/python
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: emerge 13131 2009-03-18 01:43:41Z zmedico $

I hope this helps to track this bug down.
Comment 8 Robert Schultz 2009-08-01 18:10:02 UTC
I am also having this problem.

The suggestion in comment #2 to replace /usr/bin/python with a symbolic link to python2.6 fixed it for me too.

I am now continuing to do an `emerge -vuDN world` and once it's all updated I will re-run the `eselect python set python2.6` command and try it again.

If it still fails, I will add the 'echo' part to the script and post the results here.
Comment 9 Greg 2009-08-01 18:17:50 UTC
Sorry.  I'm assuming you meant to add that to /usr/bin/python.  Forgive me if
you meant otherwise.

romulus bin # cat /usr/bin/python
#!/bin/sh
# Gentoo Python wrapper script

[[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.6"
echo "${0%/*}/${EPYTHON:-python2.6}" "$@"
"${0%/*}/${EPYTHON:-python2.6}" "$@"
romulus bin # python --version
/usr/bin/python2.6 --version
Python 2.6.2
romulus bin # emerge --info
/usr/bin/emerge: line 6: import: command not found
/usr/bin/emerge: line 8: try:: command not found
/usr/bin/emerge: line 9: import: command not found
/usr/bin/emerge: line 11: syntax error near unexpected token `('
/usr/bin/emerge: line 11: `     def exithandler(signum,frame):'

Doesn't seem to echo for emerge...

romulus bin # head -n4 /usr/bin/emerge
#!/usr/bin/python
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: emerge 13152 2009-03-23 02:15:33Z zmedico $
Comment 10 David Klann 2009-08-01 18:26:54 UTC
(In reply to comment #6)
> (In reply to comment #5)
> 
> I don't need the Portage-part of output of `emerge --info`.
> 

This seems to be deeper than just "emerge". Notice this sample emerge when I have the additional "echo" line in /usr/bin/python (after running "eselect python set python2.6"):

# emerge eselect
/usr/bin/python2.6 /usr/bin/emerge eselect
Calculating dependencies ... done!

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) app-admin/eselect-1.1.2
[Errno 8] Exec format error:
   /usr/lib/portage/bin/ebuild /usr/portage/app-admin/eselect/eselect-1.1.2.ebuild fetch
 * Fetch failed for 'app-admin/eselect-1.1.2', Log file:
 *  '/var/log/portage/app-admin:eselect-1.1.2:20090801-182135.log'

>>> Failed to emerge app-admin/eselect-1.1.2, Log file:

>>>  '/var/log/portage/app-admin:eselect-1.1.2:20090801-182135.log'

head -n 4 /usr/lib/portage/bin/ebuild:
#!/usr/bin/python -O
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: ebuild 13205 2009-03-27 04:25:35Z zmedico $

Does the "-O" option inside ebuild cause trouble?
Comment 11 David Klann 2009-08-01 18:31:50 UTC
(In reply to comment #10)
> (In reply to comment #6)
> > (In reply to comment #5)
> > 
> > I don't need the Portage-part of output of `emerge --info`.
> > 
>
> Does the "-O" option inside ebuild cause trouble?
> 

Sorry to reply to my own comment. The "-O" doesn't seem to be the problem. I added "-O" to /usr/bin/emerge to see if it would break:

# head -n 4 /usr/bin/emerge
#!/usr/bin/python -O
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: emerge 13152 2009-03-23 02:15:33Z zmedico $

# emerge eselect
/usr/bin/python2.6 -O /usr/bin/emerge eselect
Calculating dependencies ... done!

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) app-admin/eselect-1.1.2
[Errno 8] Exec format error:
   /usr/lib/portage/bin/ebuild /usr/portage/app-admin/eselect/eselect-1.1.2.ebuild fetch
 * Fetch failed for 'app-admin/eselect-1.1.2', Log file:
 *  '/var/log/portage/app-admin:eselect-1.1.2:20090801-182955.log'

>>> Failed to emerge app-admin/eselect-1.1.2, Log file:

>>>  '/var/log/portage/app-admin:eselect-1.1.2:20090801-182955.log'

But why an "Exec format error"?
Comment 12 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-01 18:35:07 UTC
Reopening.
Comment 13 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-01 18:45:31 UTC
Portage developers:
Do you have any ideas for this bug?
Comment 14 Fabian Groffen gentoo-dev 2009-08-01 19:26:08 UTC
with a simple test unrelated to portage it also fails

% cat blurb 
#!/var/tmp/launch

print('hello')

% cat /var/tmp/launch 
#!/bin/bash

/usr/bin/python2.6 "$@"

% ./blurb 
Badly placed ()'s.

interestingly it takes a while at 100% cpu, like it's searching.

% time ./blurb
Badly placed ()'s.
0.225u 0.829s 0:02.15 48.3%     0+0k 18+39io 0pf+0w


So apart from the huge performance penalty this change introduced, I'd like it when we would just get back to the symlink approach.
Comment 15 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-01 19:31:30 UTC
(In reply to comment #14)
> I'd like it when we would just get back to the symlink approach.

python.eclass needs to be able to set currently used Python version for processes called during building of packages, without causing problems for other processes.
Comment 16 Fabian Groffen gentoo-dev 2009-08-01 19:32:59 UTC
Have you ever looked at how the Java folks have solved that problem?
Comment 17 Robert Schultz 2009-08-01 22:47:20 UTC
(In reply to comment #16)
> Have you ever looked at how the Java folks have solved that problem?

Java also uses a sh script that will launch the correct java. It's in /usr/bin/run-java-tool
Comment 18 Remy Blank 2009-08-02 10:49:25 UTC
A shebang line cannot call another script, it has to call a real executable. If /usr/bin/python is a script itself, all scripts that start with:

  #!/usr/bin/python

that is, most Python scripts, including emerge, will not work.
Comment 19 Jeff Huffman 2009-08-02 10:52:39 UTC
(In reply to comment #18)
> A shebang line cannot call another script, it has to call a real executable.

That's not actually true afaik... the gentoo init system is a prime example.
Comment 20 Remy Blank 2009-08-02 11:00:46 UTC
(In reply to comment #19)
> That's not actually true afaik... the gentoo init system is a prime example.

Really? My init.d scripts have #!/sbin/runscript as their first line, and /sbin/runscript is an executable on my machines...
Comment 21 Jeff Huffman 2009-08-02 11:06:16 UTC
(In reply to comment #19)

I am completely incorrect. excuse me :-/
Comment 22 Fabian Groffen gentoo-dev 2009-08-02 11:15:45 UTC
I think we can find the answer here:

http://www.in-ulm.de/~mascheck/various/shebang/

esp. "Can #! be nested?"
Comment 23 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-02 14:31:25 UTC
People, who can reproduce this bug, please *attach* (not paste in a comment) the output of `emerge --info --verbose`.
Comment 24 Vadim Efimov 2009-08-02 14:42:50 UTC
Created attachment 199927 [details]
output emerge --info --verbose

in my install, all python progs show  ImageMagick's  import output:

 # emerge --resume
Version: ImageMagick 6.5.1-0 2009-05-21 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC

Usage: import [options ...] [ file ]
------------------cut-----------------------------
the filename suffix (i.e. image.ps).  Specify 'file' as '-' for
standard input or output.
import: delegate library support not built-in `' (X11) @ import.c/ImportImageCommand/1289.
/usr/bin/emerge: line 11: syntax error near unexpected token `('
/usr/bin/emerge: line 11: `     def exithandler(signum,frame):'
Comment 25 Mike Doty (RETIRED) gentoo-dev 2009-08-02 14:47:57 UTC
Created attachment 199932 [details]
emerge --info --verbose

I hit this bug as well.  My fix was to replace '#!/usr/bin/python' with '#!/usr/bin/env python' in /usr/lib64/portage/bin/{emerge,ebuild}.
Comment 26 Fabian Groffen gentoo-dev 2009-08-02 14:50:06 UTC
somehow makes sense because that means /usr/bin/env is in the middle avoiding a shebang-loop from the kernel's point of view
Comment 27 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-02 15:26:57 UTC
I noticed that this bug seems to be reproducible only by people who use old kernels. Does upgrading to e.g. 2.6.30 fix this bug?
Comment 28 David Klann 2009-08-02 15:33:14 UTC
Created attachment 199940 [details]
emerge --info --verbose

(In reply to comment #23)
> People, who can reproduce this bug, please *attach* (not paste in a comment)
> the output of `emerge --info --verbose`.
> 

Probably have enough by now, but emerge --info --verbose attached. I ran this with python as a symlink to python2.6.
Comment 29 DrChandra the Gentoo Person 2009-08-02 15:35:25 UTC
I fixed my python (temporarily) like this:

First, show the problem. emerge was working, and eselect python breaks it.
---
   Would you like to merge these packages? [Yes/No] no

   Quitting.

   saturn bin # eselect python list
   Available python interpreters:
     [1]   python2.6 *
     [2]   python3.1
   saturn bin # eselect python set 2
   saturn bin # eselect python list
   Available python interpreters:
     [1]   python2.6
     [2]   python3.1
   saturn bin # eselect python set python3.1
   saturn bin # eselect python list
   Available python interpreters:
     [1]   python2.6
     [2]   python3.1
   saturn bin # cat /usr/bin/python
   #!/bin/bash
   # Gentoo Python wrapper script

   [[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python3.1"
   "${0%/*}/${EPYTHON:-python3.1}" "$@"
   saturn bin # emerge -NuDva world
   /usr/bin/emerge: line 8: try:: command not found
   /usr/bin/emerge: line 11: syntax error near unexpected token `('
   /usr/bin/emerge: line 11: `     def exithandler(signum,frame):'
   saturn bin #

---
I also tried eselecting python 2.6, and that didn't work either.

Here's the fix:
---
   saturn bin # rm /usr/bin/python
   saturn bin # ln -s /usr/bin/python2.6 /usr/bin/python
---
Test it:
---
   saturn bin # emerge -NuDva world

   These are the packages that would be merged, in order:

   Calculating dependencies  ^C

   Exiting on signal 2

   saturn bin #
Comment 30 DrChandra the Gentoo Person 2009-08-02 15:36:30 UTC
(In reply to comment #27)
> I noticed that this bug seems to be reproducible only by people who use old
> kernels. Does upgrading to e.g. 2.6.30 fix this bug?
> 

My kernel is 2.6.26-gentoo-r4 from gentoo-sources.
Comment 31 DrChandra the Gentoo Person 2009-08-02 15:37:40 UTC
I added my temporary fix because I am referring to this bug a lot in #gentoo. People are hitting this and asking about it.
Comment 32 David Klann 2009-08-02 15:40:01 UTC
(In reply to comment #27)
> I noticed that this bug seems to be reproducible only by people who use old
> kernels. Does upgrading to e.g. 2.6.30 fix this bug?
> 

I encounter the (undesired) behavior on systems running the following kernels:

Linux alan 2.6.30-gentoo-r4 #1 SMP
Linux coldcrossing.com 2.6.18-ovz028stab039.1-smp #1 SMP 
Linux soive 2.6.25-gentoo-r6 #3 SMP

Note that the second one in that list is running as an OpenVZ guest.
Comment 33 Joost Ruis 2009-08-02 17:22:38 UTC
same here, had to manually run eselect python update.
Comment 34 Joost Ruis 2009-08-02 17:26:35 UTC
whoops my above command doesn't fix this at all.. bad bad.
Comment 35 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-02 20:53:43 UTC
Maintainers of bash and kernel:
Do you have ideas what could cause this bug?
Comment 36 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2009-08-02 20:58:25 UTC
*** Bug 280101 has been marked as a duplicate of this bug. ***
Comment 37 Ryan Hill (RETIRED) gentoo-dev 2009-08-02 21:26:20 UTC
http://www.opengroup.org/platform/resolutions/bwg2000-004.html

The first line of the executable script shall meet all of the following
criteria otherwise the results are unspecified:

[...]

2.  The interpreter argument is an absolute pathname of an executable
    file other than an executable script.

Not a bug.
Comment 38 Doug Goldstein (RETIRED) gentoo-dev 2009-08-03 04:25:11 UTC
If you read through the kernel code, the kernel will actually rely on this requirement in the binfmt handler. Hence the issue you're experiencing. Removing base-system.
Comment 39 Dirkjan Ochtman (RETIRED) gentoo-dev 2009-08-03 08:19:32 UTC
Can we just mask eselect-python-20090801 for now? This has just been making more victims for the past two days, which seems pretty bad.
Comment 40 Dmitri Bogomolov 2009-08-03 12:59:27 UTC
Created attachment 200017 [details]
emerge --info --verbose (g/fbsd-6.2)

after replacing #!/usr/bin/python with `#!/usr/bin/env python` in /usr/bin/emerge
Comment 41 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 15:40:10 UTC
Created attachment 200027 [details, diff]
python.eselect.patch

Please test this patch.

You can apply this patch by executing:
patch /usr/share/eselect/modules/python.eselect python.eselect.patch

Next run:
eselect python set python2.6
Comment 42 David Klann 2009-08-03 16:42:16 UTC
(In reply to comment #41)
> Created an attachment (id=200027) [edit]
> python.eselect.patch
> 
> Please test this patch.
> 
> You can apply this patch by executing:
> patch /usr/share/eselect/modules/python.eselect python.eselect.patch
> 
> Next run:
> eselect python set python2.6
> 

% sudo patch /usr/share/eselect/modules/python.eselect python.eselect.patch
patching file /usr/share/eselect/modules/python.eselect
Hunk #1 FAILED at 24.
1 out of 5 hunks FAILED -- saving rejects to file /usr/share/eselect/modules/python.eselect.rej

Here's the first few lines of /usr/share/eselect/modules/python.eselect before patching:

# Copyright 1999-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: $

DESCRIPTION="Manage the /usr/bin/python and python.1 man symlinks."
MAINTAINER="python@gentoo.org"
SVN_DATE='$Date: 2009-08-01 02:03:12 +0000 (Sat, 01 Aug 2009) $'
VERSION=$(svn_date_to_version "${SVN_DATE}" )

I appplied the failed hunk to the script, ran "eselect python set python2.6", and then
"emerge -uDNva world" and all seems well.

Comment 43 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 16:47:21 UTC
(In reply to comment #42)
> patching file /usr/share/eselect/modules/python.eselect
> Hunk #1 FAILED at 24.

You probably forgot to reinstall app-admin/eselect-python before applying this patch.
Comment 44 Fabian Groffen gentoo-dev 2009-08-03 16:54:46 UTC
your patch is not really portable because it peeks into /proc
Comment 45 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 16:57:21 UTC
(In reply to comment #44)
> your patch is not really portable because it peeks into /proc

It uses execvp(EPYTHON, argv); when /proc doesn't exist.
Comment 46 Krzysztof Pawlik (RETIRED) gentoo-dev 2009-08-03 17:53:35 UTC
(In reply to comment #45)
> (In reply to comment #44)
> > your patch is not really portable because it peeks into /proc
> 
> It uses execvp(EPYTHON, argv); when /proc doesn't exist.
> 

What if local non-root user creates /tmp/python.c and asks root to run `eselect python' (or it gets ran during update)?
Comment 47 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 18:16:59 UTC
(In reply to comment #46)
> What if local non-root user creates /tmp/python.c and asks root to run
> `eselect python' (or it gets ran during update)?

Would the following be sufficient?:
rm -f /tmp/python.c || die "Deletion of /tmp/python.c failed"
touch /tmp/python.c || die "Creation of /tmp/python.c failed"
chmod 400 /tmp/python.c || die "Setting of permissions on /tmp/python.c failed"
Comment 48 Fabian Groffen gentoo-dev 2009-08-03 18:18:42 UTC
can't you use mktemp?
Comment 49 Joakim 2009-08-03 20:23:05 UTC
(In reply to comment #41)
> Created an attachment (id=200027) [edit]
> python.eselect.patch
> 
> Please test this patch.
> 
> You can apply this patch by executing:
> patch /usr/share/eselect/modules/python.eselect python.eselect.patch
> 
> Next run:
> eselect python set python2.6
> 
Yes, was hit by this as well and the patch fixed it. Also pasting just the top part of emerge --info

Portage 2.2_rc33 (default/linux/amd64/2008.0/server, gcc-4.3.3, glibc-2.8_p20080602-r1, 2.6.24-ovz008 x86_64)
=================================================================
System uname: Linux-2.6.24-ovz008-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E6750_@_2.66GHz-with-gentoo-2.0.1
Timestamp of tree: Mon, 03 Aug 2009 18:50:01 +0000
app-shells/bash:     4.0_p28
dev-lang/python:     2.5.4-r2, 2.6.2-r1
dev-python/pycrypto: 2.0.1-r8
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.4.3-r3
sys-apps/sandbox:    2.0
sys-devel/autoconf:  2.63-r1
sys-devel/automake:  1.10.2, 1.11
sys-devel/binutils:  2.19.1-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6a
virtual/os-headers:  2.6.29
ACCEPT_KEYWORDS="amd64 ~amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"

Comment 50 Joakim 2009-08-03 20:36:56 UTC
This might be of interest, runing the patch and then eselect command in the host environment of my openvz kernel worked fine. I then went into one of the guests (all gentoo of course) and ran #emerge -uDalN system which of course failed after eselect-python and I then applyed the patch with this result:

web ~ # patch /usr/share/eselect/modules/python.eselect python.eselect.patch
patching file /usr/share/eselect/modules/python.eselect
web ~ # eselect python set python2.6
/etc/make.conf: line 12: CONFIG_PROTECT: command not found
/etc/make.conf: line 12: CONFIG_PROTECT: command not found

the 2 last lines above never came about when the command was run on host system. But after that emerge seem to work again in any case.
Comment 51 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 20:43:03 UTC
(In reply to comment #50)

Your /etc/make.conf has invalid content.
Comment 52 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 22:18:52 UTC
Created attachment 200065 [details, diff]
python.eselect.patch

I will commit this patch today, if it works correctly for everybody.
Comment 53 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-03 22:23:56 UTC
*** Bug 280229 has been marked as a duplicate of this bug. ***
Comment 54 jon R-B 2009-08-03 22:35:14 UTC
Funny thing is everything is working fine for me (bar eselect python {show,list} not being formatter fully)

emerge eselect eselect-python -vp

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] app-admin/eselect-1.1.2  USE="bash-completion -doc" 0 kB
[ebuild   R   ] app-admin/eselect-python-20090801  0 kB

Total: 2 packages (2 reinstalls), Size of downloads: 0 kB



emerge --info
Portage 2.2_rc33 (default/linux/amd64/2008.0/desktop, gcc-4.3.3, glibc-2.10.1-r0, 2.6.30-gentoo-r3 x86_64)
=================================================================
System uname: Linux-2.6.30-gentoo-r3-x86_64-Intel-R-_Core-TM-2_CPU_6600_@_2.40GHz-with-gentoo-2.0.1
Timestamp of tree: Mon, 03 Aug 2009 04:15:01 +0000
ccache version 2.4 [enabled]
app-shells/bash:     4.0_p28
dev-java/java-config: 2.1.8-r1
dev-lang/python:     2.5.4-r3, 2.6.2-r1, 3.1
dev-util/ccache:     2.4-r8
dev-util/cmake:      2.6.4-r2
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.4.3-r3
sys-apps/sandbox:    2.0
sys-devel/autoconf:  2.13, 2.63-r1
sys-devel/automake:  1.5, 1.9.6-r2, 1.10.2, 1.11
sys-devel/binutils:  2.19.1-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6a
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="amd64 ~amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe -fomit-frame-pointer -Wno-format-security -U_FORTIFY_SOURCE "
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-march=core2 -O2 -pipe -fomit-frame-pointer -Wno-format-security -U_FORTIFY_SOURCE "
DISTDIR="/usr/portage/distfiles"
FEATURES="ccache distlocks fixpackages metadata-transfer parallel-fetch preserve-libs protect-owned sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS=" ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo http://mirror.switch.ch/mirror/gentoo/ ftp://mirror.switch.ch/mirror/gentoo/ ftp://ftp.solnet.ch/mirror/Gentoo http://www.mirror.ac.uk/mirror/www.ibiblio.org/"
LANG="en_GB.UTF-8"
LC_ALL="en_GB.UTF-8"
LDFLAGS="-Wl,-O1"
LINGUAS="en_GB en uk"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/layman/oss-overlay /usr/local/portage/layman/java-overlay /usr/local/portage/layman/mpd /usr/local/portage/layman/sunrise /usr/local/portage/layman/Spring /usr/local/portage"
SYNC="rsync://192.168.0.2/gentoo-portage"
USE="X aac acl acpi aiglx amd64 bash-completion berkdb branding bzip2 cairo cdr cli consolekit cracklib crypt cups dbus dri dvd dvdr dvdread emboss encode evdev evo fam fortran gdbm gif glitz gnome gnome-keyring gpm gstreamer gtk gtk2 hal iconv ipv6 isdnlog java jpeg libnotify mad mikmod mmx mp3 mpeg mudflap multilib nautilus ncurses nls nptl nptlonly nsplugin ogg opengl openmp oss pam pcre pdf perl png ppds pppd python qt3support quicktime readline reflection samba sdl session spell spl sse sse2 ssl startup-notification svg sysfs tcpd threads tiff truetype type1 unicode usb vorbis xml xorg xulrunner xv zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en_GB en uk" USERLAND="GNU" VIDEO_CARDS="nv nvidia vesa"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS





Comment 55 Jonathan Callen (RETIRED) gentoo-dev 2009-08-03 23:35:30 UTC
I would like to suggest that instead of the current patch to python.eselect, you do something similar to gcc-config, where you have an executable in /usr/lib/misc (I'll call it python-config for the moment) that reads the EPYTHON variable, if set; otherwise reading file, /etc/env.d/python/config, which determines which version of python to call; if neither exist, defaulting to the highest version available (or whatever other default you want).  The python-config program would be compiled during src_compile, so that a C compiler is no longer required at runtime (think binpkgs), and so the reading of make.conf is no longer required: there are more places CFLAGS, CPPFLAGS, and LDFLAGS may be set.  This is just a suggestion, and I don't have code to match right now.
Comment 56 Jonathan Callen (RETIRED) gentoo-dev 2009-08-04 01:40:52 UTC
Created attachment 200082 [details, diff]
python.eselect.patch

This patch, and the following attachment, work similarly to the previous implementation in that they compile a C program that defers to the EPYTHON variable.  They differ in that instead of compiling in a default to use if EPYTHON is unset, this version first checks /etc/env.d/python/config to find a version there; if it can't find a version there, then it looks for /usr/bin/python2.?, then /usr/bin/python?.? to run.
Comment 57 Jonathan Callen (RETIRED) gentoo-dev 2009-08-04 01:42:24 UTC
Created attachment 200087 [details]
wrapper for python

This C program is to be compiled as /usr/bin/python-wrapper, and is symlinked to /usr/bin/python by python.eselect (as patched in attachment #200082 [details, diff])
Comment 58 Jonathan Callen (RETIRED) gentoo-dev 2009-08-04 03:08:44 UTC
Created attachment 200095 [details]
new wrapper for python

This fixes some bugs from the last wrapper: it now looks in the same directory as "python", finds python versions with more than 1 digit in the major or minor version, and is a bit cleaner overall.
Comment 59 Jonathan Callen (RETIRED) gentoo-dev 2009-08-04 03:20:12 UTC
(In reply to comment #58)
> Created an attachment (id=200095) [edit]
> new wrapper for python
> 
> This fixes some bugs from the last wrapper: it now looks in the same directory
> as "python", finds python versions with more than 1 digit in the major or minor
> version, and is a bit cleaner overall.
> 

It also fixes an off-by-one error that cause the previous version not to work
Comment 60 Fabian Groffen gentoo-dev 2009-08-04 06:56:56 UTC
I like the approach of gcc-config.  Small improvement, if there is no (usable) /proc on the target system, you can just examine argv[0] and make it absolute if it isn't, but contains a slash, by using the current working directory.  Otherwise you'll have to examine the path to find the binary, like gcc-config does.
Comment 61 Ryan Hill (RETIRED) gentoo-dev 2009-08-04 07:37:56 UTC
this sounds like great progress, and something that should be thoroughly tested in an overlay while the current version is masked so we stop breaking everyone and their dog's systems in the meantime. :P
Comment 62 Jonathan Callen (RETIRED) gentoo-dev 2009-08-04 10:27:52 UTC
Created attachment 200127 [details]
wrapper for python

This version of the wrapper uses argv[0] and $PATH to find where it was called from instead of using /proc/self/exe which I'm told doesn't work on *BSD.
Comment 63 Michael Edenfield 2009-08-04 12:58:23 UTC
There appears to be a related change in the kernel between 2.6.27 and 2.6.28 that will make this bug go away:

--- linux-2.6.27-hardened-r3/fs/binfmt_script.c 2008-10-09 18:13:53.000000000 -0400
+++ linux-2.6.28-hardened/fs/binfmt_script.c    2008-12-24 18:26:37.000000000 -0500
@@ -22,14 +22,15 @@
        char interp[BINPRM_BUF_SIZE];
        int retval;
 
-       if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!') || (bprm->sh_bang)) 
+       if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!') ||
+           (bprm->recursion_depth > BINPRM_MAX_RECURSION))
                return -ENOEXEC;
        /*
         * This section does the #! interpretation.
         * Sorta complicated, but hopefully it will work.  -TYT
         */
 
-       bprm->sh_bang = 1;
+       bprm->recursion_depth++;
        allow_write_access(bprm->file);
        fput(bprm->file);
        bprm->file = NULL;

Under the new behavior, you should be ok nesting up to 4 levels of shell script in a chain of #! lines.  Hardened is up to 2.6.29 so I'm going to upgrade and see if that helps.

Comment 64 Rafael Azevedo 2009-08-04 17:50:50 UTC
I'm having the same problem.
Today I tried to upgrade portage and now emerge is not working anymore.

The problem seems to be with the Python upgrade.

# head -n4 /usr/bin/emerge
#!/usr/bin/python
# Copyright Gentoo Foundation 2006
# Portage Emerge bits
# $Id: emerge 12321 2008-12-25 01:23:32Z zmedico $

head -n4 /usr/bin/python
#!/bin/bash
# Gentoo Python wrapper script

[[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.5"

# eselect python list
Available python interpreters:
  [1]   python2.4
  [2]   python2.5


This happened when I was upgrading portage...
>>> Emerging (5 of 5) sys-apps/portage-2.1.6.13 to /
>>> Downloading 'http://mirror.usu.edu/mirrors/gentoo/distfiles/portage-2.1.6.13.patch.bz2'
--14:04:52--  http://mirror.usu.edu/mirrors/gentoo/distfiles/portage-2.1.6.13.patch.bz2
           => `/usr/portage/distfiles/portage-2.1.6.13.patch.bz2'
Resolving mirror.usu.edu... 129.123.1.18
Connecting to mirror.usu.edu|129.123.1.18|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 177,383 (173K) [application/x-bzip2]

100%[====================================================================================================>] 177,383      134.62K/s

14:04:55 (134.29 KB/s) - `/usr/portage/distfiles/portage-2.1.6.13.patch.bz2' saved [177383/177383]

>>> Downloading 'http://mirror.usu.edu/mirrors/gentoo/distfiles/portage-2.1.6.tar.bz2'
--14:04:55--  http://mirror.usu.edu/mirrors/gentoo/distfiles/portage-2.1.6.tar.bz2
           => `/usr/portage/distfiles/portage-2.1.6.tar.bz2'
Resolving mirror.usu.edu... 129.123.1.18
Connecting to mirror.usu.edu|129.123.1.18|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 518,771 (507K) [application/x-bzip2]

100%[====================================================================================================>] 518,771      294.31K/s

14:04:57 (293.64 KB/s) - `/usr/portage/distfiles/portage-2.1.6.tar.bz2' saved [518771/518771]

 * checking ebuild checksums ;-) ...                                                                                                      [ ok ]
 * checking auxfile checksums ;-) ...                                                                                                     [ ok ]
 * checking miscfile checksums ;-) ...                                                                                                    [ ok ]
 * checking portage-2.1.6.tar.bz2 ;-) ...                                                                                                 [ ok ]
 * checking portage-2.1.6.13.patch.bz2 ;-) ...                                                                                            [ ok ]
>>> Unpacking source...
>>> Unpacking portage-2.1.6.tar.bz2 to /var/tmp/portage/sys-apps/portage-2.1.6.13/work
>>> Unpacking portage-2.1.6.13.patch.bz2 to /var/tmp/portage/sys-apps/portage-2.1.6.13/work
 * Applying portage-2.1.6.13.patch ...                                                                                                    [ ok ]
 * Setting portage.VERSION to 2.1.6.13 ...
>>> Source unpacked.
>>> Compiling source in /var/tmp/portage/sys-apps/portage-2.1.6.13/work/portage-2.1.6 ...
>>> Source compiled.
>>> Test phase [not enabled]: sys-apps/portage-2.1.6.13

>>> Install portage-2.1.6.13 into /var/tmp/portage/sys-apps/portage-2.1.6.13/image/ category sys-apps
patching file make.conf
>>> Completed installing portage-2.1.6.13 into /var/tmp/portage/sys-apps/portage-2.1.6.13/image/

ecompressdir: bzip2 -9 /usr/share/man
strip: x86_64-pc-linux-gnu-strip --strip-unneeded -R .comment
* checking 257 files for package collisions
>>> Merging sys-apps/portage-2.1.6.13 to /
/var/tmp/._portage_reinstall_.nwJbmp/bin/portageq: line 6: import: command not found
/var/tmp/._portage_reinstall_.nwJbmp/bin/portageq: line 8: try:: command not found
/var/tmp/._portage_reinstall_.nwJbmp/bin/portageq: line 9: import: command not found
/var/tmp/._portage_reinstall_.nwJbmp/bin/portageq: line 11: syntax error near unexpected token `('
/var/tmp/._portage_reinstall_.nwJbmp/bin/portageq: line 11: `   def exithandler(signum, frame):'
 *
 * ERROR: sys-apps/portage-2.1.6.13 failed.
 * Call stack:
 *               ebuild.sh, line   49:  Called pkg_preinst
 *             environment, line 2271:  Called has_version 'pkg_preinst'
 *               ebuild.sh, line  180:  Called die
 * The specific snippet of code:
 *                      die "unexpected portageq exit code: ${retval}"
 *  The die message:
 *   unexpected portageq exit code: 2
 *
 * If you need support, post the topmost build error, and the call stack if relevant.
 * A complete build log is located at '/var/tmp/portage/sys-apps/portage-2.1.6.13/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sys-apps/portage-2.1.6.13/temp/environment'.
 *
!!! FAILED preinst: 1

 * Messages for package sys-apps/portage-2.1.6.13:

 *
 * ERROR: sys-apps/portage-2.1.6.13 failed.
 * Call stack:
 *               ebuild.sh, line   49:  Called pkg_preinst
 *             environment, line 2271:  Called has_version 'pkg_preinst'
 *               ebuild.sh, line  180:  Called die
 * The specific snippet of code:
 *                      die "unexpected portageq exit code: ${retval}"
 *  The die message:
 *   unexpected portageq exit code: 2
 *
 * If you need support, post the topmost build error, and the call stack if relevant.
 * A complete build log is located at '/var/tmp/portage/sys-apps/portage-2.1.6.13/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sys-apps/portage-2.1.6.13/temp/environment'.
 *
 * GNU info directory index is up-to-date.
 * IMPORTANT: 1 config files in '/etc' need updating.
 * See the CONFIGURATION FILES section of the emerge
 * man page to learn how to update config files.







(In reply to comment #9)
> Sorry.  I'm assuming you meant to add that to /usr/bin/python.  Forgive me if
> you meant otherwise.
> 
> romulus bin # cat /usr/bin/python
> #!/bin/sh
> # Gentoo Python wrapper script
> 
> [[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.6"
> echo "${0%/*}/${EPYTHON:-python2.6}" "$@"
> "${0%/*}/${EPYTHON:-python2.6}" "$@"
> romulus bin # python --version
> /usr/bin/python2.6 --version
> Python 2.6.2
> romulus bin # emerge --info
> /usr/bin/emerge: line 6: import: command not found
> /usr/bin/emerge: line 8: try:: command not found
> /usr/bin/emerge: line 9: import: command not found
> /usr/bin/emerge: line 11: syntax error near unexpected token `('
> /usr/bin/emerge: line 11: `     def exithandler(signum,frame):'
> 
> Doesn't seem to echo for emerge...
> 
> romulus bin # head -n4 /usr/bin/emerge
> #!/usr/bin/python
> # Copyright 2006-2009 Gentoo Foundation
> # Distributed under the terms of the GNU General Public License v2
> # $Id: emerge 13152 2009-03-23 02:15:33Z zmedico $
> 

I first thought that the problem was with portage.. but it does not seems to be.
I did follow instructions in http://www.gentoo.org/proj/en/portage/doc/manually-fixing-portage.xml but had no success.

Any idea of how to fix this problem?

Thanks.
Rafael
Comment 65 Rafael Azevedo 2009-08-04 17:53:53 UTC
# emerge --info
/usr/bin/emerge: line 6: import: command not found
/usr/bin/emerge: line 8: try:: command not found
/usr/bin/emerge: line 9: import: command not found
/usr/bin/emerge: line 11: syntax error near unexpected token `('
/usr/bin/emerge: line 11: `     def exithandler(signum,frame):'
Comment 66 Rafael Azevedo 2009-08-04 17:56:34 UTC
When I tried to replace python, I got this new error:

bin # rm /usr/bin/python
bin # ln -s /usr/bin/python2.4 /usr/bin/python
bin # emerge --info
Traceback (most recent call last):
  File "/usr/bin/emerge", line 34, in ?
    import _emerge
ImportError: No module named _emerge

Comment 67 Fabian Groffen gentoo-dev 2009-08-04 17:58:27 UTC
symlink it to python2.5, that's what your portage is built with
Comment 68 Rafael Azevedo 2009-08-04 18:01:33 UTC
taurus temp # rm /usr/bin/python
taurus temp # ln -s /usr/bin/python2.5 /usr/bin/python
taurus temp # emerge
Traceback (most recent call last):
  File "/usr/bin/emerge", line 34, in <module>
    import _emerge
ImportError: No module named _emerge
Comment 69 Rafael Azevedo 2009-08-04 18:27:24 UTC
I just noticed that there is not dir:
/usr/lib/portage/pym/_emerge

What might have happened to it?
Comment 70 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2009-08-04 19:19:59 UTC
(In reply to comment #61)
> this sounds like great progress, and something that should be thoroughly tested
> in an overlay while the current version is masked so we stop breaking everyone
> and their dog's systems in the meantime. :P
> 

I fully agree, it should have been package.mask'd 4 days ago when the OP opened this bug.
Comment 71 Fabian Groffen gentoo-dev 2009-08-04 19:39:52 UTC
QA team, please evaluate this.  Breaking Portage is bad.  I see no intentions
by the persons involved to back out this change and develop it in a more
controlled environment.
Comment 72 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-08-04 19:40:54 UTC
Oh fun, this is the same I hit on Gentoo/FreeBSD yesterday (and for which I dekeyworded the current eselect-python).
Comment 73 Krzysztof Pawlik (RETIRED) gentoo-dev 2009-08-04 19:58:12 UTC
(In reply to comment #47)
> (In reply to comment #46)
> > What if local non-root user creates /tmp/python.c and asks root to run
> > `eselect python' (or it gets ran during update)?
> 
> Would the following be sufficient?:
> rm -f /tmp/python.c || die "Deletion of /tmp/python.c failed"
> touch /tmp/python.c || die "Creation of /tmp/python.c failed"
> chmod 400 /tmp/python.c || die "Setting of permissions on /tmp/python.c failed"
> 

No, as you still have a race condition (consider running `while /bin/true; ln
-sf /etc/passwd /tmp/python.c' by malicious user -- there's a change that `ln'
will be executed between `rm' and `touch' calls).

I really like the solution mentioned in one of the comments: the C source
should be build during merge of eselect-python, placed somewhere
(/usr/lib{,64}/misc/ is a good idea) and /usr/bin/python should be a symlink to
it.
Comment 74 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-08-04 20:02:23 UTC
Agreed with nelchael, especially it's much better to build stuff on ebuild, rather than at runtime; gcc might not be usable on a running system (think servers).
Comment 75 Rafael Azevedo 2009-08-04 20:10:04 UTC
Ok, but what can I do to have emerge working back?

Does anybody have a solution so we can have portage/emerge working back?

Now my system seems to be dead for updates.
Comment 76 Krzysztof Pawlik (RETIRED) gentoo-dev 2009-08-04 20:23:40 UTC
(In reply to comment #75)
> Ok, but what can I do to have emerge working back?
> 
> Does anybody have a solution so we can have portage/emerge working back?
> 
> Now my system seems to be dead for updates.
> 

Make /usr/bin/python to be a symlink to /usr/bin/python2.x (where x is whatever you have).
Comment 77 Eric Ireland 2009-08-04 20:47:10 UTC
Just a fyi, for people hit by this and need a quick fix, you can do the following.  Its hacktacular, and probally isnt the best, but here ya go.

Edit /usr/bin/emerge
change "#!/usr/bin/python" to "#!/usr/bin/python2.6" or 2.5 or w/e version you have installed.

Do the same to usr/lib/portage/bin/ebuild.

do the following commands
echo "=app-admin/eselect-python-20090801" >> /etc/portage.mask
emerge eselect-python
rm /usr/bin/python ## required or eselect will yell
eselect python set 1
emerge portage

Comment 78 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-04 20:55:34 UTC
This bug should be fixed in app-admin/eselect-python-20090804.
Comment 79 Rafael Azevedo 2009-08-04 21:12:06 UTC
It does not work:

taurus temp # rm /usr/bin/python
taurus temp # ln -s /usr/bin/python2.5 /usr/bin/python
taurus temp # emerge
Traceback (most recent call last):
  File "/usr/bin/emerge", line 34, in <module>
    import _emerge
ImportError: No module named _emerge

(In reply to comment #76)
> (In reply to comment #75)
> > Ok, but what can I do to have emerge working back?
> > 
> > Does anybody have a solution so we can have portage/emerge working back?
> > 
> > Now my system seems to be dead for updates.
> > 
> 
> Make /usr/bin/python to be a symlink to /usr/bin/python2.x (where x is whatever
> you have).
> 

Comment 80 Jonathan Callen (RETIRED) gentoo-dev 2009-08-04 21:18:55 UTC
(In reply to comment #79)
> It does not work:
> 
> taurus temp # rm /usr/bin/python
> taurus temp # ln -s /usr/bin/python2.5 /usr/bin/python
> taurus temp # emerge
> Traceback (most recent call last):
>   File "/usr/bin/emerge", line 34, in <module>
>     import _emerge
> ImportError: No module named _emerge

That looks like a completely different bug, which doesn't appear to have anything to do with this.
Comment 81 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-04 21:20:42 UTC
(In reply to comment #79)
> taurus temp # rm /usr/bin/python
> taurus temp # ln -s /usr/bin/python2.5 /usr/bin/python
> taurus temp # emerge
> Traceback (most recent call last):
>   File "/usr/bin/emerge", line 34, in <module>
>     import _emerge
> ImportError: No module named _emerge

You probably reproduce bug #280320.
Comment 82 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-04 21:25:41 UTC
BSD team:
Please test and keyword app-admin/eselect-python-20090804 on sparc-fbsd and x86-fbsd.
Comment 83 Mark Loeser (RETIRED) gentoo-dev 2009-08-05 00:44:12 UTC
Please mask this until we are sure everything works.  Test it an overlay before we break even more people's installations in a way they don't know how to fix.
Comment 84 Daniel Robbins 2009-08-05 08:00:31 UTC
This is a total disaster in so many ways. Are there any docs on EPYTHON? If you are going to use a shell wrapper script, you should call "exec ...python" to get rid of the shell process.
Comment 85 Daniel Robbins 2009-08-05 08:06:42 UTC
I've masked this in Funtoo Linux. 3 days late due to travel.
Comment 86 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-05 15:56:07 UTC
(In reply to comment #84)
> This is a total disaster in so many ways. Are there any docs on EPYTHON?

EPYTHON usually should be set only by python.eclass.

> If you are going to use a shell wrapper script, you should call
> "exec ...python" to get rid of the shell process.

We no longer use shell script. (Using of exec wouldn't help users experiencing this bug.)

app-admin/eselect-python-20090804 creates symlink to binary executable to avoid problems on systems not having recursive shebang interpretation functionality.
Comment 87 Michael Edenfield 2009-08-05 16:03:19 UTC
(In reply to comment #86)

> We no longer use shell script. (Using of exec wouldn't help users experiencing
> this bug.)
> 
> app-admin/eselect-python-20090804 creates symlink to binary executable to avoid
> problems on systems not having recursive shebang interpretation functionality.

At the risk of sounding dense: since this is exactly what the old pre-20090801 behavior was... what was the motivation behind switching over to a shell script in the first place, and will switching back to a symlink require the loss of some useful new feature? 

Comment 88 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-05 16:22:26 UTC
(In reply to comment #87)
> At the risk of sounding dense: since this is exactly what the old
> pre-20090801 behavior was...

Not exactly. It's now a symlink to a different executable.

> what was the motivation behind switching over to a shell script in the first
> place

To add support for EPYTHON variable.

> and will switching back to a symlink require the loss of some useful new
> feature?

/usr/bin/python is now always a symlink to /usr/bin/python-wrapper.
/usr/bin/python-wrapper also supports EPYTHON variable.
Comment 89 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-08-05 16:29:33 UTC
*cough* python-wrapper should have gone to /usr/lib/misc or /usr/libexec, not in path…
Comment 90 John Mulligan 2009-08-08 15:28:17 UTC
The new wrapper script can segfault when PATH is not set in the environment, to reproduce: 

$ /usr/bin/env -i python
Segmentation fault (core dumped)

The problem is in the find_path function, line 38.
>> char* path = strdup(getenv("PATH"));
The getenv call is returning zero. Maybe it should fall back to /bin:/usr/bin if
PATH is not set?
Comment 91 Jonathan Callen (RETIRED) gentoo-dev 2009-08-08 16:30:05 UTC
(In reply to comment #90)
> The new wrapper script can segfault when PATH is not set in the environment, to
> reproduce: 
> 
> $ /usr/bin/env -i python
> Segmentation fault (core dumped)
> 
> The problem is in the find_path function, line 38.
> >> char* path = strdup(getenv("PATH"));
> The getenv call is returning zero. Maybe it should fall back to /bin:/usr/bin
> if
> PATH is not set?
> 

That is now bug 280780, and a patch is attached.
Comment 92 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-08-14 19:53:51 UTC
This bug was fixed some time ago.
Comment 93 Fabian Groffen gentoo-dev 2010-04-06 13:16:16 UTC
I noticed a regression due to the new GENTOO_PYTHON_TARGET_SCRIPT_PATH and GENTOO_PYTHON_WRAPPER_SCRIPT_PATH variables.

Some python applications that I run within buildbot (which is twistd apparently) return the twistd usage message instead of calling the right application.

What I see is that the environment of the buildslave contains this:
  GENTOO_PYTHON_TARGET_SCRIPT_PATH=/usr/bin/twistd-2.6
  GENTOO_PYTHON_WRAPPER_SCRIPT_PATH=/usr/bin/twistd

and what gets run is this:
env PATH=${PWD}/build/buildtools/autogen:$PATH PYTHONPATH=${PWD}/build/buildtools/autogen autogen.py

which results in:
Usage: twistd [options]
Options:
      --savestats        save the Stats object rather than the text output of
                         the profiler.

which I can indeed reproduce when I run the autogen script using:
env GENTOO_PYTHON_TARGET_SCRIPT_PATH=/usr/bin/twistd-2.6 GENTOO_PYTHON_WRAPPER_SCRIPT_PATH=/usr/bin/twistd PATH=${PWD}/build/buildtools/autogen:$PATH PYTHONPATH=${PWD}/build/buildtools/autogen autogen.py


I can easily reproduce it like this:
% cat test.py
#!/usr/bin/env python

print 'hello'
% chmod 755 test.py 
% ./test.py 
hello
% env GENTOO_PYTHON_TARGET_SCRIPT_PATH=/usr/bin/twistd-2.6 GENTOO_PYTHON_WRAPPER_SCRIPT_PATH=/usr/bin/twistd PATH=.:$PATH test.py
Usage: twistd [options]
Options:
      --savestats        save the Stats object rather than the text output of
                         the profiler.
...

I think the wrapper should unset the two variables before it execs such that they never get in the way.
Comment 94 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-04-06 14:21:56 UTC
(In reply to comment #93)

Python already unsets these variables.
http://overlays.gentoo.org/proj/python/changeset?old_path=patches%2F2.6.5%2F61_all_process_data.patch&old=322&new_path=patches%2F2.6.5%2F61_all_process_data.patch&new=323

If you can reproduce a problem with dev-lang/python-2.6.5-r1, then report a new bug.