Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 150690

Summary: dev-lang/python: distutils should support ignoring of distutils.cfg and .pydistutils.cfg
Product: Gentoo Linux Reporter: Joshua Schmidlkofer <menion>
Component: [OLD] DevelopmentAssignee: Python Gentoo Team <python>
Status: RESOLVED WONTFIX    
Severity: normal CC: caster, dev-jay
Priority: High    
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---

Description Joshua Schmidlkofer 2006-10-09 16:55:41 UTC
java-config-1.3 is not installing properly.  I have no idea why.   I have done only the smallest amount of troubleshooting.  I am getting this on some older hosts.  I ahve one case where I need to run the problem into the ground.

It is installing site-packages into /var/tmp/.....

==== 
hosting tmp #  cat /var/db/pkg/dev-java/java-config-1.3.7/CONTENTS 
dir /usr
dir /usr/lib
dir /usr/lib/python2.4
dir /usr/lib/python2.4/site-packages
dir /usr/share
dir /usr/share/doc
dir /usr/share/doc/java-config-1.3.7
obj /usr/share/doc/java-config-1.3.7/ChangeLog.gz ec125b984aa2ca12264e3758d64d557d 1160437738
dir /usr/share/man
dir /usr/share/man/man1
obj /usr/share/man/man1/java-config.1.gz 9135abe4b13c048be1926858e1e5feb4 1160437738
dir /usr/bin
obj /usr/bin/java-config-1 b3110181c74632eb094a176a9c788ff7 1160437738
dir /var
dir /var/tmp
dir /var/tmp/portage
dir /var/tmp/portage/java-config-1.3.7
dir /var/tmp/portage/java-config-1.3.7/homedir
dir /var/tmp/portage/java-config-1.3.7/homedir/lib
dir /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4
dir /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages
dir /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/__init__.py d6a785e6cb18346d9990dc6162484aee 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/cfg_parse.py e1f662293df29a718fcfaaa8595059f2 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/jc_envgen.py 8763d26e9416840e597189065a019090 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/jc_exceptions.py 8208380b077dbf4c230890f94ff81643 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/jc_iface.py 1903388af549ecccb2eac4014acec4a6 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/jc_options.py 649b5d15f3d805b008f0a461e08b08c2 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/jc_output.py 32e4e06e4f504baa0741052afa29d4c6 1160437738
obj /var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config/jc_util.py 2cb56c8b4ab53c0e395d6c0b4051d3ef 1160437738
dir /etc
dir /etc/env.d
obj /etc/env.d/30java-finalclasspath 9a4212aa54844344b19beba357540ab9 1160437738
======
hosting tmp # emerge --info
Portage 2.1.1 (selinux/2005.1/x86, gcc-3.3.6, glibc-2.3.5-r2, 2.6.15-gentoo-r7 i686)
=================================================================
System uname: 2.6.15-gentoo-r7 i686 AMD Athlon(tm) MP 2800+
Gentoo Base System version 1.6.14
Last Sync: Mon, 09 Oct 2006 22:00:01 +0000
ccache version 2.3 [enabled]
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: 1.3.7, 2.0.30
dev-lang/python:     2.3.5-r2, 2.4.2
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.3
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.12
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils:  2.15.92.0.2-r10
sys-devel/gcc-config: 1.3.12-r6
sys-devel/libtool:   1.5.20
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=athlon-mp -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib/X11/xkb /var/bind"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/terminfo"
CXXFLAGS="-O2 -march=athlon-mp -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig ccache distlocks metadata-transfer sandbox selinux sesandbox sfperms strict"
GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.math.bme.hu"
LINGUAS=""
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/asylumware-portage /usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="acpi apache2 berkdb cdr crypt cscope ctype cups dvd dvdr dvdread elibc_glibc emacs fam gd gdbm gif gmp gpm imagemagick imap imlib innodb input_devices_keyboard input_devices_mouse ipv6 kernel_linux ldap ldirectord lzo lzw maildir man mime mmap mmx mysql mysqli ncurses nls nptl pam pcntl pcre perl php-4 php-5 pic pie png posix postgres ppds pthreads pwdb python readline ruby samba sasl selinux session shared sharedmem slang slp sqlite sse ssl svg tcpd userland_GNU vhosts x86 xml2 xpm xsl zeo zlib"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Joshua Schmidlkofer 2006-10-09 16:57:36 UTC
Here is the shot (uselessly I am certain) of the ebuild ... install command.

>>> Compiling source in /var/tmp/portage/java-config-1.3.7/work/java-config-1.3.7 ...
running build
running build_py
creating build
creating build/lib
creating build/lib/java_config
copying ./java_config/__init__.py -> build/lib/java_config
copying ./java_config/cfg_parse.py -> build/lib/java_config
copying ./java_config/jc_envgen.py -> build/lib/java_config
copying ./java_config/jc_exceptions.py -> build/lib/java_config
copying ./java_config/jc_iface.py -> build/lib/java_config
copying ./java_config/jc_options.py -> build/lib/java_config
copying ./java_config/jc_output.py -> build/lib/java_config
copying ./java_config/jc_util.py -> build/lib/java_config
>>> Source compiled.
>>> Test phase [not enabled]: dev-java/java-config-1.3.7

>>> Install java-config-1.3.7 into /var/tmp/portage/java-config-1.3.7/image/ category dev-java
!!! SELinux module not found. Please verify that it was installed.
running install
running build
running build_py
running install_lib
creating /var/tmp/portage/java-config-1.3.7/image/var
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages
creating /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/__init__.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/cfg_parse.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/jc_envgen.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/jc_exceptions.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/jc_iface.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/jc_options.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/jc_output.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
copying build/lib/java_config/jc_util.py -> /var/tmp/portage/java-config-1.3.7/image/var/tmp/portage/java-config-1.3.7/homedir/lib/python2.4/site-packages/java_config
>>> Completed installing java-config-1.3.7 into /var/tmp/portage/java-config-1.3.7/image/

man:
gzipping man page: java-config.1
Comment 2 Joshua Schmidlkofer 2006-10-09 17:00:23 UTC
And - on the same system - a problem with java-config-2+

--- /usr/bin/
--- /var/
--- /var/tmp/
--- /var/tmp/portage/
--- /var/tmp/portage/java-config-2.0.30/
--- /var/tmp/portage/java-config-2.0.30/homedir/
>>> /var/tmp/portage/java-config-2.0.30/homedir/bin/
>>> /var/tmp/portage/java-config-2.0.30/homedir/bin/java-config-2
>>> /var/tmp/portage/java-config-2.0.30/homedir/bin/depend-java-query
>>> /var/tmp/portage/java-config-2.0.30/homedir/bin/run-java-tool
>>> /var/tmp/portage/java-config-2.0.30/homedir/bin/gjl
--- /etc/
--- /etc/java-config-2/
Comment 3 Vlastimil Babka (Caster) (RETIRED) gentoo-dev 2006-10-10 06:19:25 UTC
java-config is installed via standard means of python package via distutils eclass, so other packages using it should exhibit the same problem, can you try e.g. www-client/pybugz ? May be also because you forgot to run python-updater or something.
And this also looks suspicious:
!!! SELinux module not found. Please verify that it was installed.
Comment 4 Joshua Schmidlkofer 2006-10-10 09:16:54 UTC
At one time we were going to use SELinux with this system. But...
  * It's too much work under Gentoo.
  * It won't give us the control we need over sockets and localhost aliases.

GRSec stuff is probably where we will go.  I have various python modules installed successfully, but I will retry a couple that I have never used.


I will also remove the SELinux python modules.
Comment 5 Joshua Schmidlkofer 2006-10-10 09:53:15 UTC
Ok, It is definately a problem in the distutils stuff.  Could this be related to setuptools?  I have a lot of TurboGears apps on this box.   Something is interfering, I will look into the distutils stuff, but do you guys have any advice?

Snip from pybugz:
--- /var/tmp/portage/
--- /var/tmp/portage/pybugz-0.6.11/
--- /var/tmp/portage/pybugz-0.6.11/homedir/
>>> /var/tmp/portage/pybugz-0.6.11/homedir/lib/
>>> /var/tmp/portage/pybugz-0.6.11/homedir/lib/python2.4/
>>> /var/tmp/portage/pybugz-0.6.11/homedir/lib/python2.4/site-packages/
>>> /var/tmp/portage/pybugz-0.6.11/homedir/lib/python2.4/site-packages/bugz.py
>>> /var/tmp/portage/pybugz-0.6.11/homedir/bin/
>>> /var/tmp/portage/pybugz-0.6.11/homedir/bin/bugz
Comment 6 Vlastimil Babka (Caster) (RETIRED) gentoo-dev 2006-10-10 13:41:36 UTC
Python people, please advise :)
Comment 7 Marien Zwart (RETIRED) gentoo-dev 2006-10-10 14:51:09 UTC
What python is that? I assume "which python" gives you /usr/bin/python which is a symlink to python2.4 which is from a portage ebuild (I've seen this kind of thing happen when someone manually installed python or something like that)?

Output of python -c 'from distutils.sysconfig import get_makefile_filename as f;print f()'? Result of grepping for /var/tmp/portage in the file that should have given you?
Comment 8 Joshua Schmidlkofer 2006-10-10 15:01:05 UTC
Output of the requested information. (Wow, I learned a thing or two about python today. =)

[snip]
hosting ~ # which python
/usr/bin/python
hosting ~ # ls -l /usr/bin/python
lrwxrwxrwx  1 root root 9 Feb 13  2006 /usr/bin/python -> python2.4
hosting ~ # python -c 'from distutils.sysconfig import get_makefile_filename as f;print f()'
/usr/lib/python2.4/config/Makefile
hosting ~ # cat /usr/lib/python2.4/config/Makefile|grep var/tmp
RUNSHARED=       LD_LIBRARY_PATH=/var/tmp/portage/python-2.4.2/work/Python-2.4.2:
hosting ~ # 
[/snip]
Comment 9 Vlastimil Babka (Caster) (RETIRED) gentoo-dev 2006-10-11 03:27:36 UTC
try some more things:
# equery belongs /usr/lib/python2.4/config/Makefile
(requires gentoolkit emerged)
# grep ^prefix= /usr/lib/python2.4/config/Makefile
(should be "prefix=         /usr", I bet you will have "prefix= homedir" or something)
Comment 10 Joshua Schmidlkofer 2006-10-11 10:00:25 UTC
In response to the query 



====================================================================
hosting / # equery b /usr/lib/python2.4/config/Makefile
[ Searching for file(s) /usr/lib/python2.4/config/Makefile in *... ]
dev-lang/python-2.4.3-r4 (/usr/lib/python2.4/config/Makefile)

hosting / # grep ^prefix= /usr/lib/python2.4/config/Makefile
prefix=         /usr

====================================================================
Comment 11 Joshua Schmidlkofer 2006-10-11 10:41:49 UTC
OK - the problem

 The local admin uses a lot of virtual hosting, and has used this to modify the distutils behaviour: http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations

 Therefore, the stock python distutils config automatically causes all modules to be installed in the local users python dirs.  

 He created a 'override' config in /root (because with lots of users it's more practical) which is essentially the same as the stock distutils.

So = 
=== /usr/lib/python2.4/distutils/distutils.cfg ===
[install]
install_lib = ~/lib/python2.4/site-packages

# This next line is optional but often quite useful; it directs EasyInstall
# and the distutils to install scripts in the user's "bin" directory.  For
# Mac OS X framework Python builds, you should use /usr/local/bin instead,
# because neither ~/bin nor the default script installation location are on
# the system PATH.
#
install_scripts = ~/bin

[easy_install]
site_dirs = ~/lib/python2.4/site-packages


=== /root/.pydistutils.cfg ===
[install]
prefix=/usr
install_lib=/usr/lib/python2.4/site-packages
install_scripts=/usr/bin

[easy_install]
site-dirs=/usr/lib/python2.4/site-packages


======================

When HOME is set to /root, distutils behaves as it should - by installing everything in the system dirs.  However, if not, then it picks up the default distutils.cfg, and that assumes that you don't have write permission to the system directories.

So, Any suggestions?  I will work around this for now, but it would be nice to have a more permanent solution.

Comment 12 Marien Zwart (RETIRED) gentoo-dev 2006-10-11 12:07:58 UTC
Ick. The first thing that comes to mind is "don't do that then": do not edit files in /usr/lib/python2.4 if you want portage to continue to function properly. Why do you need this change? Is there no way to accomplish it without editing stuff in /usr/lib? Would putting this stuff in $HOME/.pydistutils.cfg work?
Comment 13 Cliff Wells 2006-10-11 15:17:44 UTC
Ick?  This is standard Python setuputils stuff, documented at:

http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installation

This is highly desirable for any sort of shared hosting setup as it allows users to maintain their own Python libraries that override the system-wide install.

I think the easiest fix would be to check the users home directory for .pydistutils.cfg (or create a default one) prior to trying to use setuputils.

An example .pydistutils would be something like:

[install]
prefix=/usr
install_lib=/usr/lib/python2.4/site-packages
install_scripts=/usr/bin

[easy_install]
site-dirs=/usr/lib/python2.4/site-packages


Comment 14 Vlastimil Babka (Caster) (RETIRED) gentoo-dev 2006-10-11 15:49:10 UTC
Leaving entirely to you guys, seems more apropriate.
Comment 15 Joshua Schmidlkofer 2006-10-11 22:45:10 UTC
(In reply to comment #12)
> Ick. The first thing that comes to mind is "don't do that then": do not edit
> files in /usr/lib/python2.4 if you want portage to continue to function
> properly. Why do you need this change? Is there no way to accomplish it without
> editing stuff in /usr/lib? Would putting this stuff in $HOME/.pydistutils.cfg
> work?
> 

AFAIK, that would work, it's not nessecarily the solution recommended, and I think it would be better to specifically override this, unless That's Bad(tm).  Overall, I think that a good sysadmin could accomplish the stock override using skel tools.  However, I think that the question should at least be asked about the effects in both cases.
Comment 16 Joshua Schmidlkofer 2006-10-12 07:13:22 UTC
After considering this over night, I really believe that Portage should manage these changes.  For the moment, portage _always_ installs packages in a system-wide fashion.

That being the case, I think a specific .pydistutils.cfg makes more sense then depending on people not /using/ thier systems.  I have not read any specific statements about design concepts for portage.  I think that portage should continue to function normally, even with an altered distutils.cfg.  It make a lot more sense for an /admin/ to change 1 file, rather than 1 file per user.

Also, I don't see that this could possibly add significant complexity to portage.
Comment 17 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-08-13 17:27:22 UTC
I suggest to patch Distutils to ignore distutils.cfg and .pydistutils.cfg when an environmental variable (e.g. PYTHON_IGNORE_SYSTEM_CONFIGURATION_FILES) is set. This variable would be set in make.defaults in base profile (similarly to PYTHONDONTWRITEBYTECODE).
Comment 18 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-08-14 15:48:00 UTC
*** Bug 329973 has been marked as a duplicate of this bug. ***
Comment 19 Brian Harring (RETIRED) gentoo-dev 2010-08-19 08:29:11 UTC
My inclination here is to add a simple check for the file, and tell the user they're on their own if it exists, rather than hacking up distutils for (bluntly) user idiocy.  Comments?
Comment 20 Cliff Wells 2010-08-19 16:17:14 UTC
"My inclination here is to add a simple check for the file, and tell the user
they're on their own if it exists, rather than hacking up distutils for
(bluntly) user idiocy.  Comments?"

My comment: I stopped caring about this bug around 2 years ago and this thread demonstrates clearly why:

1) bugs are due to user "idiocy" (not because users are following *upstream* documentation - you ought to try reading it sometime).

2) we'll get around to explaining that to you in a few years, assuming you haven't pissed off by then.

Gentoo has never been, and never will be a distro usable as a server OS precisely because of issues such as this (well, that and suddenly relocating all the files in /etc/httpd now and again during a minor update and then blaming users for not reading volumes of announcements, but that's neither here nor there).

What a load of crap. 
Comment 21 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-08-23 12:17:07 UTC
My inclination would be to WONTFIX this, but warning about the existence could work too, I guess. Just feels (a) pretty niche and (b) you should know what you're doing.
Comment 22 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-10-27 10:18:44 UTC
Okay, since no one responded since last time, I'll close this.