The recommended upgrade to portage-2.0.49-r20 did not copy portage.pyc, output.pyc from '.../pym' to '/usr/lib/python2.2/site-packages'. Consequently, no further portage operation is possible (nor can java-config work), because '/usr/lib/portage/pym' is not in python's search path without a little help from portage.py itself. This is new with the upgrade from (now nonexistent) -r19 ==> -r20 across all (sparc at least) systems. Reproducible: Always Steps to Reproduce: 1.emerge sync 2.emerge -u portage 3.etcat versions portage Actual Results: lacewing java_config # etcat versions portage Traceback (most recent call last): File "/usr/bin/etcat", line 78, in ? import portage ImportError: No module named portage Expected Results: etcat versions portage [ Results for search key : portage ] [ Applications found : 3 ] * app-portage/kportage : [ ] app-portage/kportage-0.6.1 (0) * app-portage/portagemaster : [M ] app-portage/portagemaster-0.2.0 (0) [M ] app-portage/portagemaster-0.2.1 (0) * sys-apps/portage : [ I] sys-apps/portage-2.0.49-r20 (0) [M ] sys-apps/portage-2.0.50_pre9 (0) antaresia etc # emerge info Portage 2.0.49-r20 (default-sparc64-1.4, gcc-3.2.3, glibc-2.3.2-r3, 2.4.23-sparc-r1) ================================================================= System uname: 2.4.23-sparc-r1 sparc64 sun4u Gentoo Base System version 1.4.3.10p1 ccache version 2.2 [enabled] ACCEPT_KEYWORDS="sparc" AUTOCLEAN="yes" CFLAGS="-mcpu=ultrasparc -O3 -pipe" CHOST="sparc-unknown-linux-gnu" COMPILER="gcc3" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.1/share/config /usr/kde/3/share/config /usr/share/config /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d" CXXFLAGS="-mcpu=ultrasparc -O3 -pipe -Wno-deprecated" DISTDIR="/usr/portage/distfiles" FEATURES="ccache sandbox" GENTOO_MIRRORS="http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="sparc X arts avi berkdb bidi cjk crypt cups encode fbcon foomaticdb gdbm gif gtk imlib java jpeg libwww mad mikmod motif mozilla mpeg mysql ncurses nls opengl oss pam pdflib perl png python qt readline ruby ruby18 slang spell ssl stroke tcltk tcpd tetex tiff truetype xml2 xmms xv zlib" =============== I managed to do this to myself on two systems before noticing that something was wrong, so I don't think it's a result of sometning being broken on a specific system.
To clarify: the Summary line should start with the word 'portage'
what version of python do you have ? .pyc are created on the fly ...
Re: Python version(=2.2.3, current for sparc), etc. First, some background. 1. All systems are sparc. I have no idea what happens on other systems, but I have verified this for three different sparc systems, all current. I do not plan to keep on going. .. :-) 2. The sequence was -r15 -> portage-2.0.49-r19 (the 1-day version) --> 2.0.49-r20 3. The stuff in /usr/lib/portage/pym is fine, but things like etcat, java-config, quickpkg never have that in python's path 4. With -r19 and earlier, the .pyc files are copied (or at least present) in /usr/lib/python2.2/site-packages; with -r20 they are not. 5. The upgrade -r15 --> -r19 was on 30.xii.03, 1543UT 6. The upgrade -r19 --> -r20 was on 01.i.04, 1604UT ============== Now, as to python version, here's what etcat says: ============== lacewing etc # etcat versions dev-lang/python [ Results for search key : dev-lang/python ] [ Applications found : 1 ] * dev-lang/python : [ ] dev-lang/python-2.1.3-r1 (2.1) [ ] dev-lang/python-2.2.1-r5 (2.2) [ ] dev-lang/python-2.2.2 (2.2) [ I] dev-lang/python-2.2.3 (2.2) [M~ ] dev-lang/python-2.2.3-r1 (2.2) [M~ ] dev-lang/python-2.2.3-r5 (2.2) [M ] dev-lang/python-2.3.2-r2 (2.3) ============================= All systems are the same (i.e., current) in that regard. ============================ Now, some further information. On a system holding at version -r19, we have the .pyc files in /usr/lib/portage/pym are showing a creation time as 16:19, and the ones in /usr/lib/python2.2/site-packages with a time as 16:20 (except for ....../pym/xpak.pyc, which shows 16:31 for creation) -- all dated 30.xii.03 I do not know off hand how the emerge gets the .pyc files from .../pym to .../site-packages, but the one for -r19 did, and the one from -r19==>-r20 did not. (NOTE: I have no way of testing what -r15 ==> -r20 would do...) This has happened on three current sparc systems; no relevant nfs directories (and the third system is completely independent). Hope this helps. Regards, Ferris
what if you re-emerge the affected packages ? (like java-config)
It fails. Here's what (I think) is happening (based on a comparison of the -r15 & -r20 ebuilds): 1. -r15 compiled everything into ..../site-packages 2. -r19 did who knows what, but it got current .pyc files into both site-packages and .../pym 3. -r20 puts everything into .../pym & cleas out site-packages 4. The stuff in portage is OK with that; 5. But other things using portage modules --- like gentoolkit-0.1.37, java-config don't know about .../pym, and so 'import portage' fails: E.g., etcat from gentoolkit starts out like ========== import os,sys,string,re,pprint import getopt,glob import portage from stat import * from output import * ========== which isn't ever going to work because 'import portage' isn't going to find anything without this (as from emerge): ========== sys.path = ["/usr/lib/portage/pym"]+sys.path ========== So, since the -r20 works hard to clean out site-packages (lines 260--269), pym (270), and to recompile in pym (275--288), it will never leave anything for applications like etcat to find when they want portage modules. At least, that's what the ebuild says, and that's what happens. Regards, Ferris
Easiest bandaid is at line 288 of the -r20.ebuild, add one line like cp ${ROOT}/usr/lib/portage/pym/*.py[co] ${ROOT}/lib/python2.2/site-packages But that sort of defeats whatever's going on with moving where portage puts its modules.
Bandaid isn't quite complete, because ebuild doesn't recompile xpak.py. You need to do it, too.
I'm seeing this as well on a sparcstation 20. Portage can't find needed modules when buildpkg is in FEATURES and when recalculating the counter either. And I'm experiencing problems with java-config as well.
Add CC
This problem breaks quickpkg for me on an ultra 10 with the error "ImportError: No module named xpak".
Looks like python-2.2.3 is too old for portage..-r20. You need the python patch 'python-2.3-add_portage_search_path.patch', which you don't get until python-2.2.3-r5, which version is '~sparc' This patch explicitly adds /usr/lib/portage/pym to the search path. And, if I just put it into 'site.py' by hand and emerge portage then everything looks good again. So, portage-..-r20 depends on a version of python still marked ~sparc
In addition to sparc, ppc ia64 and arm are not stable on python-2.2.3-r5 Is it possible to provide some dev-wide notification when changes like this are going to take place so we don't end up in situations like this?
I've marked python-2.2.3-r5 stable on sparc to fix this.
So it's a depedency / python-KEYWORDS problem.
Yes, portage-2.0.49-r20 is not depending on >=dev-lang/pythong-2.2.3-r5 and dev-lang/python-2.2.3-r5 is not stable on all arches. Please in the future endeavor to double and triple check these items from a QA perspective before releasing.
what am i supposed to do with this bug? i thought the new portage manually appended /usr/lib/portage/pym to sys.path to avoid putting in the python-2.2.3-r5 dependency, at least that was true in -r16, iirc. ppc, ia64 and arm (do we still support this?) need to get python-2.2.3-r5 to stable. btw, on the top of /usr/bin/emerge in 2.0.50_pre9, it has : sys.path = ["/usr/lib/portage/pym"]+sys.path is that not enough for now?
It works for portage itself, but not for external scripts using portage like etcat. Actually I think portage had a dependency on a fixed python at one point, but either that was changed back or lost for some reason. Nick ?
Whatever portage calls to build binary packages when buildpkg is in FEATUERS was unable to find the xpak module. Also, when emerging portage, it rebuilds a counter at the end, and that would consistantly fail because it could not find a module as well.
(Re comment #17): Yes, that module being portage: The ebuild has python -c 'import portage; ...' And the "Clear invalid keys" right after is suspect, too: it requires the same thing.
pkgmerge is fixed in cvs... I just fixed the ebuilds as they were actually not correct. Devs really should be using 2.0.50*, btw. The changes were very much intentional. Python shouldn't be adding portage to the path itself, I'm not a fan of that as it makes errors harder to find and puts site-packages before portage's internals... Makes it very hard to ensure portage is using the correct modules. I'd rather have Python have no hand in touching portage's info and just let portage work around python. (Meaning I'd prefer if python didn't modify the path like it does.) Anything that uses portage should be aware of where portage's modules are. Copying pyc's is really way out in left field. The proper fix is the way emerge does it at the top of the file. Karl's been working on updating gentoolkit for the newer portages, and I'm guessing this got overlooked due to python 2.3 having the path altered already. I mentioned it to him and he'll be getting a fix pushed out soon. Fixing etcat is not a reason to mark things stable, btw. If you have issues like this in the future... and can't get definitive answers... Call or shoot me an email. Phone info is easily obtainable from somewhere on -core and can be found via on my site.
Let me tease a little bit here. Remember, this all started out with a failure from java-config: No module output because, of course, output.py is part of portage, and output.pyc moved out of python's search path. (Comments 4 & 5) java-config belongs to dev-java etcat came to light from my etcat versions `which java-config` that followed. So, all the fixes in the world to gentoolkit would not have caught this, and it is most unintuitive that a change to portage would kill java-config. Since I happened to want to see how I had java configured immediately after having updated portage, it didn't take long to figure out. But that was fortuitous. I actually use java-config about 3 times a year. I am curious what is the mechanism to make sure everything in the portage tree knows to look for this sort of thing?
I've marked python-2.2.3-r5 stable on ia64
In theory, nothing in the portage tree needs this info. portageq provides an interface for command line applications. The applications within portage reference the paths themselves. java-config is a bit odd using a portage output module... but I guess there isn't anything wrong with that. There is no catch-all mechanism for such oddities. Depending on a package module like that isn't a particularly good idea as it will continue to change, breaking it repeatedly. The mechanism that does exist is email. All known developers were sent an email notifying them of an upcoming API change and path move. Anything not fixed at this point can with a fair amount of confidence be called derrelict. gentoo-portage-dev also exists for a similar purpose. It seems the problem here was that it was reported for the wrong packages first. Portage (stable) and java-config needed the corrections... The xpak issue (quickpkg) is now fixed in 49-r21 and has been in most of 2.0.50_pre*. I gave a shout out to Aether about the java-config issue... If he's not the right guy the I'll figure it out later.
At the risk of seeming terminally dense, let me ask one more question. Suppose python doesn't add the portage path itself. Then, how are the current portage ebuilds themselves to run? Won't those python invocations python -c 'import portage; ...' used to update counters and to clear invalid entries fail? They certainly did for -r20 portage when .../pym wasn't in python's path. (E.g., line 212 in the 50_pre9 ebuild for portage.)
Created attachment 23247 [details] __init__.py for portage package ## These are some other ways to fix the problem ### (Note: these fixes are without liquidx's fix to /usr/lib/python2.2/site.py) ### (Note: I'm using portage-2.0.49-r20) jmob@shamash [portage] $ pwd /usr/lib/python2.2/site-packages/portage jmob@shamash [portage] $ ls __init__.py jmob@shamash [portage] $ cat __init__.py """ Portage package compatibility """ import os from site import prefix __path__.insert(0, os.path.join(prefix, 'lib', 'portage', 'pym')) jmob@shamash [portage] $ python -c "from portage.output import *; print red('it werks')" it werks jmob@shamash [portage] $ sudo mv __init__.py _ jmob@shamash [portage] $ python -c "from portage.output import *; print red('it werks')" Traceback (most recent call last): File "<string>", line 1, in ? ImportError: No module named portage.output jmob@shamash [portage] $ export PYTHONPATH=/usr/lib/portage/pym jmob@shamash [portage] $ python -c "from output import *; print red('it werks')" it werks
Ferris: Without python setting the search path for modules individual programs that needed to use portage would have to modify the python module search path themselves. This would be how they would do it: import os, site, sys sys.path.insert(0, os.path.join(site.prefix, 'lib', 'portage', 'pym'))
I understand that. That's how the portage module portage.py itself does it. My misunderstanding at this point was just the context in which the ebuild file gets run, because that's where the "python -c 'import portage;...'" comes from. It's a small point as long as everything works. I suppose that when emerge prepends the /usr/lib/portage/pym path to sys.path, then the entire "emerge portage" runs with a PYTHONPATH set so that python searches portage's modules first, and everything is fine. And in general, it's certainly better for anyone who wants to use portage to set the path to it himself, rather than to keep expanding everyones python path. (But either way you get a mess if this location changes frequently.) My only concern at this point is that packages in the portage tree learn how to find portage modules before these modules disappear entirely from the python search path. And this is really pretty far from my original bug, which was that on sparc, 'emerge portage' killed java-config.... :-) (And, me, I don't even use python, so I don't have anything that cares where portage is. :-))
Yeah, when one thing in a Python runtime changes sys.path it changes for the whole runtime (for the some reason if you do "import sys" in multiple locations you always get the same sys module that was first imported).
as far as i'm concerned, this bug has been solved. as for removing /usr/lib/portage/pym from python's default search path, i'll schedule that for the next python revision, as soon as i'm confident all the gentoolkit and app-portage scripts that are around is adding /usr/lib/portage/pym to their search path themselves.
