I'm not sure where to report this. This may be an issue with the stage3 tarball, portage snapshot, portage, or a particular ebuild. Basically I have some semi-automated builds that build inside a chroot. It creates a chroot environment, unpacks the latest stage3 and portage snapshot, does an "emerge --update --newuse --deep world" and then does a "emerge --depclean --with-bdeps=n" This pretty much works, or has so for about 8 months or so, however today it fails after the --depclean. The reason is it wants to remove python 2.6: --- dev-lang/python selected: 2.6.6-r1 protected: none omitted: 3.1.3-r1 --- I do not have the python3 USE flag set (nor the build flag). --- # equery d python [ Searching for packages depending on python... ] app-admin/python-updater-0.8 (dev-lang/python) app-portage/gentoolkit-0.2.4.6.1-r1 (=dev-lang/python-2*) (=dev-lang/python-2*[xml]) dev-libs/libxml2-2.7.8 (python? =dev-lang/python-2*) (python? =dev-lang/python-2*[-build,xml]) sys-apps/file-5.05 (python? dev-lang/python) sys-apps/portage-2.1.9.25 (!python3 & !build? dev-lang/python:2.6) (!python3&build? dev-lang/python:2.6) (python3? =dev-lang/python-3*) (!python3&!build? >=dev-lang/python-3) sys-libs/cracklib-2.8.16 (python? =dev-lang/python-2*) --- (I don't usually install gentoolkit, but do so to show the equery results). Anyway it appears that "!python3 & !build" should keep python:2.6, but it doesn't. But I notice there are spaces around the "&" and the others dont' have that, so maybe there's a typo? Anyway this is the stage3-amd64-20110224 stage3 and the latest portage snapshot. This succeeded at least a couple of days ago. Today is the first day I noticed it fails. If I run "emerge --depclean" without the "--with-bdeps=n" then it will not remove python 2.6. I will attach emerge --info. I am running emerge --update using using binary packages (that were generated by the same script on previous runs). I will run the script again w/o using bin-package and notify whether it still occurs.
Created attachment 264299 [details] emerge --info
@python: Did you guys change anything?
Does anything depend on app-portage/gentoolkit, dev-libs/libxml2 or sys-libs/cracklib? Are dev-libs/libxml2 or sys-libs/cracklib installed with USE="python"? Also please install sys-apps/portage-2.1.9.41 and check if this behavior still occurs.
Just to update: Even when not using pre-compiled binaries "emerge --depclean --with-bdeps=n" removes python 2.6.
(In reply to comment #3) > Does anything depend on app-portage/gentoolkit, dev-libs/libxml2 or > sys-libs/cracklib? Are dev-libs/libxml2 or sys-libs/cracklib installed with > USE="python"? Nothing depends on gentoolkit. I normally do not have it installed. I only emerged it to show the output from "equery". The other packages.. I assume pam depends on cracklib and python depends on libxml2 (I have xml USE for python). But I do not have the python USE flag enabled for those packages, so they will not pull in python 2.6 in and of themselves. > Also please install sys-apps/portage-2.1.9.41 and check if this behavior still > occurs. > I did install the sys-apps/portage-2.1.9.41 (by adding it to package.keywords before doing "emerge --update --newuse --deep world") and it still unmerges python 2.6.6-r1.
(In reply to comment #5) > Nothing depends on gentoolkit. Is app-portage/gentoolkit in @selected set (/var/lib/portage/world)? > python depends on libxml2 Python with USE="-build xml" depends on >=dev-libs/expat-2, not dev-libs/libxml2.
(In reply to comment #6) > (In reply to comment #5) > > Nothing depends on gentoolkit. > > Is app-portage/gentoolkit in @selected set (/var/lib/portage/world)? What I meant to say is this: This bug occurs when gentoolkit is *not* installed. I only installed gentoolkit after-the-fact for the purpose of submitting this bug (to show the output of "equery d python" > > python depends on libxml2 > > Python with USE="-build xml" depends on >=dev-libs/expat-2, not > dev-libs/libxml2. That perhaps is true, I'm not sure what pulled in libxml2 (perhaps it is brought in by something else or by my pulling in gentoolkit). Basically what is done is this: 1. Create a basic chroot environment a-la the handbook. 2. emerge --update --newuse --deep world 3. emerge --depclean --with-bdeps=n This will unmerge python:2.6 Ok, before I run step 3, this is the output of "equery d python" (actually this time I didn't emerge gentoolkit inside the chroot but ran # ROOT=/path/to/chroot equery d python: * These packages depend on python: app-admin/python-updater-0.8 (dev-lang/python) dev-libs/libxml2-2.7.8 (python ? =dev-lang/python-2*) (python ? =dev-lang/python-2*[-build,xml]) sys-apps/file-5.05 (python ? dev-lang/python) sys-apps/portage-2.1.9.41 (python3 ? =dev-lang/python-3*) (build ? dev-lang/python:2.7) (build ? dev-lang/python:2.6) (!build ? >=dev-lang/python-3) sys-libs/cracklib-2.8.16 (python ? =dev-lang/python-2*) # ROOT=/path/to/chroot equery d libxml2 * These packages depend on libxml2: sys-devel/gettext-0.18.1.1-r1 (dev-libs/libxml2) # ROOT=/path/to/chroot equery d cracklib * These packages depend on cracklib: sys-apps/shadow-4.1.4.3 (cracklib ? >=sys-libs/cracklib-2.7-r3) sys-libs/pam-1.1.3 (cracklib ? >=sys-libs/cracklib-2.8.3) (The output looks different because I guess I'm running a different gentoolkit outside the chroot). (I run ~amd64 outside the chroot). Anyway I don't have "python" or "python3" in my USE flags. From this output it looks like portage only requires python:2.6 if USE=build. So given these facts it appears there is nothing that depends on python:2.6. So portage appears to be doing the "right" thing.
My guess now is that before this has always been a bug with portage, but it was masked perhaps by another package that depended on python:2.6. But perhaps recently this other package has changed, and no longer depends on python:2.6, so now this bug is revealed.
sys-apps/portage[build,-python2,-python3] depends on Python 2 to ensure that stages contain Python 2. You can simply uninstall Python 2, if you don't need it.
(In reply to comment #9) > sys-apps/portage[build,-python2,-python3] depends on Python 2 to ensure that > stages contain Python 2. You can simply uninstall Python 2, if you don't need > it. > The problem is not that I don't want python2. The fact is that I do want it because portage needs it, however the --depclean removes python2. The stable version of portage doesn't have a python2 USE flag, I installed the testing version per comment #3. Perhaps I am not being clear enough in the description of the problem? When I run the --depclean command, it removes python2. Then portage becomes unusable after that.
(In reply to comment #10) > The problem is not that I don't want python2. The fact is that I do want it > because portage needs it, however the --depclean removes python2. Actually stable Portage works correctly with Python 3. Only some other packages (e.g. dev-libs/libxml2[python] or sys-libs/cracklib[python]) would still need Python 2. > When I run the --depclean command, it removes python2. Then portage becomes > unusable after that. I noticed that you were using dev-lang/python-2.6.6-r1. Could you synchronize the tree and recreate stages with dev-lang/python-2.6.6-r2? (This version contains improved implementation of eselect_python_update(), which should properly handle uninstallation of the last version of Python 2.)
(In reply to comment #11) ... > I noticed that you were using dev-lang/python-2.6.6-r1. Could you synchronize > the tree and recreate stages with dev-lang/python-2.6.6-r2? (This version > contains improved implementation of eselect_python_update(), which should > properly handle uninstallation of the last version of Python 2.) > I have no idea how to recreate stages (I'm not a portage dev). I only download the official stage3. I did sync to the latest tree (using the latest official stage3) and the problem persists. To relieve any ambiguity I will show my exact steps (more or less): # mkdir /var/tmp/357001 # wget http://gentoo.osuosl.org/releases/amd64/current-stage3/stage3-amd64-20110224.tar.bz2 # tar xjpf stage3-amd64-20110224.tar.bz2 -C /var/tmp/357001 # mount -t proc none /var/tmp/357001/proc # mount -o bind /dev /var/tmp/357001/dev # wget http://gentoo.osuosl.org/snapshots/portage-latest.tar.xz # tar xjf portage-latest.tar.bz2 -C /var/tmp/357001/usr # cp -L /etc/resolv.conf /var/tmp/357001/etc/ # chroot /var/tmp/357001 /bin/bash # env-update # source /etc/profile # # basic configurations (make.conf, locale.gen, package.use) # emerge --sync --quiet # emerge --update --newuse --deep world # # normally i use bin-packages, but it occurs even if i don't # etc-update # emerge --depclean --with-bdeps=n ... dev-lang/python selected: 2.6.6-r1 protected: none omitted: 3.1.3-r1 ... All selected packages: perl-core/Scalar-List-Utils-1.23 sys-apps/attr-2.4.44 virtual/acl-0 sys-apps/acl-2.2.49 net-misc/curl-7.20.0-r2 dev-libs/expat-2.0.1-r3 dev-util/pkgconfig-0.25-r2 dev-libs/libxml2-2.7.8 dev-perl/Net-SMTP-SSL-1.01 dev-vcs/git-1.7.3.4-r1 dev-perl/Authen-SASL-2.15 virtual/perl-Digest-MD5-2.39 sys-libs/gdbm-1.8.3-r4 dev-libs/libpcre-8.02 sys-apps/tcp-wrappers-7.6-r8 dev-perl/Error-0.170.160 virtual/perl-digest-base-1.16 dev-lang/python-2.6.6-r1 dev-perl/Digest-SHA1-2.12 sys-libs/cracklib-2.8.16 dev-perl/IO-Socket-SSL-1.35 sys-devel/gettext-0.18.1.1-r1 dev-perl/Digest-HMAC-1.01-r1 dev-perl/Net-SSLeay-1.36 virtual/perl-Scalar-List-Utils-1.23 sys-libs/db-4.8.30 virtual/perl-libnet-1.22 After this the unmerging of packages goes, after it unmerges dev-lang/python-2.6.6-r1 the rest of the packages fail because there is no python2 interpreter and, of course, subsequent calls to emerge fail. Anyway if you can tell me how to make a stage3 I might be willing to try that.
(In reply to comment #12) dev-lang/python-2.6.6-r2 has been stabilized on amd64 on 2011-02-26, so this bug probably will be fixed in the next stage.
stage3-amd64-20110303 is now available and contains dev-lang/python-2.6.6-r2. Please test if this bug is now fixed.
(In reply to comment #14) > stage3-amd64-20110303 is now available and contains dev-lang/python-2.6.6-r2. > Please test if this bug is now fixed. Well I tried last night, but it didn't resolve the issue. I already had a feeling it would not resolve the issue and here's why? # ROOT=/path/to/chroot equery d python: * These packages depend on python: app-admin/python-updater-0.8 (dev-lang/python) dev-libs/libxml2-2.7.8 (python ? =dev-lang/python-2*) (python ? =dev-lang/python-2*[-build,xml]) sys-apps/file-5.05 (python ? dev-lang/python) sys-apps/portage-2.1.9.41 (python3 ? =dev-lang/python-3*) (build ? dev-lang/python:2.7) (build ? dev-lang/python:2.6) (!build ? >=dev-lang/python-3) sys-libs/cracklib-2.8.16 (python ? =dev-lang/python-2*) If this output is to be believed, then portage does not depend on python:2.6 (unless the build USE flag is in effect). So this would lead me to think one of two things is happening here: 1. The bug is in the portage ebuild and is not depending on python:2.6 2. The bug is in the portage itself and is not resolving the dependencies correctly. I did look at the ebuild for portage-2.1.9.25. It's hard for me to get my head around, but this part: !build? ( || ( dev-lang/python:2.8 dev-lang/python:2.7 dev-lang/python:2.6 >=dev-lang/python-3 ) Looks like it will accept python2 or python3. Since python3 is the latest version installed in the stage3, it doesn't think it needs python:2.6 (even though it was built with python2.
(In reply to comment #15) > Well I tried last night, but it didn't resolve the issue. So what exactly happens? > 1. The bug is in the portage ebuild and is not depending on python:2.6 Portage actually does not need dev-lang/python:2.6. >=dev-lang/python-3 suffices for Portage. > 2. The bug is in the portage itself and is not resolving the dependencies > correctly. Portage correctly resolves the dependencies.
(In reply to comment #16) > (In reply to comment #15) > > Well I tried last night, but it didn't resolve the issue. > > So what exactly happens? The same thing happens. --depclean removes python:2.6. Then everything fails. > Portage actually does not need dev-lang/python:2.6. >=dev-lang/python-3 > suffices for Portage. Yes, but if portage was built with python2 (I'm assuming it is in the stage3), then it *does* need python2. Maybe it can work with python3, but it fails so I'm assuming it can't. > > > 2. The bug is in the portage itself and is not resolving the dependencies > > correctly. > > Portage correctly resolves the dependencies. This is probably correct. What I know is this. emerge --depclean removes python2.6. After this portage fails right in the middle of the "--depclean". Subsequent calls to emerge fail because portage can't find python2.6. I'm assuming this is not the desired behavior.
The following commands work for me: # eselect python set python2.6 # CONFIG_PROTECT="-*" emerge -Cv dev-lang/python:2.6 !!! CONFIG_PROTECT is empty * This action can remove important packages! In order to be safer, use * `emerge -pv --depclean <atom>` to check for reverse dependencies before * removing packages. dev-lang/python selected: 2.6.6-r2 protected: none omitted: 2.4.6 2.5.4-r4 2.7.2_pre20110227 3.0.1 3.1.4_pre20110227 3.2.1_pre20110227 3.3_pre20110227 All selected packages: dev-lang/python-2.6.6-r2 >>> 'Selected' packages are slated for removal. >>> 'Protected' and 'omitted' packages will not be removed. >>> Waiting 6 seconds before starting... >>> (Control-C to abort)... >>> Unmerging in: 6 5 4 3 2 1 >>> Unmerging dev-lang/python-2.6.6-r2... ... Switching to python3.3 Switching to python2.7 ... >>> Regenerating /etc/ld.so.cache... * GNU info directory index is up-to-date. # eselect python show python3.3 # eselect python show --python2 python2.7 # eselect python show --python3 python3.3 # emerge -ptv dev-lang/python:2.6 These are the packages that would be merged, in reverse order: Calculating dependencies... done! [ebuild NS ] dev-lang/python-2.6.6-r2 [2.4.6, 2.5.4-r4, 2.7.2_pre20110227, 3.0.1, 3.1.4_pre20110227, 3.2.1_pre20110227, 3.3_pre20110227] USE="doc examples gdbm ipv6 ncurses readline sqlite ssl threads tk (wide-unicode) xml -berkdb -build -wininst" ELIBC="(-uclibc)" 0 kB Total: 1 package (1 in new slot), Size of downloads: 0 kB
(In reply to comment #17) > (In reply to comment #16) > > So what exactly happens? > > The same thing happens. --depclean removes python:2.6. Then everything fails. Please exactly explain what and how fails. Please provide some log. > > Portage actually does not need dev-lang/python:2.6. >=dev-lang/python-3 > > suffices for Portage. > > Yes, but if portage was built with python2 (I'm assuming it is in the stage3), > then it *does* need python2. Portage doesn't need to be reinstalled after switching Python version. Portage doesn't use 2to3. Portage sources are the same regardless of Python version used during installation of Portage.
Zac, can you provide some feedback about how this does or does not affect Portage?
Portage is supposed to work with any version of python matched by it's dependencies, so if something's not working the we need a log of the specific failure like Arfrever said.
As of the March 3 amd64 stage3 and the March 2 x86 stage 3, the default system python version is 3.1. I am not familiar enough with Python to make any kind of inference here, but I'm going to go out on a limb and suggest that these are related issues. Albert, can you post the output of eselect python list and post it here please? Hope this helps narrow the field down a bit.
This problem seems to have been solved with the latest stage3, but I was masking it. In my scripts I have: eselect python set python2.6 Which is run just after unpacking the stage3 tarball. This was a work-around for bug #330655. Commenting out that line fixes the problem, however I did have this problem in earlier stage3 even without the eselect (re: comment #12 where I perform the steps manually). I will attach the (sanitized) log. As you will see --depclean clears out python2.6 and as soon as that happens there are postrm failures on subsequent emerges. Then emerge fails to run. Anyway with the 03 March stage3 and removing my previous work-around this is not a problem. Thanks, I think this can be closed now.
Created attachment 265149 [details] Failure when using 2011-02-24 stage3
(In reply to comment #23) > As > you will see --depclean clears out python2.6 and as soon as that happens there > are postrm failures on subsequent emerges. Then emerge fails to run. Rather than being a "depclean problem" per-se, this it more an issue with the python-2.6 ebuild's pkg_postrm failing to use `eselect python` to set a suitable default interpreter. Also, it's possible that certain python eclass functions might die in other ebuilds due to the "active" version of python that they got installed with having been removed (though I've only observed this sort of problem during installation (not sure if it happens during postrm) of binary packages that call python_set_active_version during pkg_setup).
(In reply to comment #25) > Rather than being a "depclean problem" per-se, this it more an issue with the > python-2.6 ebuild's pkg_postrm failing to use `eselect python` to set a > suitable default interpreter. pkg_postrm() sets a suitable Python interpreter, but it affects only future Portage processes. Current Portage process still uses portage._python_interpreter, which is calculated only once during initialization of Portage. > Also, it's possible that certain python eclass functions might die in other > ebuilds due to the "active" version of python that they got installed with > having been removed Some functions would call die(), so it's recommended to uninstall Python only after switching Python version to a version not scheduled for uninstallation and rebuilding reverse dependencies using python-updater. (python_mod_cleanup() can be used only pkg_postrm() and tries to avoid calling functions, which would call die() when Python interpreter is unavailable. python_mod_cleanup() might trigger call to die() e.g. when python_pkg_setup() hasn't been called in pkg_setup() and python_execute_function() hasn't been called and python_mod_optimize() hasn't been called in pkg_postinst() and no Python interpreter is available.)
I have fixed Portage to disallow uninstallation of currently used Python interpreter: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commitdiff;h=cab19378be406d18a80114d00007e12b3310586d
I'd like to add a special package set to the depclean code which pulls both portage and the running python interpreter, so that it protects their dependencies too.
Btw the March 3 amd64 stage3 stage3 seems to be gone from the mirrors now: http://gentoo.osuosl.org/releases/amd64/current-stage3/ so this issue has returned.
If there's nothing in the stages to pull in python-2*, two possible obvious solutions are: (1) Start setting python3 as the default interpreter (2) Enable python2 for sys-apps/portage via package.use.force (force is needed since stage1 is built with USE=-*), until it's decided that (1) is desired.
(In reply to comment #30) > If there's nothing in the stages to pull in python-2*, two possible obvious > solutions are: > > (1) Start setting python3 as the default interpreter > (2) Enable python2 for sys-apps/portage via package.use.force (force is needed > since stage1 is built with USE=-*), until it's decided that (1) is desired. The current issue affects only stage3. I've done some testing and from stage1 up until the merge of python-2.6.6-r2 on stage3, the default interpreter is set to python2.6. The update to python3.1 is done by eselect_python_update on pkg_postinst of python-2.6.6-r2.
(In reply to comment #31) > (In reply to comment #30) > > If there's nothing in the stages to pull in python-2*, two possible obvious > > solutions are: > > > > (1) Start setting python3 as the default interpreter > > (2) Enable python2 for sys-apps/portage via package.use.force (force is needed > > since stage1 is built with USE=-*), until it's decided that (1) is desired. > > The current issue affects only stage3. I've done some testing and from stage1 > up until the merge of python-2.6.6-r2 on stage3, the default interpreter is set > to python2.6. The update to python3.1 is done by eselect_python_update on > pkg_postinst of python-2.6.6-r2. Well, we need to clarify which problem we're trying to solve. In comment #30 I was talking about solving the problem of python-2* being removed by depclean. In comment 31, you're talking about the default interpreter setting, which is a separate issue.
(In reply to comment #32) > (In reply to comment #31) > > (In reply to comment #30) > > > If there's nothing in the stages to pull in python-2*, two possible obvious > > > solutions are: > > > > > > (1) Start setting python3 as the default interpreter > > > (2) Enable python2 for sys-apps/portage via package.use.force (force is needed > > > since stage1 is built with USE=-*), until it's decided that (1) is desired. > > > > The current issue affects only stage3. I've done some testing and from stage1 > > up until the merge of python-2.6.6-r2 on stage3, the default interpreter is set > > to python2.6. The update to python3.1 is done by eselect_python_update on > > pkg_postinst of python-2.6.6-r2. > > Well, we need to clarify which problem we're trying to solve. In comment #30 I > was talking about solving the problem of python-2* being removed by depclean. > In comment 31, you're talking about the default interpreter setting, which is a > separate issue. True, I was mostly referring to bug 357927. But you did talk about switching the default interpreter to python3, that's why I added a note that this is only hitting us on stage3 for now. We may one day get to a point that python-2.6 is not pulled in for stage1, but that isn't the case *yet*.
(In reply to comment #27) > I have fixed Portage to disallow uninstallation of currently used Python > interpreter: > http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commitdiff;h=cab19378be406d18a80114d00007e12b3310586d This is in 2.1.9.43 and 2.2.0_alpha27.
(In reply to comment #22) > As of the March 3 amd64 stage3 and the March 2 x86 stage 3, the default system > python version is 3.1. > This problem seems to have returned: stage3-amd64-20110224.tar.bz2 sudo chroot /var/scratch/marduk/vabuild # eselect python list Available Python interpreters: [1] python2.6 * [2] python3.1 # python --version Python 2.6.6
(In reply to comment #35) > (In reply to comment #22) > > As of the March 3 amd64 stage3 and the March 2 x86 stage 3, the default system > > python version is 3.1. > > > > This problem seems to have returned: > > stage3-amd64-20110224.tar.bz2 > > sudo chroot /var/scratch/marduk/vabuild > # eselect python list > Available Python interpreters: > [1] python2.6 * > [2] python3.1 > > # python --version > Python 2.6.6 Sorry. I did not make myself clear. As of Mar 2/3 it was *incorrectly* python 3.1. This was *not* intended behavior and hence the stages were nuked from the mirrors. I'm not sure where this puts the dependency removal issue.. As per comment 34, I think the issue is still fixed with the newer portages. Can you confirm that please?
(In reply to comment #36) > > I'm not sure where this puts the dependency removal issue.. As per comment 34, > I think the issue is still fixed with the newer portages. Can you confirm that > please? I guess I'm confused now as to what the *fix* is supposed to be. When the default python was set to python3.1 that *did* fix this particular issue. [scenario #1] When python3.1 was the system python, the --depclean would unmerge python2.6 and then continue successully. [scenario #2] When python2.6 is the system python, the --depclean would unmerge python2.6 and then everything else would fail (as per attachment #2 [details]). Today, when I tried again, it seems python2.6 is the system python and therefore I experience the second scenario. Ok, so what I want to know is this: what is the intended direction wrt the stage3 tarballs and the intended steps to be taken on the part of its users? What I am doing is automating the building of Gentoo virtual appliances. So it's a script that does all this. I unpack the latest stage3, do updates and then do an "emerge --depclean --with-bdeps=n". My intention is that the appliance need not contain anything more than what's needed to run the appliance. Ideally I'd only want one version of python installed. Which version should that be? Need I automatically "eselect python xxx" after unpacking the stage3? If so which should I pick for the system python? Previously I have always gone by whatever is set up in the stage3. In the past this was always python2[.6]. Then it was changed to python3. At first I was told this was a bug. But then I was told it was intentional. Now I'm just confused. If I knew what is the intended direction (of the Gentoo developers) then I'd at least know what is the *correct* work-around when/if this happens again.
(In reply to comment #37) Until about 177 remaining packages are fixed to use appropriate version of Python (or scheduled for deletion) in situation when Python 3.* is set as main active version of Python, Python 2.* should be set as main active version of Python in official stages. Currently you should use `eselect python update` before using `emerge --depclean`. In the future, Python 3.* will be intentionally set as main active version of Python in official stages.
(In reply to comment #38) > (In reply to comment #37) > > Until about 177 remaining packages are fixed to use appropriate version of > Python (or scheduled for deletion) in situation when Python 3.* is set as main > active version of Python, Python 2.* should be set as main active version of > Python in official stages. Currently you should use `eselect python update` > before using `emerge --depclean`. In the future, Python 3.* will be > intentionally set as main active version of Python in official stages. To be clear, from the Release Engineering viewpoint, stages should currently be setting python-2.6 as the default interpreter. The last stages having python-3.1 as the default interpreter was an error and not our intention. One day in the future we may want to have python-3.1 as the default interpreter, but that is likely going to take some time and that decision will fall to the RelEng team as we're responsible for the stages. About having more than one python version around, you'll have to talk to the python team as that is their option.
