Created attachment 337830 [details, diff] git patch explaining the conversion I have converted both slots of wxpython to distutils-r1. I'd like to know your approval before I commit it.
Created attachment 337832 [details] 2.8.12.1-r1 ebuild
Created attachment 337834 [details] 2.9.4.1-r1 ebuild
If this drops support of python 2.5 we'll have to review the rdeps.
This breaks: dev-python/etsdevtools dev-python/ipython (<0.12 ie. all stable versions) dev-python/python-dsv dev-python/traitsbackendwx dev-python/winpdb dev-util/spe media-gfx/dispcalgui media-gfx/fontypython (i just fixed this in 0.4.4, needs to be stabilized though)
(In reply to comment #3) > If this drops support of python 2.5 we'll have to review the rdeps. I think I have removed 2.5 because the code didn't actually support it. If I recall correctly, it used 'with' in a few places which triggered errors. (In reply to comment #4) > This breaks: > > dev-python/etsdevtools > dev-python/ipython (<0.12 ie. all stable versions) > dev-python/python-dsv > dev-python/traitsbackendwx > dev-python/winpdb > dev-util/spe > media-gfx/dispcalgui > media-gfx/fontypython (i just fixed this in 0.4.4, needs to be stabilized > though) I'd say that this is marginal since the deps aren't migrated to -r1. It will hit only those unlikely users of py2.5 we can still have, and there is some hope that at least some of those deps will be migrated to -r1 (and thus dropped py2.5 support) soon.
(In reply to comment #5) > (In reply to comment #3) > > If this drops support of python 2.5 we'll have to review the rdeps. > > I think I have removed 2.5 because the code didn't actually support it. If I > recall correctly, it used 'with' in a few places which triggered errors. 2.9 fails but 2.8 still works. Either way I'm happy to drop it (testing wxpython is the only reason I still have 2.5 installed). > (In reply to comment #4) > > This breaks: > > > > dev-python/etsdevtools > > dev-python/ipython (<0.12 ie. all stable versions) > > dev-python/python-dsv > > dev-python/traitsbackendwx > > dev-python/winpdb > > dev-util/spe > > media-gfx/dispcalgui > > media-gfx/fontypython (i just fixed this in 0.4.4, needs to be stabilized > > though) > > I'd say that this is marginal since the deps aren't migrated to -r1. It will > hit only those unlikely users of py2.5 we can still have, and there is some > hope that at least some of those deps will be migrated to -r1 (and thus > dropped py2.5 support) soon. Fair enough. While I don't like breaking packages I think that revbumping all of these and waiting for them to go stable again is overkill. There's a missing flag-o-matic inherit in the 2.9 ebuild (for append-flags). Otherwise looks good to me.
(In reply to comment #6) > (In reply to comment #5) > > (In reply to comment #3) > > > If this drops support of python 2.5 we'll have to review the rdeps. > > > > I think I have removed 2.5 because the code didn't actually support it. If I > > recall correctly, it used 'with' in a few places which triggered errors. > > 2.9 fails but 2.8 still works. Either way I'm happy to drop it (testing > wxpython is the only reason I still have 2.5 installed). Well, I seen some of the modules actually fail but this could've been something minor. Since 2.5 is old and no longer supported in 2.9, I assumed it's not really worth fixing. > > I'd say that this is marginal since the deps aren't migrated to -r1. It will > > hit only those unlikely users of py2.5 we can still have, and there is some > > hope that at least some of those deps will be migrated to -r1 (and thus > > dropped py2.5 support) soon. > > Fair enough. While I don't like breaking packages I think that revbumping > all of these and waiting for them to go stable again is overkill. > > There's a missing flag-o-matic inherit in the 2.9 ebuild (for append-flags). > Otherwise looks good to me. Thanks, fixed and committed.
Reopening. This broke slotting in a somewhat interesting way. If you install both wxpython-2.8.12.1-r1 and wxpython-2.9.4.1-r1 they collide as both install site-packages/wx (they both install two copies of wx, one into site-packages/wx-2.9.4-gtk2 or similar and one straight into site-packages/wx). This is caused by a bad interaction between the wxpython setup.py and distutils-r1.eclass passing --build-lib explicitly. With --build-lib commented out in the eclass everything works again. Here's what goes on in a bit more detail: The wxpython setup.py has two calls to setup(). The first one is for wxpython itself and installs (via extra_path, I think) into a slot-safe directory. The second one is for wxversion.py and installs into site-packages itself (this is then fixed up in the ebuild). But this only works more or less by accident. The distutils install step is not smart enough to install only the target files: it installs everything in the build directory. If --build-lib is not passed to setup.py the first call uses something like build/lib.linux-86_64-2.7 (build-platlib) as build directory (because wx itself contains extension modules), and the second uses build/lib (build-purelib, because the common code is pure python). These two being different makes this dual setup() call work. But distutils-r1.eclass explicitly passes --buid-lib ${BUILD_DIR}/lib, which means the two builds build into the same directory, and the two installs install out of the same directory. So everything is installed twice, once into the versioned directory, once directly into site-packages. I'd argue this is a bug in the wxpython setup.py: this'll always do the wrong thing if you pass --build-lib explicitly (you don't even need to do the build and install phase separately). I don't know how to sensibly fix it. It'd help to have a separate setup.py for each call to setup(), in which case the ebuild could call each with a completely separate build directory. Or perhaps the single setup.py could force a different value for build_base for each call to setup() (which'd have the same effect of untangling the shared build directory).
(In reply to comment #8) > The wxpython setup.py has two calls to setup(). The first one is for > wxpython itself and installs (via extra_path, I think) into a slot-safe > directory. The second one is for wxversion.py and installs into > site-packages itself (this is then fixed up in the ebuild). But this only > works more or less by accident. The distutils install step is not smart > enough to install only the target files: it installs everything in the build > directory. If --build-lib is not passed to setup.py the first call uses > something like build/lib.linux-86_64-2.7 (build-platlib) as build directory > (because wx itself contains extension modules), and the second uses > build/lib (build-purelib, because the common code is pure python). These two > being different makes this dual setup() call work. But distutils-r1.eclass > explicitly passes --buid-lib ${BUILD_DIR}/lib, which means the two builds > build into the same directory, and the two installs install out of the same > directory. So everything is installed twice, once into the versioned > directory, once directly into site-packages. Thanks for the explanation. Do I understand correctly that distutils install step ignores even the package lists passed to it? > I'd argue this is a bug in the wxpython setup.py: this'll always do the > wrong thing if you pass --build-lib explicitly (you don't even need to do > the build and install phase separately). I don't know how to sensibly fix > it. Relying on specific behavior from something that much broken by design like distutils, and not supporting one of the more common arguments to it is a bug. The simplest workaround would be redefining both the build-dirs again in the setup.py. I think a few packages do that actually.
(In reply to comment #9) > Thanks for the explanation. Do I understand correctly that distutils install > step ignores even the package lists passed to it? Yep, both from observing what the build system does in -r0 and -r1 of these ebuilds, and from the following comment in distutils.command.install_lib: # Install everything: simply dump the entire contents of the build # directory to the installation directory (that's the beauty of # having a build directory!) Obviously that assumes the build directory is current, and not shared with an unrelated build, which is more or less what happens here. > > I'd argue this is a bug in the wxpython setup.py: this'll always do the > > wrong thing if you pass --build-lib explicitly (you don't even need to do > > the build and install phase separately). I don't know how to sensibly fix > > it. > > Relying on specific behavior from something that much broken by design like > distutils, and not supporting one of the more common arguments to it is a > bug. > > The simplest workaround would be redefining both the build-dirs again in the > setup.py. I think a few packages do that actually. You could pass --build-platlib "${BUILD_DIR}/lib-${ABI}" --build-purelib "${BUILD_DIR}/lib", or something along those lines. The only ebuild I see touching platlib or purelib is egenix-mx-base, explicitly passing --build-platlib to be ${BUILD_DIR}/lib, which I suspect works around a different kind of breakage triggered by the eclass passing --build-lib: assuming egenix-mx-base uses extension modules it would normally end up with build_lib being build_platlib, but the eclass explicitly sets build_lib while leaving --build-platlib alone, which means those are now different directories. If the eclass explicitly set --build-platlib and --build-purelib, leaving --build-lib alone, that line could probably be removed from the ebuild.
(In reply to comment #10) > > > I'd argue this is a bug in the wxpython setup.py: this'll always do the > > > wrong thing if you pass --build-lib explicitly (you don't even need to do > > > the build and install phase separately). I don't know how to sensibly fix > > > it. > > > > Relying on specific behavior from something that much broken by design like > > distutils, and not supporting one of the more common arguments to it is a > > bug. > > > > The simplest workaround would be redefining both the build-dirs again in the > > setup.py. I think a few packages do that actually. > > You could pass --build-platlib "${BUILD_DIR}/lib-${ABI}" --build-purelib > "${BUILD_DIR}/lib", or something along those lines. Well, since this is intended only as a work-around, that could be even something like '--build-purelib "${BUILD_DIR}"/lib-common'. I'm limited on computing power, could you try appending the following to mydistutilsargs? mydistutilsargs=( #... build --build-purelib "${BUILD_DIR}"/lib-common ) I think 'moving' just one of the directories should satisfy their hack. > The only ebuild I see > touching platlib or purelib is egenix-mx-base, explicitly passing > --build-platlib to be ${BUILD_DIR}/lib, which I suspect works around a > different kind of breakage triggered by the eclass passing --build-lib: That's just their hacky mxSetup. They do some magical overrides and end up overriding build_platlib without respecting --build-lib properly.
No, the easy way to fix this is to *not* pass --build-lib, so that one call of setup() defaults it to platlib and the other to purelib. If you pass --build-lib to setup.py the platlib and purelib values don't matter. You need the "if self.build_lib is None:" in distutils.command.build to trigger for wxpython's setup.py to work right. The only thing distutils uses build-platlib and build-purelib for is to initialize build-lib, *iff* build-lib isn't initialized explicitly. Which distutils-r1.eclass does, and I don't see a sane way to get it not to.
(In reply to comment #12) > No, the easy way to fix this is to *not* pass --build-lib, so that one call > of setup() defaults it to platlib and the other to purelib. 'Easy ways' don't usually require fixing a lot of other ebuilds nor adding more switches to the eclass. > If you pass > --build-lib to setup.py the platlib and purelib values don't matter. You > need the "if self.build_lib is None:" in distutils.command.build to trigger > for wxpython's setup.py to work right. > > The only thing distutils uses build-platlib and build-purelib for is to > initialize build-lib, *iff* build-lib isn't initialized explicitly. Which > distutils-r1.eclass does, and I don't see a sane way to get it not to. Eh, this thing is terribly mis-designed. No wonder why Python folks have abandoned it... Ok. As an alternative, I think I can make distutils-r1 pass --build-platlib and --build-purelib directly. This should keep all other stuff working, and at the same make it possible to get wxwidgets to work. Is that ok?
(In reply to comment #13) > Ok. As an alternative, I think I can make distutils-r1 pass --build-platlib > and --build-purelib directly. This should keep all other stuff working, and > at the same make it possible to get wxwidgets to work. Is that ok? Yes, that's what I was trying to suggest. I changed the eclass to pass --build-purelib ${BUILD_DIR}/lib --build-platlib ${BUILD_DIR}/lib-${ABI}, removed --build-lib, and adjusted distutils-r1_run_phase accordingly, and that worked. Normally I'd be against changing the eclass for this, but I'm optimistic about this not breaking anything else (and I suspect you can remove the platlib override from the egenix-mx-base afterwards), as it makes distutils driven by distutils-r1.eclass more similar than distutils invoked "normally" (no extra flags passed).
(In reply to comment #14) > (In reply to comment #13) > > Ok. As an alternative, I think I can make distutils-r1 pass --build-platlib > > and --build-purelib directly. This should keep all other stuff working, and > > at the same make it possible to get wxwidgets to work. Is that ok? > > Yes, that's what I was trying to suggest. I changed the eclass to pass > --build-purelib ${BUILD_DIR}/lib --build-platlib ${BUILD_DIR}/lib-${ABI}, > removed --build-lib, and adjusted distutils-r1_run_phase accordingly, and > that worked. No. Eclass shall pass the same directory as both paths so that we could reuse the built sources during the tests.
(In reply to comment #15) > > Yes, that's what I was trying to suggest. I changed the eclass to pass > > --build-purelib ${BUILD_DIR}/lib --build-platlib ${BUILD_DIR}/lib-${ABI}, > > removed --build-lib, and adjusted distutils-r1_run_phase accordingly, and > > that worked. > > No. Eclass shall pass the same directory as both paths You'll still break this ebuild, and any other ebuilds relying on platlib and purelib being different (which should be rare but might exist) > so that we could > reuse the built sources during the tests. You shouldn't normally need purelib and platlib to be the same for that. A normal setup.py will only use one (which is determined by whether or not it's pure-python). You can just use both paths to PYTHONPATH and it'll work. Please provide a counterexample if this is not the case.
(In reply to comment #16) > (In reply to comment #15) > > > Yes, that's what I was trying to suggest. I changed the eclass to pass > > > --build-purelib ${BUILD_DIR}/lib --build-platlib ${BUILD_DIR}/lib-${ABI}, > > > removed --build-lib, and adjusted distutils-r1_run_phase accordingly, and > > > that worked. > > > > No. Eclass shall pass the same directory as both paths > > You'll still break this ebuild, and any other ebuilds relying on platlib and > purelib being different (which should be rare but might exist) But you will be able to override it in the ebuild. > > so that we could > > reuse the built sources during the tests. > > You shouldn't normally need purelib and platlib to be the same for that. A > normal setup.py will only use one (which is determined by whether or not > it's pure-python). You can just use both paths to PYTHONPATH and it'll work. > Please provide a counterexample if this is not the case. If distutils actually works like that, I'd be happy to do that. But we'd have to check the existing ebuilds. Let me check that...
(In reply to comment #17) > > > so that we could > > > reuse the built sources during the tests. > > > > You shouldn't normally need purelib and platlib to be the same for that. A > > normal setup.py will only use one (which is determined by whether or not > > it's pure-python). You can just use both paths to PYTHONPATH and it'll work. > > Please provide a counterexample if this is not the case. > > If distutils actually works like that, I'd be happy to do that. But we'd > have to check the existing ebuilds. Let me check that... Ok, you are correct indeed. However, there's still the case of ebuilds using paths based on "${BUILD_DIR}"/lib. Right now, ebuilds can safely assume that the interim files are stored there. With the change, they would actually have to keep track of existence of extension modules. Since wxwidgets hackery is really ugly and depending on very specific fragment of distutils behavior, I believe it's a corner case which we shouldn't sacrifice the comfort of existing solution upon.
Ok, in that case you'll have to either hack up the wxpython ebuilds to cope (I see no good way of doing that, unless duplicating esetup.py and everything leading up to it is acceptable), or hack up the wxpython build system to cope (again not pretty, I think the tidiest approach would be having it grow two setup.pys so you can pass them different build directories), or mask the -r1 versions of the wxpython ebuilds, as they're currently broken (anything pulling in multiple slots breaks)
...although on second thought I should add I still prefer passing platlib and purelib, since it seems considerably more sensible to me to keep distutils-r1 doing something similar to what regular setup.py invocation does, than for ebuilds to make assumptions about what ends up where in the build directory. I think ebuilds using those files are at least as guilty of "depending on very specific fragment of distutils behavior" as wxpython is.
I'm not following you. If esetup.py did: --build-platlib $BUILD_DIR/lib --build-purelib $BUILD_DIR/lib then wxpython could just do: esetup.py ... build --build-purelib $BUILD_DIR/lib-common and that should fix the issue by moving one of the build-dirs. Correct?
(In reply to comment #21) > I'm not following you. > > If esetup.py did: > > --build-platlib $BUILD_DIR/lib --build-purelib $BUILD_DIR/lib > > then wxpython could just do: > > esetup.py ... build --build-purelib $BUILD_DIR/lib-common > > and that should fix the issue by moving one of the build-dirs. Correct? Ahh, I somehow overlooked that option. Yes, that should work. I think.
*** Bug 462188 has been marked as a duplicate of this bug. ***
Feel free to commit whatever works. Python is not my area of expertise. ;p
Committed a fix quickly. You may want to revbump it though, I have to leave already.
