| Summary: | dev-python/numpy: create slot for python2 | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Pacho Ramos <pacho> |
| Component: | Current packages | Assignee: | Gentoo Science Related Packages <sci> |
| Status: | RESOLVED WONTFIX | ||
| Severity: | normal | CC: | fabio.coatti, harrisl, jasmin+gentoo, lssndrbarbieri, mgorny, python, redneb, scott, silvio.gerli, urcindalo |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=662518 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | numpy-1.16.5-r1.ebuild | ||
|
Description
Pacho Ramos
2019-12-17 21:29:33 UTC
Now imagine all the confusion arising from people having two incompatible versions installed simultaneously. What confusion? I remember to have seen (when porting to newer python) some python modules having different versions for older python targets, and that wasn't a major issue for users. In other major distributions is neither a confusion, and I am not suggesting to have 1.16.x for python3, only for python2. That way people not needing python2 support will simply have the newest numpy, while people needing py2 will have both and use the latest they can for both... and I see no problem with that. Please take care that some people (like me) also use Gentoo for work, and there are lots of packages that still need python2 support for numpy. Tools like VMD and PyMol are widely used in scientific community, and we cannot migrate immediately from them. Forcing all of us to simply change the distribution doesn't look acceptable to me. And you know that I help, when possible, to get rid of old packages... but without causing us to simply need to change the distribution (in my case also at home as I also need those tools from there). Of course the package can be renamed instead of using a slot... I don ' t care about that :) Created attachment 600158 [details]
numpy-1.16.5-r1.ebuild
This would be the idea, that looks to work ok on my system
(In reply to Pacho Ramos from comment #0) > The reasoning is explained in bug 662518 > > In summary: there are still lots of packages (usually scientific ones) that > will still need python2 numpy for some time. Current situation makes > impossible to have python 3.8 and 2.7 at the same time due to only > numpy-1.17.x supporting py3.8. > > Other distributions like Debian, Fedora and Arch are already providing > 1.16.x as python2 provider to solve this. > > I volunteer on maintaining this as I need many of those scientific packages > still requiring numpy for python2. I can try to work on an ebuild if you > want. > > Thanks a lot I understand our pains (I work in science too by the way) and what you're trying to achieve. That said, I'm inclined to be against this, because it will greatly reduce the incentive to remove py2 from the tree ("why remove foo/bar? We also have a py2 version of numpy? We are you removing other py2-only applications but not foo/bar?"). Furthermore, this will include a ton of other issues: who does py2 security work? Will you track RHEL to include py2 patches? This is a massive can of worms, and I feel like it increases upstream's complacency instead of reducing it ("just use Gentoo, they still have py2 until 2029"). Here's a compromise: Why not maintain such ebuilds in the science overlay? you can add a SLOTed numpy there for python 2? (In reply to Pacho Ramos from comment #0) > The reasoning is explained in bug 662518 > > In summary: there are still lots of packages (usually scientific ones) that > will still need python2 numpy for some time. Current situation makes > impossible to have python 3.8 and 2.7 at the same time due to only > numpy-1.17.x supporting py3.8. > > Other distributions like Debian, Fedora and Arch are already providing > 1.16.x as python2 provider to solve this. > > I volunteer on maintaining this as I need many of those scientific packages > still requiring numpy for python2. I can try to work on an ebuild if you > want. > > Thanks a lot I understand your pains (I work in science too by the way) and what you're trying to achieve. That said, I'm inclined to be against this, because it will greatly reduce the incentive to remove py2 from the tree ("why remove foo/bar? We also have a py2 version of numpy? We are you removing other py2-only applications but not foo/bar?"). Furthermore, this will include a ton of other issues: who does py2 security work? Will you track RHEL to include py2 patches? This is a massive can of worms, and I feel like it increases upstream's complacency instead of reducing it ("just use Gentoo, they still have py2 until 2029"). Here's a compromise: Why not maintain such ebuilds in the science overlay? you can add a SLOTed numpy there for python 2? Separate package name, please. And no combined py2+py3 revdeps, drop py2 from all of them. Regarding the incentive, personally I don't think upstreams will really take care of us. Even if we have a great support for scientific packages, we are not such a popular distribution, having Centos and Ubuntu LTS with python2 support is a bad enough incentive for them because I think that they mostly look to that more popular distributions. I expect anyway that once python2 will finally stop completely they will be more "pushed". Regarding the overlay... well, in general I prefer to not rely too much on external overlays as, from my experience, they tend to get not really well maintained. Please don't take me wrong, I am not blaming on sci overlay, but I use it due to some corner packages... and many times I need to manually tweak ebuilds because some are in a good shape, others don't because they are really old... Personally I would rely on overlay only if in some months this becomes worse and it's more clear that we cannot keep it in the tree (of course at some point it will be impossible... but maybe not in the next months yet). For example in Gnome2 -> Gnome3 migration, we were able to deal with it in the tree (even with lots of complains from users hating GTK3) without having a gnome-2 overlay and I think it worked better, finding a balance between encouraging migration to gnome3 while not cutting suddenly all the support for gnome2. (In reply to Michał Górny from comment #7) > Separate package name, please. And no combined py2+py3 revdeps, drop py2 > from all of them. Regarding the name, fine. What name do you prefer? python2-numpy like Fedora? About the combined py2+py3 revdeps... could we see it a bit case by case? For example in pygtk bug you can see that I was reviewing it and it's easy to drop the numpy requirement, in applications supporting both, of course it makes sense to simply support python3 only, but there are some libraries that could be harder to achieve *immediately* as they are a bit like numpy (too central). My plan was to try to drop python2 support when possible (usually apps and libs without lots of reverse deps), for the others work on conditional support and open one bug for them to let me track each one in upstream side. For dead things start treecleaning (a bit like we do in treecleaners now). This reminds me a bit about the migration from gnome2 to gnome3 or gtk2 to gtk3. But, of course, count on me for working on this stuff. I was even thinking in joining one of the teams (sci or python)... but I wasn't sure as with python team I can contribute more in "packaging" things, this work with this reverse deps, of course tracking other distributions, helping when possible.. I already try to help as much as I can porting to new python targets (is the reason I hit this so quickly as I was trying to add python 3.8 to my setup) but I have always being unsure about having enough knowledge for that :/ Regarding sci team, the reason I didn't join yet is that, even if I rely on lots of biology and chemistry packages... I have no idea about many other packages maintained by them, some really difficult to build+maintain. Personally, I think I could help more directly in python team... but it's up to you to accept me or not. If you prefer me to contribute this things from "outside", it's ok too. (In reply to Pacho Ramos from comment #8) > Regarding the incentive, personally I don't think upstreams will really take > care of us. Even if we have a great support for scientific packages, we are > not such a popular distribution, having Centos and Ubuntu LTS with python2 > support is a bad enough incentive for them because I think that they mostly > look to that more popular distributions. I expect anyway that once python2 > will finally stop completely they will be more "pushed". > > Regarding the overlay... well, in general I prefer to not rely too much on > external overlays as, from my experience, they tend to get not really well > maintained. Please don't take me wrong, I am not blaming on sci overlay, but > I use it due to some corner packages... and many times I need to manually > tweak ebuilds because some are in a good shape, others don't because they > are really old... Personally I would rely on overlay only if in some months > this becomes worse and it's more clear that we cannot keep it in the tree > (of course at some point it will be impossible... but maybe not in the next > months yet). For example in Gnome2 -> Gnome3 migration, we were able to deal > with it in the tree (even with lots of complains from users hating GTK3) > without having a gnome-2 overlay and I think it worked better, finding a > balance between encouraging migration to gnome3 while not cutting suddenly > all the support for gnome2. > > (In reply to Michał Górny from comment #7) > > Separate package name, please. And no combined py2+py3 revdeps, drop py2 > > from all of them. > > Regarding the name, fine. What name do you prefer? python2-numpy like > Fedora? About the combined py2+py3 revdeps... could we see it a bit case by > case? For example in pygtk bug you can see that I was reviewing it and it's > easy to drop the numpy requirement, in applications supporting both, of > course it makes sense to simply support python3 only, but there are some > libraries that could be harder to achieve *immediately* as they are a bit > like numpy (too central). My plan was to try to drop python2 support when > possible (usually apps and libs without lots of reverse deps), for the > others work on conditional support and open one bug for them to let me track > each one in upstream side. For dead things start treecleaning (a bit like we > do in treecleaners now). This reminds me a bit about the migration from > gnome2 to gnome3 or gtk2 to gtk3. > > But, of course, count on me for working on this stuff. I was even thinking > in joining one of the teams (sci or python)... but I wasn't sure as with > python team I can contribute more in "packaging" things, this work with this > reverse deps, of course tracking other distributions, helping when > possible.. I already try to help as much as I can porting to new python > targets (is the reason I hit this so quickly as I was trying to add python > 3.8 to my setup) but I have always being unsure about having enough > knowledge for that :/ > > Regarding sci team, the reason I didn't join yet is that, even if I rely on > lots of biology and chemistry packages... I have no idea about many other > packages maintained by them, some really difficult to build+maintain. > > Personally, I think I could help more directly in python team... but it's up > to you to accept me or not. If you prefer me to contribute this things from > "outside", it's ok too. When talking about incentive, I don't mean upstreams (they don't even know about Gentoo in most cases). I'm talking about carving out exceptions for sci-* packages that can retain py2-only support, yet we've already started last-riting py2-only packages in other categories (concurrently with trying to reduce the maintenance burden on the python team). The comparison with GTK is not suitable, as GTK has always between orthogonal between versions. With python, the way the USE dependencies work make this much harder, hence I don't think you can use Gnome as an example here. How do you want to address the py2 exception? Sci is special? Given how abandoned sci is in general, lots of people will complain that sci-* stuff gets a py2-exception, but the rest of Gentoo doesn't. Once you give sci a waiver, that's a point of no return and *everyone* will demand one for their special workflows. It is not only sci, as soon as you will see the deptree you will find lots of different packages libs leading to lots of different packages getting affected. At least from now, simply starting to treeclean packages with dead upstreams that are not even planning to port will allow to kill lots of them. After that we will lead with remaining packages. Many of the packages I am talking about are not dead upstream, upstream know the problem but they are still in progress of treecleaning. The mention about gnome was more for showing that it's not the first time I got involved on this kind of "pain" migrations to show that I am not here to simply say "ey, keep my package but I won't do anything else". I have started doing lots of that porting simply from numpy to the reverse deps and I have already catched many packages that can be kicked right now, I am simply waiting the "ok" for starting to review all the numpy deptree. If I am worried is because I have checked the deptree and I try to think about how to start dealing with it right now and progressively checking for packages to be kicked right now, others to drop py2 support in favor of only py3, and the remaining ones that, of course, I will report and follow upstream with their relevant tracker bug here to easily allow all of us to track the progress. On the other hand, keeping situation as-is will simply lead to most people not doing anything and, the worse, not even trying python 3.8 to not hit this issues. In the long term *I think* (maybe I am wrong) that, paradoxically, the way I am suggesting will lead to *effectively* more migration work to get done right now than the other that will likely end up with the job getting delayed and delayed. That is the reason I think I could help a bit more from python team (or outside) than sci team, because, even if for me I hit the problem from sci packages, I am sure that we will see problems like this appearing in packages from many other areas and I would also like to contribute there if I am able to. One advantage of being in python team would be that maybe other maintainers wouldn't complain so much if I revbump some of their apps without py2 support :/ It's something that I wondered when I was checking the reverse deps and seeing candidates that I could right now revbump like that. Hi Pacho, (In reply to Pacho Ramos from comment #8) > Regarding sci team, the reason I didn't join yet is that, even if I rely on > lots of biology and chemistry packages... I have no idea about many other > packages maintained by them, some really difficult to build+maintain. Even if you are inclined to go the other way, I would like to send you a welcome signal to join sci. As long as you care of biology and chemistry packages, there is no need to maintain packages outside your field. That's how we work now. Yours, Benda OK, thanks At first I would like to start with python for this porting work... but of course I will help with the sci packages I am using, but maybe as co-maintainer of them before :/ @python, what do you think finally? Thanks (In reply to Pacho Ramos from comment #14) > @python, what do you think finally? > > Thanks I'm still skeptical, because this will likely extend the EOL of py2 to 2029. At what point do we finally kill py2? I want some concrete and agreed upon dates. I cannot give you a concrete date... that would be lying, but I think that this will lead to killing it faster than pretending to simply treeclean all reverse deps right now (that, in fact, would only lead to lots of flames and fights and, at the end, paradoxically, would lead to slower deprecation as many people wouldn't cooperate) In fact, as soon as I start porting reverse deps and looking the remainers with their upstream (and starting to kill those without upstream), we will be effectively more close to kill py2 sooner than current way that will simply lead to people neither moving to python3.8. My intention is to maintain *for now* this parts waiting for *active upstreams* for most of the reverse deps for porting. I think that is the way other distributions are doing... for example, when I was reviewing reverse deps, I saw Fedora having patches for some of the reverse deps porting them to python3. Of course, when I am going to go through them, I will directly port them to python3 only or, for example, for reverse deps like deluge, keep only python3 support to simply drop python2 requirement. Of course, dead packages with dead upstream won't block python2 removal, and I was planning to start killing them when I go into trying to port them. (In reply to Pacho Ramos from comment #16) > I cannot give you a concrete date... that would be lying, but I think that > this will lead to killing it faster than pretending to simply treeclean all > reverse deps right now (that, in fact, would only lead to lots of flames and > fights and, at the end, paradoxically, would lead to slower deprecation as > many people wouldn't cooperate) > > In fact, as soon as I start porting reverse deps and looking the remainers > with their upstream (and starting to kill those without upstream), we will > be effectively more close to kill py2 sooner than current way that will > simply lead to people neither moving to python3.8. > > My intention is to maintain *for now* this parts waiting for *active > upstreams* for most of the reverse deps for porting. I think that is the way > other distributions are doing... for example, when I was reviewing reverse > deps, I saw Fedora having patches for some of the reverse deps porting them > to python3. Of course, when I am going to go through them, I will directly > port them to python3 only or, for example, for reverse deps like deluge, > keep only python3 support to simply drop python2 requirement. Of course, > dead packages with dead upstream won't block python2 removal, and I was > planning to start killing them when I go into trying to port them. This smooth transition makes more sense to me. The only reason that we are not doing it is lack of man power. It is fine provided you would like to devote to this plan and the new slot does not cause conflicts during `emerge`. (In reply to Pacho Ramos from comment #14) > @python, what do you think finally? As I said, I'm fine with it as long as it's a separate package. (In reply to Michał Górny from comment #18) > (In reply to Pacho Ramos from comment #14) > > @python, what do you think finally? > > As I said, I'm fine with it as long as it's a separate package. Hi Michał, do you have some reference for explaining why a separate package is superior than a separate SLOT? Maybe you have already written it somewhere and I have missed it. Benda Separate package is better because it's separate and doesn't pretend to be officially supported part of numpy. It's also better because it doesn't carry all the random mislogic associated with slot which isn't desirable here and therefore avoid all the issues caused by developers failing to account for it correctly. Here's my compromise solution: I am fine with a differently named numpy package (and I second mgorny's opposition to using a SLOT, which proved to be highly confusing for openssl and other packages), but only for packages where a py3 port is reasonably likely to happen and there is upstream activity or a commitment has been made. Packages for which there is no activity and they are clearly abandoned cannot be moved to numpy-py27. For instance sci-biology/tophat is dead, and the authors are unwilling to expend any more energy on porting it. Changing this package to use the numpy-py27 package only serves to extend the lifetime of py2.7 without any hope of porting, and I will oppose that. (In reply to David Seifert from comment #21) > Here's my compromise solution: I am fine with a differently named numpy > package (and I second mgorny's opposition to using a SLOT, which proved to > be highly confusing for openssl and other packages), but only for packages > where a py3 port is reasonably likely to happen and there is upstream > activity or a commitment has been made. Packages for which there is no > activity and they are clearly abandoned cannot be moved to numpy-py27. > > For instance sci-biology/tophat is dead, and the authors are unwilling to > expend any more energy on porting it. Changing this package to use the > numpy-py27 package only serves to extend the lifetime of py2.7 without any > hope of porting, and I will oppose that. I agree Then, only two final questions: what name do you prefer for the package? python2-numpy? numpy-python2? (I don't care) And, finally, do you prefer if I join to python team or do you prefer me to do it from outside? Thanks And... then? :) (In reply to Pacho Ramos from comment #23) > And... then? :) Maybe numpy-python2, so it'll tab-complete better. You can add python@ as co-maint, if you want to let us chime in occassionally but make yourself the primary maintainer. OK, sure And, about me joining too python team for the porting of python packages (even being the maintainer of numpy-python2)? Sure. Feel free to join #gentoo-python too. OK, I will do Best regards The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=00d6926266e8a86f6d7f49f1e78a3a674d449a7c commit 00d6926266e8a86f6d7f49f1e78a3a674d449a7c Author: Pacho Ramos <pacho@gentoo.org> AuthorDate: 2019-12-25 15:36:54 +0000 Commit: Pacho Ramos <pacho@gentoo.org> CommitDate: 2019-12-25 16:57:12 +0000 dev-python/numpy-python2: Package supporting py2 for smoother transition As discussed in https://bugs.gentoo.org/703240 I will maintain a numpy-python2 package to allow easier transition of reverse deps allowing to still port them to python 3.8. The idea of this package is to drop it as soon as possible, as a consequence: - python2-only packages depending on this should have a bug opened and blocking bug #703754 to allow us to track upstream progress on python3 porting. As a consequence, if the package upstream is dead or not planning to port to python3 ever, it should be treecleaned instead of adding a dependency on this package. - python3 packages also needing python2 support. During a transition period, it's possible that we will need to temporally allow some packages (usually libraries) to support python2 until other reverse deps of that libs are handled. You should then open a bug report blocking bug #703756 to follow the progress on porting them completely to python3. As a consequence, if your package supports both, you must try to simply remove python2 support over depending on this package, while only adding a dep on numpy-python2 if it is really hard to handle all reverse deps immediately. In that case, a bug report need to be opened as explained. Closes: https://bugs.gentoo.org/703240 Package-Manager: Portage-2.3.82, Repoman-2.3.20 Signed-off-by: Pacho Ramos <pacho@gentoo.org> dev-python/numpy-python2/Manifest | 4 + .../files/numpy-1.15.4-no-hardcode-blas.patch | 76 +++++++++ dev-python/numpy-python2/metadata.xml | 7 + .../numpy-python2/numpy-python2-1.16.5.ebuild | 169 +++++++++++++++++++++ 4 files changed, 256 insertions(+) (In reply to Larry the Git Cow from comment #28) > The bug has been closed via the following commit(s): > > https://gitweb.gentoo.org/repo/gentoo.git/commit/ > ?id=00d6926266e8a86f6d7f49f1e78a3a674d449a7c > > commit 00d6926266e8a86f6d7f49f1e78a3a674d449a7c > Author: Pacho Ramos <pacho@gentoo.org> > AuthorDate: 2019-12-25 15:36:54 +0000 > Commit: Pacho Ramos <pacho@gentoo.org> > CommitDate: 2019-12-25 16:57:12 +0000 > > dev-python/numpy-python2: Package supporting py2 for smoother transition > > As discussed in https://bugs.gentoo.org/703240 I will maintain a > numpy-python2 package to allow easier transition of reverse deps > allowing to > still port them to python 3.8. > > The idea of this package is to drop it as soon as possible, as a > consequence: > - python2-only packages depending on this should have a bug opened and > blocking bug #703754 to allow us to track upstream progress on python3 > porting. As a consequence, if the package upstream is dead or not > planning > to port to python3 ever, it should be treecleaned instead of adding a > dependency on this package. > - python3 packages also needing python2 support. During a transition > period, > it's possible that we will need to temporally allow some packages > (usually > libraries) to support python2 until other reverse deps of that libs are > handled. You should then open a bug report blocking bug #703756 to follow > the progress on porting them completely to python3. As a consequence, if > your package supports both, you must try to simply remove python2 support > over depending on this package, while only adding a dep on numpy-python2 > if > it is really hard to handle all reverse deps immediately. In that case, a > bug report need to be opened as explained. > > Closes: https://bugs.gentoo.org/703240 > Package-Manager: Portage-2.3.82, Repoman-2.3.20 > Signed-off-by: Pacho Ramos <pacho@gentoo.org> > > dev-python/numpy-python2/Manifest | 4 + > .../files/numpy-1.15.4-no-hardcode-blas.patch | 76 +++++++++ > dev-python/numpy-python2/metadata.xml | 7 + > .../numpy-python2/numpy-python2-1.16.5.ebuild | 169 > +++++++++++++++++++++ > 4 files changed, 256 insertions(+) Thank you for pushing it forward! Reopening, because leads to unresolvable blockers now, because some packages like =dev-python/pygame-1.9.6-r1 are pulling in numpy-python2 now, while others like dev-python/pygtk still depend on dev-python/numpy with python2_7. So, all reverse dependencies of numpy should be updated, and until then numpy-python2 should be package.masked. (In reply to Ulrich Müller from comment #30) > Reopening, because leads to unresolvable blockers now, because some packages > like =dev-python/pygame-1.9.6-r1 are pulling in numpy-python2 now, while > others like dev-python/pygtk still depend on dev-python/numpy with python2_7. > > So, all reverse dependencies of numpy should be updated, and until then > numpy-python2 should be package.masked. This is generally why I was hesitant to allow this to proceed in the first place, as the bifurcation needs to be done in one atomic change for this whole thing to work without ugly blockers. But, the blockers were already there, as soon as newer numpy-1.17 was pulled, I was unable to update due to all the blockers due to some packages wanting to downgrade numpy while others updating it due to python 3.8. It is not a regression. We're working on a solution using virtuals. Thanks David (In reply to Pacho Ramos from comment #32) > But, the blockers were already there, as soon as newer numpy-1.17 was > pulled, I was unable to update due to all the blockers due to some packages > wanting to downgrade numpy while others updating it due to python 3.8. It is > not a regression. Ok Pacho, we've discussed the options, and it boils down to the following: We will undo the changes, and require you to do a *complete* bifurcation of the numpy-py2/py3-dependent depgraphs. What this means: BEFORE: sci-chemistry/foo RDEPEND on numpy[py2-only] some-app/bla RDEPEND on numpy[py2+py3] AFTER: sci-chemistry/foo RDEPEND on dev-python/numpy-python2[py2] some-app/bla RDEPEND on numpy[py3-only] Or said differently: After a series of commits from you which you should do via CI, we can remove python2.7 from PYTHON_COMPAT on all numpy ebuilds. Please also consider using IRC, communicating and pondering ideas via bugzilla is not great team interaction. For what is worth, I don't like this idea. Please kill python2 from ::gentoo, you can move packages that need python2 to the ::science overlay or to a new ::python2 overlay. I think the ship has sailed on this, so closing. We're nearly there wrt Python 2 removal and it's already gone from distutils etc. |
