Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 940507 - dev-lang/python:3.13 -gil fails to install
Summary: dev-lang/python:3.13 -gil fails to install
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords: PullRequest
Depends on:
Blocks:
 
Reported: 2024-09-30 07:01 UTC by Eray Aslan
Modified: 2024-10-19 14:24 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eray Aslan gentoo-dev 2024-09-30 07:01:25 UTC
building with USE=-gil fails install stage with

[...]
rm: cannot remove '/var/tmp/portage/dev-lang/python-3.13.0_rc2_p2/image/usr/lib/python3.13/ensurepip/_bundled': No such file or directory

apparently, free threaded CPython ABI gets a "t" at the end. The following patch for python-3.13 ebuilds seems to work:

--- python-3.13.0_rc2_p2.ebuild 2024-09-24 07:23:39.629610296 +0200
+++ python-3.13.0_rc2_p2.ebuild 2024-09-30 08:48:54.029844315 +0200
@@ -537,7 +537,9 @@
 }
 
 src_install() {
-       local libdir=${ED}/usr/lib/python${PYVER}
+       local t=""
+       ! use gil && t="t"
+       local libdir=${ED}/usr/lib/python${PYVER}${t}
 
        # -j1 hack for now for bug #843458
        emake -j1 DESTDIR="${D}" TEST_MODULES=no altinstall
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2024-10-08 15:11:35 UTC
Ok, so apparently freethreaded is meant to be completely separate, with its own extension ABI, .so suffix and wheel naming.  Therefore, I think the cleanest way forward would be to move it to a separate package, and a separate PYTHON_TARGETS value.
Comment 3 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-10-08 15:12:01 UTC
ok
Comment 4 Eray Aslan gentoo-dev 2024-10-09 06:53:50 UTC
Sounds good. Thank you. I would also help with experimentation.

Semi related question: Any thoughts on how to handle free threaded versions of libraries like numpy?
Comment 5 Eray Aslan gentoo-dev 2024-10-09 06:54:29 UTC
*It would*
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2024-10-09 08:51:50 UTC
(In reply to Eray Aslan from comment #4)
> Semi related question: Any thoughts on how to handle free threaded versions
> of libraries like numpy?

What about them?  It's a separate target, everything is separate and added into COMPAT separately.
Comment 7 Eray Aslan gentoo-dev 2024-10-10 12:33:52 UTC
(In reply to Michał Górny from comment #6)
> What about them?  It's a separate target, everything is separate and added
> into COMPAT separately.

yes, that part is OK. What I meant to ask is how we will (or will we) support packaging both the regular and free threaded versions of a library like numpy?

For example, https://github.com/numpy/numpy/commit/2d7ede41ce54be25fb20efac706c578a535bb5b4 added:

https://files.pythonhosted.org/packages/7c/8e/fc1fdd83a55476765329ac2913321c4aed5b082a7915095628c4ca30ea72/numpy-2.1.2-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

(vs https://files.pythonhosted.org/packages/44/26/63f5f4e5089654dfb858f4892215ed968cd1a68e6f4a83f9961f84f855cb/numpy-2.1.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-10-10 12:37:30 UTC
The different PYTHON_TARGETS will be co-installable, just like they are now?
Comment 9 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-10-10 12:43:28 UTC
(In reply to Sam James from comment #8)
> The different PYTHON_TARGETS will be co-installable, just like they are now?

i.e. you will have a free-threaded numpy in /usr/lib/python3.13t/site-packages/numpy, and a regular GIL numpy in /usr/lib/python3.13/site-packages, if you have PYTHON_TARGETS="python3_13 python3_13t"?
Comment 10 Eray Aslan gentoo-dev 2024-10-10 13:48:32 UTC
(In reply to Sam James from comment #9)
> i.e. you will have a free-threaded numpy in
> /usr/lib/python3.13t/site-packages/numpy, and a regular GIL numpy in
> /usr/lib/python3.13/site-packages, if you have PYTHON_TARGETS="python3_13
> python3_13t"?

the way you build free-threaded and regular GIL numpy are different (the commit above adds an env variable to the build environment for example). So all relevant ebuilds will need to change to support it. this is not just an eclass level change. and it is fine I am just trying to make sure I understand it correctly

we will also not be allowing mixing and matching, i.e. free-threaded library in a regular GIL python env (or vice versa). no strong objections but it might prove to be too restrictive for some use case

fwiw, https://py-free-threading.github.io/tracking/ seems to be good source. the whole data stack from bottom up seems to be trying to go the free-threaded route. meta seems to be pushing for it
Comment 11 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-10-10 13:51:11 UTC
(In reply to Eray Aslan from comment #10)
> (In reply to Sam James from comment #9)
> > i.e. you will have a free-threaded numpy in
> > /usr/lib/python3.13t/site-packages/numpy, and a regular GIL numpy in
> > /usr/lib/python3.13/site-packages, if you have PYTHON_TARGETS="python3_13
> > python3_13t"?
> 
> the way you build free-threaded and regular GIL numpy are different (the
> commit above adds an env variable to the build environment for example). So
> all relevant ebuilds will need to change to support it. this is not just an
> eclass level change. and it is fine I am just trying to make sure I
> understand it correctly
> 

Well, sure, that'll be added when testing 3.13t for an ebuild. But note that CIBW_FREE_THREADED_SUPPORT is just for cibuildwheel anyway, not for meson.

> we will also not be allowing mixing and matching, i.e. free-threaded library
> in a regular GIL python env (or vice versa). no strong objections but it
> might prove to be too restrictive for some use case

I don't think we could sanely do it any other way tbh.

> 
> fwiw, https://py-free-threading.github.io/tracking/ seems to be good source.
> the whole data stack from bottom up seems to be trying to go the
> free-threaded route. meta seems to be pushing for it

Thanks!
Comment 12 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-10-10 13:56:47 UTC
(In reply to Sam James from comment #11)
> (In reply to Eray Aslan from comment #10)
> > (In reply to Sam James from comment #9)
> > > i.e. you will have a free-threaded numpy in
> > > /usr/lib/python3.13t/site-packages/numpy, and a regular GIL numpy in
> > > /usr/lib/python3.13/site-packages, if you have PYTHON_TARGETS="python3_13
> > > python3_13t"?
> > 
> > the way you build free-threaded and regular GIL numpy are different (the
> > commit above adds an env variable to the build environment for example). So
> > all relevant ebuilds will need to change to support it. this is not just an
> > eclass level change. and it is fine I am just trying to make sure I
> > understand it correctly
> > 
> 
> Well, sure, that'll be added when testing 3.13t for an ebuild. But note that
> CIBW_FREE_THREADED_SUPPORT is just for cibuildwheel anyway, not for meson.
> 

What we might want to do is some QA check for 3.13t + DISTUTILS_EXT to make sure that the build really was marked as supported free-threading.
Comment 13 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2024-10-19 14:24:11 UTC
We have dev-lang/python:3.13t for this now.