Porting dev-python/numpy from distutils to meson : proof of concept Reproducible: Always Upstream has already done the hard part for life after distutils ... # Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 EAPI=8 EGIT_REPO_URI="https://github.com/${PN}/${PN}.git" PYTHON_COMPAT=( python3_{9..12} pypy3 ) PYTHON_REQ_USE="threads(+)" FORTRAN_NEEDED=lapack inherit flag-o-matic fortran-2 git-r3 meson multiprocessing pypi python-r1 toolchain-funcs DOC_PV=${PV} DESCRIPTION="Fast array and numerical python library" HOMEPAGE=" https://numpy.org/ https://github.com/numpy/numpy/ https://pypi.org/project/numpy/ " SRC_URI="" LICENSE="BSD" SLOT="0" KEYWORDS="" IUSE="lapack test" RDEPEND=" lapack? ( >=virtual/cblas-3.8 >=virtual/lapack-3.8 ) " BDEPEND=" ${RDEPEND} <dev-python/cython-3[${PYTHON_USEDEP}] >=dev-python/cython-0.29.30[${PYTHON_USEDEP}] lapack? ( virtual/pkgconfig ) test? ( $(python_gen_cond_dep ' >=dev-python/cffi-1.14.0[${PYTHON_USEDEP}] ' 'python*') dev-python/charset_normalizer[${PYTHON_USEDEP}] >=dev-python/hypothesis-5.8.0[${PYTHON_USEDEP}] dev-python/pytest-xdist[${PYTHON_USEDEP}] >=dev-python/pytz-2019.3[${PYTHON_USEDEP}] ) " PATCHES=() src_configure() { configuring() { meson_src_configure \ -Dblas="cblas,blas" \ -Dlapack="lapack" \ -Ddisable-svml=false \ -Ddisable-threading=false \ -Ddisable-simd-optimizations=true \ -Drelaxed-strides-debug=false } python_foreach_impl configuring } src_compile() { python_foreach_impl meson_src_compile } src_test() { python_foreach_impl meson_src_test } src_install() { installing() { meson_src_install } python_foreach_impl installing }
Created attachment 862762 [details] build.log
You'll need to use DISTUTILS_USE_PEP517=meson-python, ideally. But yeah, I did take a look at this before, and gave up as it wasn't ready: https://github.com/gentoo/gentoo/pull/28725.
(In reply to Sam James from comment #2) > You'll need to use DISTUTILS_USE_PEP517=meson-python, ideally. I tried that and things failed miserably. I also noticed in the logs that fortran was not getting used at all, so I cleaned that out. # Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 EAPI=8 EGIT_REPO_URI="https://github.com/${PN}/${PN}.git" PYTHON_COMPAT=( python3_{9..12} pypy3 ) PYTHON_REQ_USE="threads(+)" inherit flag-o-matic git-r3 meson multiprocessing pypi python-r1 toolchain-funcs DESCRIPTION="Fast array and numerical python library" HOMEPAGE=" https://numpy.org/ https://github.com/numpy/numpy/ https://pypi.org/project/numpy/ " SRC_URI="" LICENSE="BSD" SLOT="0" KEYWORDS="" IUSE="test" RDEPEND=" >=virtual/cblas-3.8 >=virtual/lapack-3.8 " BDEPEND=" ${RDEPEND} <dev-python/cython-3[${PYTHON_USEDEP}] >=dev-python/cython-0.29.30[${PYTHON_USEDEP}] virtual/pkgconfig test? ( $(python_gen_cond_dep ' >=dev-python/cffi-1.14.0[${PYTHON_USEDEP}] ' 'python*') dev-python/charset_normalizer[${PYTHON_USEDEP}] >=dev-python/hypothesis-5.8.0[${PYTHON_USEDEP}] dev-python/pytest-xdist[${PYTHON_USEDEP}] >=dev-python/pytz-2019.3[${PYTHON_USEDEP}] ) " src_configure() { local emesonargs=( -Dblas="cblas" -Dlapack="lapack" -Ddisable-svml=false -Ddisable-threading=false -Ddisable-simd-optimizations=true -Drelaxed-strides-debug=false ) python_foreach_impl meson_src_configure } src_compile() { python_foreach_impl meson_src_compile } src_test() { python_foreach_impl meson_src_test } src_install() { python_foreach_impl meson_src_install }
(In reply to cyrillic from comment #3) > (In reply to Sam James from comment #2) > > You'll need to use DISTUTILS_USE_PEP517=meson-python, ideally. > > I tried that and things failed miserably. I also noticed in the logs that > fortran was not getting used at all, so I cleaned that out. > Fortran not being used sounds pretty suspicious.
(In reply to Sam James from comment #4) > Fortran not being used sounds pretty suspicious. I agree. Even though things will compile and install without errors, it looks like the meson port is a bit half-baked at the moment.
(In reply to cyrillic from comment #5) > (In reply to Sam James from comment #4) > > Fortran not being used sounds pretty suspicious. > > I agree. Even though things will compile and install without errors, it > looks like the meson port is a bit half-baked at the moment. Yeah :( Unfortunately, I tried last night to get tests running and that didn't go great either.
mgorny and I have got a bit further, but blocked on https://github.com/numpy/numpy/issues/23909 right now..
Made some progress, more later.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=358874f189c9f8d2e1532ba9933ee57ac8942a95 commit 358874f189c9f8d2e1532ba9933ee57ac8942a95 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-12 15:20:37 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-12 15:21:15 +0000 dev-python/numpy: add 1.25.0_rc1 Closes: https://bugs.gentoo.org/907309 Signed-off-by: Sam James <sam@gentoo.org> dev-python/numpy/Manifest | 1 + .../numpy-1.25.0_rc1-meson-pyproject.toml.patch | 392 +++++++++++++++++++++ dev-python/numpy/numpy-1.25.0_rc1.ebuild | 127 +++++++ 3 files changed, 520 insertions(+)
The Fortran thing ended up being a red herring - see https://github.com/numpy/numpy/blob/4af0babbd67edc5db0fc54e2f905ceb64eba925e/INSTALL.rst#L72 (scipy needs it directly, but not numpy).
Very nice ... I will be trying it out shortly.
Created attachment 863753 [details] build.log Compile and install works fine with python3.11 If I bump dev-python/meson-python to allow python3.12 , then dev-python/numpy seems to be working with python3.12 as well.
Thanks! I can add 3.12 to all of its deps (doing it now) but numpy itself fails tests w/ 3.12: ``` FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[ones-1-<subarray>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[ones-1-<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[ones-1-<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[ones-1-<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[zeros-0-<subarray>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[zeros-0-<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[zeros-0-<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_create_delete[zeros-0-<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_item_setting[<subarray>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_item_setting[<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_item_setting[<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_item_setting[<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 3) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape0-index0-2-<subarray>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape0-index0-2-<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape0-index0-2-<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape0-index0-2-<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape1-index1-4-<subarray>] - assert (4294967295 - 4294967295) == (6 * 4) FAILED core/tests/test_regression.py::TestRegression::test_structured_arrays_with_objects2 - AssertionError FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape1-index1-4-<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 4) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape1-index1-4-<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 4) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape1-index1-4-<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 4) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape2-index2-2-<subarray>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape2-index2-2-<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape2-index2-2-<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape2-index2-2-<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape3-index3-2-<subarray>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape3-index3-2-<subarray in field>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape3-index3-2-<structured subarray 1>] - assert (4294967295 - 4294967295) == (6 * 2) FAILED core/tests/test_dtype.py::TestStructuredObjectRefcounting::test_structured_object_indexing[shape3-index3-2-<structured subarray 2>] - assert (4294967295 - 4294967295) == (12 * 2) FAILED lib/tests/test_format.py::test_python2_python3_interoperability - File "<string>", line 1 ======================================================== 30 failed, 31179 passed, 941 skipped, 29 xfailed, 6 xpassed, 979 warnings in 81.67s (0:01:21) ========================================================= * ERROR: dev-python/numpy-1.25.0_rc1::gentoo failed (test phase): ```
Maybe the testsuite depends on distutils even though building numpy no longer does.
(In reply to cyrillic from comment #14) > Maybe the testsuite depends on distutils even though building numpy no > longer does. Nah, these failures are *only* with 3.12. 3.10+3.11 are fine.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=598a530efd89c78a4a079895e362b1db6a4cf54d commit 598a530efd89c78a4a079895e362b1db6a4cf54d Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-26 12:48:09 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-26 13:14:16 +0000 dev-python/numpy: enable py3.12 It turns out the test failures were non-issues and are now fixed upstream. Closes: https://bugs.gentoo.org/907309 Signed-off-by: Sam James <sam@gentoo.org> ...y-1.25.0-skip-python3.12-irrelevant-tests.patch | 187 +++++++++++++++++++++ dev-python/numpy/numpy-1.25.0.ebuild | 3 +- 2 files changed, 189 insertions(+), 1 deletion(-)
