Bug 912051 – sys-apps/portage-3.0.50: fails to install for a target if meson[python_targets_$target] is not installed (e.g. portage[python_targets_python3_11 fails without dev-util/meson[...]) with site-packages not found error

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description konsolebox 2023-08-11 00:45:38 UTC]

Created attachment 867597 [details]
build.log

Exact error message is:

find: ‘/var/tmp/portage/sys-apps/portage-3.0.50/image/usr/lib/python3.11/site-packages’: No such file or directory

See build.log for details.

I first encountered this while trying to install a forked sys-apps/portage-3.0.50 in my local overlay.  I had to do this in order to fix it:

[ebuild     U  ] dev-python/setuptools-67.8.0-r1::gentoo [67.7.2::gentoo] USE="-test" PYTHON_TARGETS="python3_10 python3_11 (-pypy3) (-python3_12)" 2,435 KiB
[ebuild     U  ] dev-util/meson-1.1.1::gentoo [0.64.1::gentoo] USE="(-test)" PYTHON_TARGETS="python3_10 python3_11* (-pypy3) (-python3_12) (-python3_9%)" 2,076 KiB

The command was something like `emerge -av1 meson`.

Notice that meson had to merge with python3_11 flag newly enabled while setuptools did not.  I think meson.eclass needs to respect PYTHON_COMPAT and apply `[python_targets_*,...]` to its generated dev-util/meson dependency rule otherwise people would always have to depend on @world again just to fix their systems.  Either way this is clearly a broken/inconsistent dependency rule issue.

I verified the problem by reinstalling meson with python_targets_python3_11 disabled and running `ebuild merge` directly against ::gentoo's portage-3.0.50.ebuild.

Also before sending this bug report, I decided to also temporarily remove sys-apps/portage::gentoo from package.mask and run `emerge -av sys-apps/portage::gentoo` just to leave no doubts.  Still only one package tried to emerge and same error appears.

[Comment 1 Sam James 2023-08-11 00:50:17 UTC]

Please include emerge --info.

> Notice that meson had to merge with python3_11 flag newly enabled while
> setuptools did not.  I think meson.eclass needs to respect PYTHON_COMPAT and

I don't think so. meson is usually run as just a binary. If that's not the case here, the dependency in the portage ebuild should be fixed

[Comment 2 konsolebox 2023-08-11 01:01:03 UTC]

(In reply to Sam James from comment #1)
> Please include emerge --info.

I don't think this is necessary and the build.log suffice.  Even my report already suffices.  Please just try to reproduce.

1. Install dev-util/meson with USE=-python_targets_python3_11
2. Merge portage-3.0.50

If that doesn't cause an error, my report is invalid.

> > Notice that meson had to merge with python3_11 flag newly enabled while
> > setuptools did not.  I think meson.eclass needs to respect PYTHON_COMPAT and
> 
> I don't think so. meson is usually run as just a binary. If that's not the
> case here, the dependency in the portage ebuild should be fixed

If meson is just run as a binary and other files of it aren't dependent on then python version - when perhaps they are required by other packages during build-time, shouldn't it just be installed as a single target then?  If it can't be converted to a single target python install for some reason, then it has to respect PYTHON_COMPAT.

[Comment 3 Sam James 2023-08-11 01:22:38 UTC]

(In reply to konsolebox from comment #2)
> (In reply to Sam James from comment #1)
> > Please include emerge --info.
> 
> I don't think this is necessary and the build.log suffice.  Even my report
> already suffices.  Please just try to reproduce.

We always ask for emerge --info in bugs as it avoids wasting time with issues. In future, please follow the procedure.

> > > Notice that meson had to merge with python3_11 flag newly enabled while
> > > setuptools did not.  I think meson.eclass needs to respect PYTHON_COMPAT and
> > 
> > I don't think so. meson is usually run as just a binary. If that's not the
> > case here, the dependency in the portage ebuild should be fixed
> 
> If meson is just run as a binary and other files of it aren't dependent on
> then python version - when perhaps they are required by other packages
> during build-time, shouldn't it just be installed as a single target then? 
> If it can't be converted to a single target python install for some reason,
> then it has to respect PYTHON_COMPAT.

No, because things *CAN* import it, but many things don't. meson-python, for example, does. Normal things just calling `meson` do not.

> 
> 1. Install dev-util/meson with USE=-python_targets_python3_11
> 2. Merge portage-3.0.50
> 
> If that doesn't cause an error, my report is invalid.
> 

Reproduced.

[Comment 4 konsolebox 2023-08-11 03:20:03 UTC]

(In reply to Sam James from comment #3)
> No, because things *CAN* import it, but many things don't. meson-python, for
> example, does. Normal things just calling `meson` do not.

Also consider python-version-specific variables that might be inherited by meson which may have affected some stuff in my_src_install.

I "fixed" the package by manually adding dev-util/meson[${PYTHON_USEDEP}] to BDEPEND but I won't be submitting a patch since besides it's trivial I think it's a dirty workaround.  For one I think meson.eclass has to allow itself to be configured to assign dev-util/meson[${PYTHON_USEDEP}] to BDEPEND instead, instead of just dev-util/meson.  Second is the cause itself has to be thoroughly identified and a comment needs to be added that explains why [${PYTHON_USEDEP}] has to be used.

> Reproduced.

Please set this report as confirmed then.

[Comment 5 Sam James 2023-08-11 03:24:46 UTC]

(In reply to konsolebox from comment #4)
> Also consider python-version-specific variables that might be inherited by
> meson which may have affected some stuff in my_src_install.
> 

Yep, possibly.

> I "fixed" the package by manually adding dev-util/meson[${PYTHON_USEDEP}] to
> BDEPEND but I won't be submitting a patch since besides it's trivial I think
> it's a dirty workaround.  For one I think meson.eclass has to allow itself
> to be configured to assign dev-util/meson[${PYTHON_USEDEP}] to BDEPEND
> instead, instead of just dev-util/meson

It's likely that any cases of this should be doing it themselves rather than having the eclass do it.

> Second is the cause itself has to
> be thoroughly identified and a comment needs to be added that explains why
> [${PYTHON_USEDEP}] has to be used.
> 

That's why the bug remains open.

> Please set this report as confirmed then.

We don't in general use UNCONFIRMED/CONFIRMED, but if it makes you happy, I can do it.

[Comment 6 Mike Gilbert 2023-08-11 17:55:35 UTC]

I think we can probably avoid the dev-util/meson[${PYTHON_USEDEP}] dependency by passing EPYTHON to the python.find_installation() call in meson.build.

https://gitweb.gentoo.org/proj/portage.git/tree/meson.build?h=portage-3.0.50#n11

[Comment 7 James Le Cuirot 2023-08-11 18:24:40 UTC]

That shouldn't be necessary. It's supposed to pick it up from the PATH, which the eclass sets up. I haven't quite wrapped my head around this one yet, but I'll have a play around.

[Comment 8 James Le Cuirot 2023-08-11 20:21:20 UTC]

I've reproduced it. It's very odd. Meson really shouldn't care about what version it's running under or installed for. Hell, it's supposed to be able to detect Python 2. It feels like a Meson bug.

Interestingly, if I run Meson with 3.12 from a git clone of its repo, but with 3.11 in the PATH, then it does work.

I'll keep digging.

[Comment 9 James Le Cuirot 2023-08-11 20:22:49 UTC]

(In reply to James Le Cuirot from comment #8)
> Interestingly, if I run Meson with 3.12 from a git clone of its repo, but
> with 3.11 in the PATH, then it does work.

Cancel that, it doesn't work. I forgot that the shebang for meson.py was "/usr/bin/env python3" rather than "/usr/bin/python3". Still interesting though.

[Comment 10 James Le Cuirot 2023-08-11 23:18:28 UTC]

Okay, I see what's happening now.

I was told that Meson respects the PATH. That's actually not true in this case. It, wrongly in my opinion, simply returns the path of the Python interpreter that is running Meson.

https://github.com/mesonbuild/meson/blob/90ce0841441506e3f409ab59ded1df8f2e6e7363/mesonbuild/modules/python.py#L384

I had also thought that Meson was always running with your preferred Python version. That's not true either. EPYTHON is set when Meson runs and python-exec respects that, so it'll use the version the ebuild is currently building for, assuming Meson is available for that version. When this bug is triggered, Meson is unavailable for some Python version, so it falls back to the preferred Python version instead.

You can avoid this by adding "python" to the toolchain file. We don't provide a good way to customise the toolchain file at the moment, so that isn't entirely straightforward. It's probably the best short term solution though.

I think the proper fix would be to make Meson respect the PATH.

[Comment 11 James Le Cuirot 2023-08-12 09:08:28 UTC]

The Meson fix is actually quite simple. Please try it out. If upstream is happy (hi Eli!) then we could just backport this.

[Comment 12 Eli Schwartz 2023-08-13 02:08:00 UTC]

> No, because things *CAN* import it, but many things don't. meson-python, for example, does. Normal things just calling `meson` do not.

meson-python invokes meson as a CLI tool using subprocess.run, and in general no software should ever import meson other than meson's own entrypoint script. There is no public API at all -- even the location of the main() function may change at any time as an internal implementation detail, but most probably will not because there is no real need.

> I was told that Meson respects the PATH. That's actually not true in this case. It, wrongly in my opinion, simply returns the path of the Python interpreter that is running Meson.

This is some pretty old code and it's been in the back of my mind to fix it for a while. What actually happens is a bit complex:

- find_installation() with no arguments will use meson's own sys.executable
- find_installation('python3') will use whichever python3 executable is on PATH
- find_installation(get_option('python_bin')) ditto except selected by the project option -Dpython_bin=whatever
- setting the binary path for `python = 'whatever'` in a machine file overrides detection

I agree that ideally without arguments we would first look up a python3 from PATH. Also if python3 is explicitly specified but cannot be found (like on crazy FreeBSD systems where they refuse to install a python3 symlink to python3.9 or whatever is current), we should fall back to sys.executable. And the machine file lookup should prefer checking for a versioned python entry before checking for an unversioned entry.


Although the short-term workaround is one of:
- pass the python binary you want as an argument to find_installation
- have the portage ebuild pass EPYTHON into the machine file

> You can avoid this by adding "python" to the toolchain file. We don't provide a good way to customise the toolchain file at the moment, so that isn't entirely straightforward. It's probably the best short term solution though.

You mean individual ebuilds don't have a convenient way to add settings to the eclass-created toolchain file? This is not a problem, the ebuild can just create a second toolchain file and specify it as an *additional* argument to meson setup. You can have multiple toolchain files, and they will compose together. See https://mesonbuild.com/Machine-files.html#loading-multiple-machine-files

[Comment 13 Sam James 2023-08-13 03:58:28 UTC]

*** Bug 912203 has been marked as a duplicate of this bug. ***

[Comment 14 Larry the Git Cow 2023-08-17 07:42:26 UTC]

The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0686381dfc3d39c0f844d25aeb8471b9a5de657e

commit 0686381dfc3d39c0f844d25aeb8471b9a5de657e
Author:     James Le Cuirot <[e-mail]>
AuthorDate: 2023-08-16 21:34:25 +0000
Commit:     James Le Cuirot <[e-mail]>
CommitDate: 2023-08-17 07:41:30 +0000

    dev-util/meson: Backport fix to respect PATH when finding Python
    
    Closes: https://bugs.gentoo.org/912051
    Signed-off-by: James Le Cuirot <[e-mail]>

 dev-util/meson/files/meson-1.2.1-python-path.patch | 26 ++++++++++++++++++++++
 .../{meson-1.2.1.ebuild => meson-1.2.1-r1.ebuild}  |  4 ++++
 2 files changed, 30 insertions(+)

[Comment 15 Larry the Git Cow 2023-10-06 20:08:37 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f5f361e355257f8098df5f56f7c43aed4b452831

commit f5f361e355257f8098df5f56f7c43aed4b452831
Author:     Sam James <[e-mail]>
AuthorDate: 2023-10-06 20:07:55 +0000
Commit:     Sam James <[e-mail]>
CommitDate: 2023-10-06 20:07:55 +0000

    sys-apps/portage: depend on >=meson-1.2.1-r1 for path fix
    
    Bug: https://bugs.gentoo.org/912051
    Signed-off-by: Sam James <[e-mail]>

 sys-apps/portage/portage-3.0.50.ebuild | 3 +++
 sys-apps/portage/portage-3.0.51.ebuild | 3 +++
 sys-apps/portage/portage-3.0.52.ebuild | 3 +++
 sys-apps/portage/portage-9999.ebuild   | 3 +++
 4 files changed, 12 insertions(+)