I'll start off this bug report by saying this is a bit of a doozy, and it's the result of what I believe to be a more widespread issue regarding the current implementation of EPREFIX within cross-compilation contexts. Let's start with how to reproduce: PREFIX="$HOME/prefix" wget https://gitweb.gentoo.org/repo/proj/prefix.git/plain/scripts/bootstrap-prefix.sh chmod +x bootstrap-prefix.sh ./bootstrap-prefix.sh "$PREFIX" noninteractive "$PREFIX/startprefix" emerge crossdev crossdev -t x86_64-pc-linux-musl USE=-su x86_64-pc-linux-musl-emerge systemd-tmpfiles It should be noted, this was executed in an Ubuntu 20.04 environment, which has an installed python of version 3.8, this will be relevant in a second. What went wrong? meson.build:624:8: ERROR: Problem encountered: python3 jinja2 missing Jinja is insalled within the host gentoo prefix, so let's see what meson-log.txt has to say: /home/u/gentoo/usr/x86_64-pc-linux-musl/tmp/portage/sys-apps/systemd-tmpfiles-249.9/temp/python3.9/bin/python3: 2: exec: /usr/bin/python3.9: not found And for clarity, these are the contents of temp/python3.9/bin/python3: #!/bin/sh exec "/usr/bin/python3.9" "${@}" This file is created using the contents of the ${PYTHON} variable within _python_wrapper_setup, and should point to the host version of python, ensured to be installed through the BDEPENDS variable in an ebuild. The ${PYTHON} variable is derived from ${EPREFIX}/usr/bin/${impl} in _python_export. However, because x86_64-pc-linux-musl-emerge resets the EPREFIX variable, instead of the expected behavior where the host implementation of python used (that is, that which is installed within the host prefix), an incompatible path is formed by combining the host's rootfs with the target EPREFIX. I'm not sure what the best way to solve this issue is. Many ebuilds work fine with this setup, as they rarely rely on EPREFIX to find host tools, or the host tools were required to build the prefix in the first place There's currently no known way to work around this, aside from editing the eclass directly.
It's worth noting that the bootstrap-prefix.sh script hits a similar issue when it doesn't have python yet in the target prefix. This is currently solved with a symlink: https://gitweb.gentoo.org/repo/proj/prefix.git/tree/scripts/bootstrap-prefix.sh#n2118 I'm not sure what the difference between EPREFIX and PORTAGE_OVERRIDE_EPREFIX is, now I'm noticing the variable...
Not read your report in detail (yet?) but I'll note that we have a fork of python-utils-r1.eclass in the prefix overlay and while I've carried forward the changes we have there, I don't understand right now why we have them.
I see, I can tell from the code that this issue was partially addressed within the modified eclass: https://gitweb.gentoo.org/repo/proj/prefix.git/tree/eclass/python-utils-r1.eclass#n335 It seems to fall back to checking within BROOT if the python implementation isn't found in EPREFIX. This is at least in the right direction, but it only being a fallback wouldn't fix OP's situation (jinja may not be installed in the non-gentoo host system). Going by what ebuild(5) says: BROOT = "${EPREFIX}" Beginning with EAPI 7, the absolute path to the root directory containing build dependencies satisfied by BDEPEND, typically executable build tools. This includes any applicable offset pre- fix. Do not modify this variable. BROOT seems the right variable to prefix the PYTHON path with, instead of straight EPREFIX. This would make sense with the way BROOT is being set in /usr/bin/cross-emerge. That is, unless the PYTHON variable set by _python_export is used to store paths that are used in runtime (as opposed to buildtime), in which case more work would be needed... I just can't tell where in portage this variable is being initialized as described in ebuild(5), but that's a bit more minor. Another thing I've tried looking into is the interaction between PORTAGE_OVERRIDE_EPREFIX and EPREFIX, since the latter will (always?) be set to the former, yet the latter may also be picked up from the environment... It doesn't seem too useful within this scenario but it confuses me.
Interestingly, (and annoyingly imo), distutils-r1.eclass' esetup.py does call ${EPYTHON} directly from the PATH, instead of hardcoding the location like python-utils-r1.eclass does. This means that in our situation, packages that use distutils-r1 will install their site-packages files to the EPREFIX chosen at build-time for the host (thus ending up at $BROOT/usr/$CHOST/$BROOT, thanks to ROOT=$BROOT/usr/$CHOST). Thankfully, setup.py scripts have a --prefix option to override the hardcoded prefix and use the current EPREFIX variable. I think this should be passed from within distutils-r1.eclass, like is being done within econf and similar. Due to the lack of an EXTRA_ESETUPPY equivalent to EXTRA_ECONF, this can't be worked around without modifying the ebuild directly either.
Unfortunately, using setup.py is deprecated and we're still figuring out how to deal with PEP 517. However, we're probably going to be overriding prefix anyway, so maybe that'll be a good place to start. You can try playing with "installer", particularly with my today's PR adding --prefix option: https://github.com/pradyunsg/installer/pull/103
Interesting. I guess this sub-issue with distutils-r1.eclass will have to wait until PEP517/518/pyproject.toml and all these recent developments are a bit more mature. I have no idea how any of that works as most of my python knowledge is quite antiquated by today's standards...
Closing this bug as old. My original issue has been solved by _python_export using ${BROOT} instead of ${EPREFIX} whenever available. There's still an issue with regards to installed shebangs with esetup.py, I'm currently using `esetup.py build --executable="${EPREFIX}/usr/bin/${EPYTHON}"` to ensure the right shebangs are used when scripts are installed in /usr/lib/python-exec, and `esetup.py install --prefix="${EPREFIX}/usr"` to ensure that files get installed in ${EPREFIX}. I need both of these flags for portage to be emerged correctly into the target EPREFIX, but I'm not sure how well this would play with other setup.py-based ebuilds, so I might open a different issue about this some other time.
