Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 716618 - sys-apps/portage: environmental PYTHONPATH settings like PYTHONPATH=. can interfere portage internals
Summary: sys-apps/portage: environmental PYTHONPATH settings like PYTHONPATH=. can int...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - External Interaction (show other bugs)
Hardware: All Linux
: Normal enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-04-07 20:21 UTC by Sebastian Pipping
Modified: 2021-11-26 18:18 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (build.log,6.07 KB, text/x-log)
2020-04-07 20:21 UTC, Sebastian Pipping
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sebastian Pipping gentoo-dev 2020-04-07 20:21:02 UTC
Created attachment 631218 [details]
build.log

There seems to be some weird python 2.7 3.6 mix-up happening here.  This is on a stable box.  Cannot reproduce locally easily.

 * python2_7: running distutils-r1_run_phase python_install_all
Traceback (most recent call last):
  File "/usr/lib/portage/python3.6/doins.py", line 17, in <module>
    import argparse
  File "/usr/lib64/python3.6/argparse.py", line 89, in <module>
    import re as _re
  File "/usr/lib64/python3.6/re.py", line 142, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

Any ideas what might be going on?
Comment 1 Sebastian Pipping gentoo-dev 2020-04-10 18:35:00 UTC
I had a chance to debug this further.  What I found:

 - Python 3.6 is eselected for a system-wide interpreter, as expected.

 - einstalldocs calls dodoc which calls doins (/usr/lib/portage/python3.6/ebuild-helpers/doins) which calls doins.py (/usr/lib/portage/python3.6/doins.py) with PYTHONPATH=/usr/lib64/python3.6/site-packages:. where . is /var/tmp/portage/dev-python/enum34-1.1.6-r1/work/enum34-1.1.6 .

 - As a result, doins.py ends up with this sys.path when started:
    1. ""
    2. "/var/tmp/portage/dev-python/enum34-1.1.6-r1/work/enum34-1.1.6-python2_7/lib"
    3. "/var/tmp/portage/dev-python/enum34-1.1.6-r1/work/enum34-1.1.6"
    4. "/usr/lib64/python36.zip"
    5. "/usr/lib64/python3.6"
    6. "/usr/lib64/python3.6/lib-dynload"
    7. "/usr/lib64/python3.6/site-packages"

 - As a result, for "import enum" file [3]/enum/__init__.py of enum34 without attribute IntFlag wins the the race over Python 3.6's own [5]/enum.py .
  
For a true fix, it seems either we need to

 a) get ":." out of ${PORTAGE_PYTHONPATH} or 

 b) get /usr/lib64/pythonX.Y into ${PORTAGE_PYTHONPATH} before ".".

If you agree with that assessment, it might be a bug in portage, both stable 2.3.89-r1 and unstable 2.3.98-r1 .  So I'll hand over to the portage team to get their opinion.

PS: I have not been able to find a workaround for enum34-1.1.6-r1.ebuild, yet.
Comment 2 Zac Medico gentoo-dev 2020-04-10 19:26:03 UTC
It looks like you have a PYTHONPATH=. setting in your environment which passes through to PORTAGE_PYTHONPATH. In the absence of an inherited PYTHONPATH setting, we should have just PORTAGE_PYTHONPATH=/usr/lib64/python3.6/site-packages here.
Comment 3 Zac Medico gentoo-dev 2020-04-10 19:28:00 UTC
In order to check the effective PORTAGE_PYTHONPATH setting, I temporarily patched the ebuild like this:

> --- a/dev-python/enum34/enum34-1.1.6-r1.ebuild
> +++ b/dev-python/enum34/enum34-1.1.6-r1.ebuild
> @@ -25,5 +25,6 @@ python_test() {
>  python_install_all() {
>         use doc && local DOCS=( enum/doc/. enum/README )
>  
> +       die "PORTAGE_PYTHONPATH=$PORTAGE_PYTHONPATH"
>         distutils-r1_python_install_all
>  }
Comment 4 Sebastian Pipping gentoo-dev 2020-04-10 20:00:20 UTC
I did something similar and my PORTAGE_PYTHONPATH is "/usr/lib64/python3.6/site-packages:." with ":." at that very place.
Comment 5 Sebastian Pipping gentoo-dev 2020-04-10 20:02:41 UTC
(In reply to Zac Medico from comment #2)
> It looks like you have a PYTHONPATH=. setting in your environment which
> passes through to PORTAGE_PYTHONPATH.

Let me double-check that, one moment
Comment 6 Sebastian Pipping gentoo-dev 2020-04-10 20:13:14 UTC
You're instinct was right: From where emerge was called, there was PYTHONPATH=. in the environment.  Once I dropped that, the build failure went away.

In theory, is this something portage has a chance to protect itself against?  I imagine something like checking the environment for PYTHONPATH, dropping it, and re-executing?

Should we close this as invalid or should we keep it open for portage?
Comment 7 Zac Medico gentoo-dev 2020-04-10 20:33:29 UTC
(In reply to Sebastian Pipping from comment #6)
> Should we close this as invalid or should we keep it open for portage?

The PYTHONPATH pass-through is a feature which is useful for testing portage from a git checkout, for example:

https://wiki.gentoo.org/wiki/Project:Portage#Testing_Portage

It can also be useful as a means to temporarily override arbitrary python libraries.

I suppose we could detect non-absolute paths in PYTHONPATH, display a warning message, and filter them out.
Comment 8 Zac Medico gentoo-dev 2020-04-10 21:00:38 UTC
Even if portage filters the PYTHONPATH=. value from PORTAGE_PYTHONPATH, it's not allowed to do the same for the regular PYTHONPATH variable in the ebuild environment, since that would deviate from PMS behavior. We can see that an environmental PYPTHONPATH=. does pass through to the ebuild environment when we patch the ebuild like this:

> --- a/dev-python/enum34/enum34-1.1.6-r1.ebuild
> +++ b/dev-python/enum34/enum34-1.1.6-r1.ebuild
> @@ -25,5 +25,7 @@ python_test() {
>  python_install_all() {
>         use doc && local DOCS=( enum/doc/. enum/README )
>  
> +       die "PYTHONPATH=$PYTHONPATH"
> +
>         distutils-r1_python_install_all
>  }