Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 465546 - dev-python/pypy: install layout likely wrong (was: distutils installs data files to its libdir rather than /usr/share)
Summary: dev-python/pypy: install layout likely wrong (was: distutils installs data fi...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal major (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-04-11 15:51 UTC by Michał Górny
Modified: 2020-11-18 09:03 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-04-11 15:51:36 UTC
I've run into this with java-config:

    data_files = [
        ('share/java-config-2/launcher', ['src/launcher.bash']),
        ('share/man/man1/', ['man/java-config-2.1']),

CPython installs the files into /usr/share/...

PyPy installs them into /usr/lib64/pypyX.Y/share/...

We should probably either fix this quickly or review the packages which have pypy support enabled and install datafiles, and drop pypy support in them.
Comment 1 Mike Gilbert gentoo-dev 2013-04-11 21:59:46 UTC
Yeah, pypy has that funky definition of sys.prefix. I wonder if there is a way we can correct that in pypy itself.
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-04-19 14:52:37 UTC
This also breaks automake:

>>>> sysconfig.get_path('purelib')
'/usr/lib64/pypy2.0/lib-python'
>>>> sysconfig.get_path('purelib', vars={'base':'/usr'})
'/usr/lib-python'

Since automakes tries to override sys.prefix using that 'base' arg, it gets the wrong dir.

I think we should just use sys.prefix=/usr, and override all kinds of paths to pypy subdirs.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-05-01 07:58:32 UTC
So how about overriding all the install paths in pypy and using /usr as install prefix instead? I'd work on that but -- as you already know -- my system doesn't meet pypy's build requirements ;).
Comment 4 Ian Delaney (RETIRED) gentoo-dev 2013-05-06 09:21:48 UTC
This is kind of hairy & scary.  However

Python 2.7.3 (4b60269153b5b10c69155b6270bcce7a31a86281, May 05 2013, 12:34:18)
[PyPy 2.0.0-beta2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``Is there a pypy time? - if you
can feel it (?) then there is''
>>>> import sysconfig
>>>> sysconfig.get_path('purelib', vars={'base':'/usr'})
'/usr/lib-python/2.7'
>>>> sysconfig.get_path('purelib', vars={'purelib':'/usr'})
'/usr/lib64/pypy2.0/lib-python/2.7'
>>>> sysconfig.get_path('stdlib')
'/usr/lib64/pypy2.0/lib-python/2.7'
>>>> sysconfig.get_path('include')
'/usr/include'
>>>> sysconfig.get_path('platinclude')
'/usr/lib64/pypy2.0/include'
>>>> sysconfig.get_path('data')
'/usr/share'

archtester pypy # ls /usr/lib64/pypy2.0/
include  lib_pypy  lib-python  pypy-c  restkit  share  site-packages  socketpool
archtester pypy # ls -ld /usr/share/pypy2.0/
drwxr-xr-x 2 testuser testuser 4096 May  6 16:11 /usr/share/pypy2.0/

Changing of sysconfig paths is piece of proverbial cake

_INSTALL_SCHEMES = {

2 examples;
    'posix_home': {
        'stdlib': '{base}/lib/python',
        'platstdlib': '{base}/lib/python',
        'purelib': '{base}/lib/python',
        'platlib': '{base}/lib/python',
        'include': '{base}/include/python',
        'platinclude': '{base}/include/python',
        'scripts': '{base}/bin',
        'data'   : '{base}',
        },
    'pypy': {
        'stdlib': '{base}/lib-python/{py_version_short}',
        'platstdlib': '{base}/lib-python/{py_version_short}',
        'purelib': '{base}/lib-python/{py_version_short}',
        'platlib': '{base}/lib-python/{py_version_short}',
        'include': '{base}/include',
        'platinclude': '{base}/include',
        'scripts': '{base}/bin',
        'data'   : '{base}',
        },

The edits that yields the paths in the pypy shell above are;

    'pypy': {
        'stdlib': '{base}/lib-python/{py_version_short}',
        'platstdlib': '{base}/lib-python/{py_version_short}',
        'purelib': '{base}/lib-python/{py_version_short}',
        'platlib': '{base}/lib-python/{py_version_short}',
         '/usr/include',
        'platinclude': '{base}/include',
        'scripts': '{base}/bin',
        'data'   : '/usr/share',
        },

I was initially going to surplant {base} with usr/lib($ARCH) however it doesn't appear required. For mgorny and the automake there is

>>>> sysconfig.get_path('purelib', vars={'purelib':'/usr'})
'/usr/lib64/pypy2.0/lib-python/2.7'

and I'm still not clear whether it should be 
/usr/lib64/pypy2.0/lib-python/2.7 OR /usr/lib64/pypy2.0/lib-python

however just make it what it need be in 

'purelib': '{base}/ .......

I'm presuming the sysconfig.get_path('purelib', vars={'base':'/usr'})
comes from the automake which can be patched by taking out the 
, vars={'purelib':'/usr'} or re-setting it to vars={'purelib':'/usr'}

I haven't yet found where {base} is spawned and set and neither has anyone else but I don't think we need to.

As for shifting distutils module or sysconfig itself, well let's get a consensus.  Shifting things around in /image is fine but it has to match pypy's internal dir maps'.
Ought 'platinclude': == 'include': ?
Ought 'data'   : == 'usr/share/pypy2.0"
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-08-01 12:03:17 UTC
Note: pypy/module/sys/initpath.py will need to be modified to count the prefix differently.
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-08-01 12:14:05 UTC
Files that definitely need to be hacked: lib-python/2.7/distutils/sysconfig_pypy.py lib-python/2.7/sysconfig.py pypy/interpreter/app_main.py pypy/module/sys/initpath.py pypy/sandbox/pypy_interact.py pypy/tool/lib_pypy.py

We can possibly omit sandbox as it doesn't work anyway :P.
Comment 7 Ian Delaney (RETIRED) gentoo-dev 2013-08-01 12:19:11 UTC
ooh bravo, finally
Comment 8 Ian Delaney (RETIRED) gentoo-dev 2014-07-03 15:18:28 UTC
(In reply to Michał Górny from comment #5)
> Note: pypy/module/sys/initpath.py will need to be modified to count the
> prefix differently.

Are you sure? Really sure of yourself?

~/cvsPortage/gentoo-x86/dev-java/java-config $ ebuild java-config-2.2.0.ebuild clean install

running install_egg_info
Writing /mnt/gen2/TmpDir/portage/dev-java/java-config-2.2.0/image//_pypy/usr/lib64/pypy/site-packages/java_config-2.2.0-py2.7.egg-info
 * python2_7: running distutils-r1_run_phase distutils-r1_python_install_all
>>> Completed installing java-config-2.2.0 into /mnt/gen2/TmpDir/portage/dev-java/java-config-2.2.0/image/

ecompressdir: bzip2 -9 /usr/share/doc
ecompressdir: bzip2 -9 /usr/share/man

Only 2 lines in 2 files were edited to achieve this outcome.

1. /usr/lib64/pypy/lib-python/2.7/sysconfig.py
line 37 edited from
        'data'   : '{base}',
to 
        'data'   : '/usr',

2. /usr/lib64/pypy/lib-python/2.7/distutils/command/install.py
line 91 edited with exactly the same change.

The install.py required this rather verbose sed statement;
        sed -e "s/'data'   : '\$base'/'data'   : '\/usr'/" \
                -i lib-python/2.7/distutils/command/install.py \
                && einfo "install.py edited" || die 

Attempts to make a patch of it in the usual way failed in a way I have not seen before nor understand, however that's not the issue here.  
The pypy/lib-python/2.7/sysconfig.py just took a regular patch, though It seems the sed statement could be used on both.

Whether the '{base}' could be substituted with another variable assigned to /usr or "${ROOT}"usr or such afaiac is purely a style issue with which I have little concern.  These two edits work.

Whether this makes for a comprehensive fix is undecided and unclear.  It does make for a step forward.
Comment 9 Mike Gilbert gentoo-dev 2014-07-03 15:37:59 UTC
Some information from a recent IRC exchange:

<mgorny> if you override the dirs into which distutils install
<mgorny> it will try to install into your directories instead of ~ when called by user
<mgorny> you need to change sys.prefix to match it
<mgorny> otherwise it will install to /usr/share
<mgorny> and some programs will look for resources in /usr/lib64/pypy2.0/usr/share

<mgorny> do whatever you want but make sure that 'pip install --user' and virtualenv both work
<mgorny> if you break either, the patch is a no-go


In summary: We need to account for installation of python packages by the user into their home directory using the various methods available. If I interpret mgorny's comments correctly, Ian's patch from comment 8 will break this.
Comment 10 Ian Delaney (RETIRED) gentoo-dev 2014-07-04 00:05:54 UTC
In summary: We need to account for installation of python packages by the user into their home directory using the various methods available. If I interpret mgorny's comments correctly, Ian's patch from comment 8 will break this.

fair comment.  This is why I never declared it a final solution, but rather a single step.  We, or he, has known what is wrong now for 14 months at this point in time and still no-one has done anything about a fix.  Well, how about it....!!
Comment 11 Mike Gilbert gentoo-dev 2014-07-04 00:29:58 UTC
(In reply to Ian Delaney from comment #10)

Your comment is not helpful in the least. Please keep this technical.
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-07-04 05:58:11 UTC
(In reply to Ian Delaney from comment #10)
> fair comment.  This is why I never declared it a final solution, but rather
> a single step.  We, or he, has known what is wrong now for 14 months at this
> point in time and still no-one has done anything about a fix.  Well, how
> about it....!!

I don't know if you know it but we are volunteers and we do what time permits us to do. We don't get paid nor given hardware. So I suggest you start by funding me hardware capable of building PyPy, preferably fast considering the amount of work needed, and then give me a proper payment for doing work *you want being done*.

My comments were only supposed to help you or anyone that attempts to solve the real problem. If you can't make use of it, that's your problem. But don't blame us if you're running in circles and wasting your time doing something that certainly won't work.
Comment 13 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-11-27 19:55:05 UTC
I have committed new revisions of PyPy with a better path hack that would hopefully solve this. Long story short, now Gentoo hackery is only applied if package would normally be installed to /usr/lib*/pypy. But we apply it to install_data path too, so hopefully all data subdirs will land in /usr as expected.

Should be fixed in 2.6.0-r1 (2.6.0-r2 in pypy-bin) or 4.0.1. I still need to port it to PyPy3 though. Probably with a patch for new ncurses.
Comment 14 Tom Gillespie 2019-10-24 16:01:33 UTC
Has the pypy3 fix for this been implemented?

While working on an ipython-7.8.0 ebuild (linked below) with pypy3 support
I encountered the following error which directed me to this bug. I think it
is caused by the copying which was logged as well (above the error but
listed below the error here).

 * ERROR: dev-python/ipython-7.8.0::tgbugs-overlay failed (install phase):
 *   Package installs 'share' in PyPy prefix, see bug #465546.
 * 

copying docs/man/ipython.1 -> /var/tmp/portage/dev-python/ipython-7.8.0/image/_pypy3/usr/lib/pypy3.6/share/man/man1


https://github.com/tgbugs/tgbugs-overlay/blob/master/dev-python/ipython/ipython-7.8.0.ebuild
Comment 15 Larry the Git Cow gentoo-dev 2019-11-01 08:07:16 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bc29c449012661931f5ed9716547a9d88a90b792

commit bc29c449012661931f5ed9716547a9d88a90b792
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2019-11-01 07:46:30 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2019-11-01 08:03:23 +0000

    dev-python/pypy3-bin: Fix Gentoo path patch
    
    Bug: https://bugs.gentoo.org/465546
    Signed-off-by: Michał Górny <mgorny@gentoo.org>

 dev-python/pypy3-bin/files/7.0.0-gentoo-path.patch                      | 2 +-
 .../pypy3-bin/{pypy3-bin-7.2.0.ebuild => pypy3-bin-7.2.0-r1.ebuild}     | 0
 2 files changed, 1 insertion(+), 1 deletion(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=75a8f49809d9c082d9bbd6403aa0d637f5d93eac

commit 75a8f49809d9c082d9bbd6403aa0d637f5d93eac
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2019-11-01 07:45:49 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2019-11-01 08:03:21 +0000

    dev-python/pypy3: Fix Gentoo path patch
    
    Bug: https://bugs.gentoo.org/465546
    Signed-off-by: Michał Górny <mgorny@gentoo.org>

 dev-python/pypy3/files/7.0.0-gentoo-path.patch                 | 2 +-
 dev-python/pypy3/{pypy3-7.2.0.ebuild => pypy3-7.2.0-r1.ebuild} | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
Comment 16 Tom Gillespie 2020-03-02 05:27:00 UTC
Not to dig this one up yet again, but I think there is an issue with the option set for headers. Specifically the line here https://github.com/gentoo/gentoo/blob/1da1f7fe7ebc3a02ada674c2a65900ed076ae0a6/dev-python/pypy3/files/7.0.0-gentoo-path.patch#L30.

When I enabled pypy3 support for pybind11 install fails due to a collision between pypy3's eval.h and pybind11's eval.h. pybind11 tries to do the right thing, but the install_dir provided for install_headers does not include the package name as it does for cpython. I don't know whether this a a pypy issue or perhaps a pybind11 issue (since pybind11 has a custom InstallHeaders class). It does seem that all other cases set 'headers': '$base/include/$dist_name' though (that might be a one line fix for the issue if it doesn't cause other issues).


Here are two lines from the build logs.

copying include/pybind11/eval.h -> /var/tmp/portage/dev-python/pybind11-2.4.3/image/_python3.6/usr/include/python3.6m/pybind11/

copying include/pybind11/eval.h -> /var/tmp/portage/dev-python/pybind11-2.4.3/image/_pypy3/usr/lib/pypy3.6/include/
Comment 17 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-03-02 05:33:24 UTC
Could you file a separate bug I could reference?