I've run into this with java-config: data_files = [ ('share/java-config-2/launcher', ['src/launcher.bash']), ('share/man/man1/', ['man/java-config-2.1']), CPython installs the files into /usr/share/... PyPy installs them into /usr/lib64/pypyX.Y/share/... We should probably either fix this quickly or review the packages which have pypy support enabled and install datafiles, and drop pypy support in them.
Yeah, pypy has that funky definition of sys.prefix. I wonder if there is a way we can correct that in pypy itself.
This also breaks automake: >>>> sysconfig.get_path('purelib') '/usr/lib64/pypy2.0/lib-python' >>>> sysconfig.get_path('purelib', vars={'base':'/usr'}) '/usr/lib-python' Since automakes tries to override sys.prefix using that 'base' arg, it gets the wrong dir. I think we should just use sys.prefix=/usr, and override all kinds of paths to pypy subdirs.
So how about overriding all the install paths in pypy and using /usr as install prefix instead? I'd work on that but -- as you already know -- my system doesn't meet pypy's build requirements ;).
This is kind of hairy & scary. However Python 2.7.3 (4b60269153b5b10c69155b6270bcce7a31a86281, May 05 2013, 12:34:18) [PyPy 2.0.0-beta2] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``Is there a pypy time? - if you can feel it (?) then there is'' >>>> import sysconfig >>>> sysconfig.get_path('purelib', vars={'base':'/usr'}) '/usr/lib-python/2.7' >>>> sysconfig.get_path('purelib', vars={'purelib':'/usr'}) '/usr/lib64/pypy2.0/lib-python/2.7' >>>> sysconfig.get_path('stdlib') '/usr/lib64/pypy2.0/lib-python/2.7' >>>> sysconfig.get_path('include') '/usr/include' >>>> sysconfig.get_path('platinclude') '/usr/lib64/pypy2.0/include' >>>> sysconfig.get_path('data') '/usr/share' archtester pypy # ls /usr/lib64/pypy2.0/ include lib_pypy lib-python pypy-c restkit share site-packages socketpool archtester pypy # ls -ld /usr/share/pypy2.0/ drwxr-xr-x 2 testuser testuser 4096 May 6 16:11 /usr/share/pypy2.0/ Changing of sysconfig paths is piece of proverbial cake _INSTALL_SCHEMES = { 2 examples; 'posix_home': { 'stdlib': '{base}/lib/python', 'platstdlib': '{base}/lib/python', 'purelib': '{base}/lib/python', 'platlib': '{base}/lib/python', 'include': '{base}/include/python', 'platinclude': '{base}/include/python', 'scripts': '{base}/bin', 'data' : '{base}', }, 'pypy': { 'stdlib': '{base}/lib-python/{py_version_short}', 'platstdlib': '{base}/lib-python/{py_version_short}', 'purelib': '{base}/lib-python/{py_version_short}', 'platlib': '{base}/lib-python/{py_version_short}', 'include': '{base}/include', 'platinclude': '{base}/include', 'scripts': '{base}/bin', 'data' : '{base}', }, The edits that yields the paths in the pypy shell above are; 'pypy': { 'stdlib': '{base}/lib-python/{py_version_short}', 'platstdlib': '{base}/lib-python/{py_version_short}', 'purelib': '{base}/lib-python/{py_version_short}', 'platlib': '{base}/lib-python/{py_version_short}', '/usr/include', 'platinclude': '{base}/include', 'scripts': '{base}/bin', 'data' : '/usr/share', }, I was initially going to surplant {base} with usr/lib($ARCH) however it doesn't appear required. For mgorny and the automake there is >>>> sysconfig.get_path('purelib', vars={'purelib':'/usr'}) '/usr/lib64/pypy2.0/lib-python/2.7' and I'm still not clear whether it should be /usr/lib64/pypy2.0/lib-python/2.7 OR /usr/lib64/pypy2.0/lib-python however just make it what it need be in 'purelib': '{base}/ ....... I'm presuming the sysconfig.get_path('purelib', vars={'base':'/usr'}) comes from the automake which can be patched by taking out the , vars={'purelib':'/usr'} or re-setting it to vars={'purelib':'/usr'} I haven't yet found where {base} is spawned and set and neither has anyone else but I don't think we need to. As for shifting distutils module or sysconfig itself, well let's get a consensus. Shifting things around in /image is fine but it has to match pypy's internal dir maps'. Ought 'platinclude': == 'include': ? Ought 'data' : == 'usr/share/pypy2.0"
Note: pypy/module/sys/initpath.py will need to be modified to count the prefix differently.
Files that definitely need to be hacked: lib-python/2.7/distutils/sysconfig_pypy.py lib-python/2.7/sysconfig.py pypy/interpreter/app_main.py pypy/module/sys/initpath.py pypy/sandbox/pypy_interact.py pypy/tool/lib_pypy.py We can possibly omit sandbox as it doesn't work anyway :P.
ooh bravo, finally
(In reply to Michał Górny from comment #5) > Note: pypy/module/sys/initpath.py will need to be modified to count the > prefix differently. Are you sure? Really sure of yourself? ~/cvsPortage/gentoo-x86/dev-java/java-config $ ebuild java-config-2.2.0.ebuild clean install running install_egg_info Writing /mnt/gen2/TmpDir/portage/dev-java/java-config-2.2.0/image//_pypy/usr/lib64/pypy/site-packages/java_config-2.2.0-py2.7.egg-info * python2_7: running distutils-r1_run_phase distutils-r1_python_install_all >>> Completed installing java-config-2.2.0 into /mnt/gen2/TmpDir/portage/dev-java/java-config-2.2.0/image/ ecompressdir: bzip2 -9 /usr/share/doc ecompressdir: bzip2 -9 /usr/share/man Only 2 lines in 2 files were edited to achieve this outcome. 1. /usr/lib64/pypy/lib-python/2.7/sysconfig.py line 37 edited from 'data' : '{base}', to 'data' : '/usr', 2. /usr/lib64/pypy/lib-python/2.7/distutils/command/install.py line 91 edited with exactly the same change. The install.py required this rather verbose sed statement; sed -e "s/'data' : '\$base'/'data' : '\/usr'/" \ -i lib-python/2.7/distutils/command/install.py \ && einfo "install.py edited" || die Attempts to make a patch of it in the usual way failed in a way I have not seen before nor understand, however that's not the issue here. The pypy/lib-python/2.7/sysconfig.py just took a regular patch, though It seems the sed statement could be used on both. Whether the '{base}' could be substituted with another variable assigned to /usr or "${ROOT}"usr or such afaiac is purely a style issue with which I have little concern. These two edits work. Whether this makes for a comprehensive fix is undecided and unclear. It does make for a step forward.
Some information from a recent IRC exchange: <mgorny> if you override the dirs into which distutils install <mgorny> it will try to install into your directories instead of ~ when called by user <mgorny> you need to change sys.prefix to match it <mgorny> otherwise it will install to /usr/share <mgorny> and some programs will look for resources in /usr/lib64/pypy2.0/usr/share <mgorny> do whatever you want but make sure that 'pip install --user' and virtualenv both work <mgorny> if you break either, the patch is a no-go In summary: We need to account for installation of python packages by the user into their home directory using the various methods available. If I interpret mgorny's comments correctly, Ian's patch from comment 8 will break this.
In summary: We need to account for installation of python packages by the user into their home directory using the various methods available. If I interpret mgorny's comments correctly, Ian's patch from comment 8 will break this. fair comment. This is why I never declared it a final solution, but rather a single step. We, or he, has known what is wrong now for 14 months at this point in time and still no-one has done anything about a fix. Well, how about it....!!
(In reply to Ian Delaney from comment #10) Your comment is not helpful in the least. Please keep this technical.
(In reply to Ian Delaney from comment #10) > fair comment. This is why I never declared it a final solution, but rather > a single step. We, or he, has known what is wrong now for 14 months at this > point in time and still no-one has done anything about a fix. Well, how > about it....!! I don't know if you know it but we are volunteers and we do what time permits us to do. We don't get paid nor given hardware. So I suggest you start by funding me hardware capable of building PyPy, preferably fast considering the amount of work needed, and then give me a proper payment for doing work *you want being done*. My comments were only supposed to help you or anyone that attempts to solve the real problem. If you can't make use of it, that's your problem. But don't blame us if you're running in circles and wasting your time doing something that certainly won't work.
I have committed new revisions of PyPy with a better path hack that would hopefully solve this. Long story short, now Gentoo hackery is only applied if package would normally be installed to /usr/lib*/pypy. But we apply it to install_data path too, so hopefully all data subdirs will land in /usr as expected. Should be fixed in 2.6.0-r1 (2.6.0-r2 in pypy-bin) or 4.0.1. I still need to port it to PyPy3 though. Probably with a patch for new ncurses.
Has the pypy3 fix for this been implemented? While working on an ipython-7.8.0 ebuild (linked below) with pypy3 support I encountered the following error which directed me to this bug. I think it is caused by the copying which was logged as well (above the error but listed below the error here). * ERROR: dev-python/ipython-7.8.0::tgbugs-overlay failed (install phase): * Package installs 'share' in PyPy prefix, see bug #465546. * copying docs/man/ipython.1 -> /var/tmp/portage/dev-python/ipython-7.8.0/image/_pypy3/usr/lib/pypy3.6/share/man/man1 https://github.com/tgbugs/tgbugs-overlay/blob/master/dev-python/ipython/ipython-7.8.0.ebuild
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bc29c449012661931f5ed9716547a9d88a90b792 commit bc29c449012661931f5ed9716547a9d88a90b792 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2019-11-01 07:46:30 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-11-01 08:03:23 +0000 dev-python/pypy3-bin: Fix Gentoo path patch Bug: https://bugs.gentoo.org/465546 Signed-off-by: Michał Górny <mgorny@gentoo.org> dev-python/pypy3-bin/files/7.0.0-gentoo-path.patch | 2 +- .../pypy3-bin/{pypy3-bin-7.2.0.ebuild => pypy3-bin-7.2.0-r1.ebuild} | 0 2 files changed, 1 insertion(+), 1 deletion(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=75a8f49809d9c082d9bbd6403aa0d637f5d93eac commit 75a8f49809d9c082d9bbd6403aa0d637f5d93eac Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2019-11-01 07:45:49 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-11-01 08:03:21 +0000 dev-python/pypy3: Fix Gentoo path patch Bug: https://bugs.gentoo.org/465546 Signed-off-by: Michał Górny <mgorny@gentoo.org> dev-python/pypy3/files/7.0.0-gentoo-path.patch | 2 +- dev-python/pypy3/{pypy3-7.2.0.ebuild => pypy3-7.2.0-r1.ebuild} | 0 2 files changed, 1 insertion(+), 1 deletion(-)
Not to dig this one up yet again, but I think there is an issue with the option set for headers. Specifically the line here https://github.com/gentoo/gentoo/blob/1da1f7fe7ebc3a02ada674c2a65900ed076ae0a6/dev-python/pypy3/files/7.0.0-gentoo-path.patch#L30. When I enabled pypy3 support for pybind11 install fails due to a collision between pypy3's eval.h and pybind11's eval.h. pybind11 tries to do the right thing, but the install_dir provided for install_headers does not include the package name as it does for cpython. I don't know whether this a a pypy issue or perhaps a pybind11 issue (since pybind11 has a custom InstallHeaders class). It does seem that all other cases set 'headers': '$base/include/$dist_name' though (that might be a one line fix for the issue if it doesn't cause other issues). Here are two lines from the build logs. copying include/pybind11/eval.h -> /var/tmp/portage/dev-python/pybind11-2.4.3/image/_python3.6/usr/include/python3.6m/pybind11/ copying include/pybind11/eval.h -> /var/tmp/portage/dev-python/pybind11-2.4.3/image/_pypy3/usr/lib/pypy3.6/include/
Could you file a separate bug I could reference?
