| Summary: | PyPy (New package) | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Forza <forza> |
| Component: | New packages | Assignee: | Default Assignee for New Packages <maintainer-wanted> |
| Status: | RESOLVED FIXED | ||
| Severity: | enhancement | CC: | djc, domen, filmor, hongqn, marek, mpagano, nirbheek, orzel, osadchiy.ilya, python, steffen, vomacko |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | http://morepypy.blogspot.com & http://codespeak.net/pypy | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
Initial, barebone ebuild
barebone ebuild with some additions Patch to disable interactive debugger Improved pypy ebuild Pypy ebuild with improved optional modules pypy ebuild pypy bin ebuild (draft) pypy ebuild pypy ebuild pypy-1.5-x86.patch : link dynamically to libffi pypy ebuild Patch to fix executable stack and disable debug of translation pypy ebuild with SLOT=$PV |
||
|
Description
Forza
2010-01-09 17:09:56 UTC
Created attachment 267499 [details]
Initial, barebone ebuild
Incomplete ebuild, stuff missing:
* full dependency list
* mercurial support (9999 ebuild)
* eselect python integration - optional default python implementation
Nice!! How much memory does this need to build? I've had some problems with that in the past, on a server box that has 2GB memory. Yup, It would be wise to add a warning. On my dual core 64bit machine it takes exactly 1 hour to compile. Memory usage: 1.2 GB on 32-bit, 2.4 GB on 64-bit. Created attachment 267683 [details]
barebone ebuild with some additions
Created attachment 267685 [details, diff]
Patch to disable interactive debugger
Thanks for the ebuild. I've added some use-flagged dependencies (from http://codespeak.net/pypy/dist/pypy/doc/getting-started-python.html#translating-the-pypy-python-interpreter), check for memory requirements, added SLOT="0", and fixed permissions of pypy-c: http://bugs.gentoo.org/attachment.cgi?id=267683 I also created a patch for translate.py to disable interactive debugger in case of failure: http://bugs.gentoo.org/attachment.cgi?id=267685 Things to be done: - Disable linking against optional libraries if they are present but disabled in use flags (I think --withoutmod-* option does it) - Somehow allow installing modules (site-packages) to pypy - Can pypy be translated with older version of pypy? If not, then it should depend on dev-lang/python - virtual/python is deprecated. Nothing should depend on it or provide it. - $(get_libdir) should be used. - The ebuild should properly use python.eclass. Created attachment 267767 [details]
Improved pypy ebuild
Changed:
- use $(get_libdir)
- removed usage of virtual/python
TODO:
- support tests
- support different backends (jvm, .net, ...)
Also, I had no luck using python.eclass correctly. If user builds Python modules for one version of PyPy (e.g. 1.4), then will they work with a different version (e.g. 1.6 or 2.0) without rebuilding? If not, then a versioned directory should be used (e.g. /usr/$(get_libdir)/pypy1.4). PyPy implements Python versions. Since they are not 100% compatible we should separate versions indeed. Created attachment 268013 [details]
Pypy ebuild with improved optional modules
Changed:
- Disable linking against optional libraries if they are present but disabled
in use flags
Created attachment 268723 [details]
pypy ebuild
Small change:
- Use slotted installation directory
Tried to understand how python.eclass should be used, but it seems more useful for python modules (dev-lang/python uses it, dev-java/jython doesn't). It looks though that in future python.eclass should be augmented to include pypy.
Also I started thinking about pypy-bin package, as installation from source requires lots of memory and time. I've downloaded precompiled pypy to see if it works and see that it fails to find libbz2 library. It looks for libbz2.so.1.0 and I have "/lib64/libbz2.so.1.0.6" and "/lib64/libbz2.so.1 -> libbz2.so.1.0.6". Manually creating "/lib64/libbz2.so.1.0 -> libbz2.so.1.0.6" makes it work. Now how should it be done in a beautiful way? Some flag to ldconfig? Created attachment 270347 [details] pypy bin ebuild (draft) Here is a first version of ebuild for precompiled pypy. The libbz2 problem that I referred earlier was already solved, see bug 338321. I made pypy-bin install into /usr/lib64/pypy-bin2.5 and it doesn't require regular pypy not being present; but perhaps it's wrong, I don't know. I'm against adding pypy-bin package. It would needlessly complicate code in python.eclass etc. pypy ebuild shouldn't install pypy or pypy-c symlink in /usr/bin to avoid file collisions. PYTHON_ABI="2.5" is only for CPython 2.5. PyPy 1.4 should use probably PYTHON_ABI="2.5-pypy-1.4" (and SLOT="1.4"). pypy ebuild should ensure that building of PyPy will succeed when main active version of Python is set to CPython 3.*. Created attachment 270421 [details]
pypy ebuild
Fixing SLOT and PYTHON_ABI
(In reply to comment #16) > I'm against adding pypy-bin package. It would needlessly complicate code in > python.eclass etc. Is there still a problem if they are installed to same directory (/usr/lib64/pypy1.4/)? > pypy ebuild should ensure that building of PyPy will succeed when main active > version of Python is set to CPython 3.*. I think it doesn't work. What's the right way to solve it? Use "python2" in translate_cmd? Or it is the place where python.eclass can help us? (In reply to comment #18) > (In reply to comment #16) > > pypy ebuild should ensure that building of PyPy will succeed when main active > > version of Python is set to CPython 3.*. > I think it doesn't work. What's the right way to solve it? Use "python2" in > translate_cmd? You can use "python2" or "$(PYTHON -2)". Created attachment 270431 [details]
pypy ebuild
Using $(PYTHON -2).
I get these (from building 1.5 with the ebuild supplied in comment 20): * QA Notice: The following files contain writable and executable sections * Files with such sections will not work properly (or at all!) on some * architectures/operating systems. A bug should be filed at * http://bugs.gentoo.org/ to make sure the issue is fixed. * For more information, see http://hardened.gentoo.org/gnu-stack.xml * Please include the following list of files in your report: * Note: Bugs should be filed for the respective maintainers * of the package in question and not hardened@g.o. * RWX --- --- usr/lib64/pypy1.4/pypy-c * QA Notice: Package has poor programming practices which may compile * fine but exhibit random runtime failures. * [platform:WARNING] implement_30.c:11563: warning: implicit declaration of function 'pypysig_reinstall' * [platform:WARNING] implement_37.c:14641: warning: implicit declaration of function 'init_pycobject' * [platform:WARNING] implement_37.c:14733: warning: implicit declaration of function 'init_capsule' * [platform:WARNING] implement_56.c:35202: warning: implicit declaration of function 'RAND_status' * [platform:WARNING] implement_57.c:7730: warning: implicit declaration of function 'wcscoll' * [platform:WARNING] implement_60.c:48616: warning: implicit declaration of function 'pypy_macro_wrapper_FD_ZERO' * [platform:WARNING] implement_60.c:48666: warning: implicit declaration of function 'pypy_macro_wrapper_FD_SET' * [platform:WARNING] implement_61.c:5324: warning: implicit declaration of function 'forkpty' * [platform:WARNING] implement_61.c:6113: warning: implicit declaration of function 'openpty' * [platform:WARNING] implement_63.c:39604: warning: implicit declaration of function 'RAND_egd' * [platform:WARNING] implement_63.c:41338: warning: implicit declaration of function 'pypy_macro_wrapper_FD_ISSET' * [platform:WARNING] implement_72.c:25319: warning: implicit declaration of function 'RAND_add' * [platform:WARNING] debug_print.c:86: warning: implicit declaration of function 'gettimeofday' Also, can we do something more clever with the SLOT? I.e. use $PV and versionator, such that renaming the ebuild will still do the right thing. Testing pypy-1.5.ebuil
Hey if I have no libffi.a in /usr/lib, I get this error:
[Timer] Total: --- 2103.8 s
[translation:ERROR] Error:
[translation:ERROR] Traceback (most recent call last):
[translation:ERROR] File "./pypy/translator/goal/translate.py", line 304, in main
[translation:ERROR] drv.proceed(goals)
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/driver.py", line 809, in proceed
[translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip())
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/tool/taskengine.py", line 116, in _execute
[translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/driver.py", line 286, in _do
[translation:ERROR] res = func()
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/driver.py", line 395, in task_pyjitpl_lltype
[translation:ERROR] from pypy.jit.metainterp.warmspot import apply_jit
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/jit/metainterp/warmspot.py", line 28, in <module>
[translation:ERROR] from pypy.jit.metainterp.optimizeopt import ALL_OPTS_NAMES
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/jit/metainterp/optimizeopt/__init__.py", line 8, in <module>
[translation:ERROR] from pypy.jit.metainterp.optimizeopt.fficall import OptFfiCall
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/jit/metainterp/optimizeopt/fficall.py", line 3, in <module>
[translation:ERROR] from pypy.rlib.libffi import Func
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/rlib/libffi.py", line 5, in <module>
[translation:ERROR] from pypy.rlib import clibffi
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/rlib/clibffi.py", line 75, in <module>
[translation:ERROR] link_files = [find_libffi_a()]
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/rlib/clibffi.py", line 70, in find_libffi_a
[translation:ERROR] raise ImportError("'libffi.a' not found in %s" % (dirlist,))
[translation:ERROR] ImportError: 'libffi.a' not found in ['/usr/lib']
* ERROR: dev-python/pypy-1.5 failed (compile phase):
* compile error
*
* Call stack:
* ebuild.sh, line 56: Called src_compile
* environment, line 4581: Called die
* The specific snippet of code:
* ${translate_cmd} || die "compile error"
*
* If you need support, post the output of 'emerge --info =dev-python/pypy-1.5',
* the complete build log and the output of 'emerge -pqv =dev-python/pypy-1.5'.
* This ebuild is from an overlay named 'local_portage': '/usr/local/portage/'
* The complete build log is located at '/dev/shm/portage/dev-python/pypy-1.5/temp/build.log'.
* The ebuild environment file is located at '/dev/shm/portage/dev-python/pypy-1.5/temp/environment'.
* S: '/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src'
I had to install it manually from compiled libffi-3.0.9 tarball and now it works. Libffi.a is not installed by portage versions of libffi (3.0.9 nor 3.0.9-r2)
(In reply to comment #22) As a workaround, dev-libs/libffi[static-libs] could be in dependencies. The optimal solution would be to modify PyPy to use shared libffi library (libffi.so). (In reply to comment #23) > (In reply to comment #22) > > As a workaround, dev-libs/libffi[static-libs] could be in dependencies. > The optimal solution would be to modify PyPy to use shared libffi library > (libffi.so). Strange, I have dev-libs/libffi-3.0.9-r2 without [static-libs] and I don't see libffi.a anywhere, but I have compilation running to the end, producing working binary. `lld` shows that the binary depends on libffi.so.5. From python scripts I see that static/dynamic linkage depends on "platform". It seems taht 32bit linux has static linkage and 64bit linux has dynamic one. Vit, do you have 32 or 64 bits? /////////////// From rlib/clibffi.py: # On some platforms, we try to link statically libffi, which is small # anyway and avoids endless troubles for installing. On other platforms # libffi.a is typically not there, so we link dynamically. includes = ['ffi.h'] def find_libffi_a(): dirlist = platform.library_dirs_for_libffi_a() for dir in dirlist: result = os.path.join(dir, 'libffi.a') if os.path.exists(result): return result raise ImportError("'libffi.a' not found in %s" % (dirlist,)) if hasattr(platform, 'library_dirs_for_libffi_a'): # platforms on which we want static linking libraries = [] link_files = [find_libffi_a()] else: # platforms on which we want dynamic linking libraries = ['ffi'] link_files = [] /////////////// From translator/platform/linux.py: class Linux(BaseLinux): shared_only = () # it seems that on 32-bit linux, compiling with -fPIC # gives assembler that asmgcc is not happy about. def library_dirs_for_libffi_a(self): # places where we need to look for libffi.a return self.library_dirs_for_libffi() + ['/usr/lib'] class Linux64(BaseLinux): pass /////////////// (In reply to comment #24) > (In reply to comment #23) > > (In reply to comment #22) > > > > As a workaround, dev-libs/libffi[static-libs] could be in dependencies. > > The optimal solution would be to modify PyPy to use shared libffi library > > (libffi.so). > > Strange, I have dev-libs/libffi-3.0.9-r2 without [static-libs] and I don't see > libffi.a anywhere, but I have compilation running to the end, producing working > binary. `lld` shows that the binary depends on libffi.so.5. > From python scripts I see that static/dynamic linkage depends on "platform". It > seems taht 32bit linux has static linkage and 64bit linux has dynamic one. > > Vit, do you have 32 or 64 bits? > My system is 32 bit (In reply to comment #25) > > seems taht 32bit linux has static linkage and 64bit linux has dynamic one. > > > > Vit, do you have 32 or 64 bits? > > > My system is 32 bit I have 64 bit, so I can't check, can you please see if it works after removing these 3 lines from translator/platform/linux.py: def library_dirs_for_libffi_a(self): # places where we need to look for libffi.a return self.library_dirs_for_libffi() + ['/usr/lib'] (In reply to comment #26) > (In reply to comment #25) > I have 64 bit, so I can't check, can you please see if it works after removing > these 3 lines from translator/platform/linux.py: > def library_dirs_for_libffi_a(self): > # places where we need to look for libffi.a > return self.library_dirs_for_libffi() + ['/usr/lib'] Yes it works now. Created attachment 272135 [details, diff]
pypy-1.5-x86.patch : link dynamically to libffi
Here is a patch to link libffi dynamically on x86. Should fit 1.5, don't know what's the case with 1.4.1.
Created attachment 272137 [details] pypy ebuild Using the above patch (attachment #272135 [details, diff]) pypy_ebuild (version 1.5) has bad slot "1.4", so pypy-1.5 is installed in /usr/lib/pypy1.4/. This should be fixed in ebuild. Created attachment 274797 [details, diff]
Patch to fix executable stack and disable debug of translation
Hi. This patch fixes "executable stack" problem (reported by Dirkjan), and also one of the "implicit function declaration". Don't know if it's worth effort to fix the other ones.
[Like in previous version, this patch also disables interactive degugger that would start if translation stage fails during installation]
Created attachment 274799 [details]
pypy ebuild with SLOT=$PV
As suggested by Dirkjan and Vit, I've made SLOT to change with versions. I'm not sure if it's the right thing to do ,I definitely don't want every update to be installed separately. I think we first need to understand how additional packages can be installed, to see if they need to be rebuild on version change etc. Then we will see which slots are needed.
(In reply to comment #32) > As suggested by Dirkjan and Vit, I've made SLOT to change with versions. I'm > not sure if it's the right thing to do ,I definitely don't want every update to > be installed separately. I think we first need to understand how additional > packages can be installed, to see if they need to be rebuild on version change > etc. Python modules installed for PyPy need to be rebuilt between PyPy 1.5 and 1.6 (due to difference in e.g. imp.get_magic()), but not between PyPy 1.5 and 1.5.1. PV="1.5" => SLOT="1.5" PV="1.5.1" => SLOT="1.5" PV="1.6.2" => SLOT="1.6" I've just committed something like this to the tree as dev-python/pypy-1.5. Please have a look and file followup bugs as necessary. Are we missing libffi patch in pypy ebuild in the tree (32bit system)?
[translation:ERROR] Error:
[translation:ERROR] Traceback (most recent call last):
[translation:ERROR] File "./pypy/translator/goal/translate.py", line 304, in main
[translation:ERROR] drv.proceed(goals)
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/driver.py", line 809, in proceed
[translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip())
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/tool/taskengine.py", line 116, in _execute
[translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/driver.py", line 286, in _do
[translation:ERROR] res = func()
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/translator/driver.py", line 395, in task_pyjitpl_lltype
[translation:ERROR] from pypy.jit.metainterp.warmspot import apply_jit
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/jit/metainterp/warmspot.py", line 28, in <module>
[translation:ERROR] from pypy.jit.metainterp.optimizeopt import ALL_OPTS_NAMES
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/jit/metainterp/optimizeopt/__init__.py", line 8, in <module>
[translation:ERROR] from pypy.jit.metainterp.optimizeopt.fficall import OptFfiCall
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/jit/metainterp/optimizeopt/fficall.py", line 3, in <module>
[translation:ERROR] from pypy.rlib.libffi import Func
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/rlib/libffi.py", line 5, in <module>
[translation:ERROR] from pypy.rlib import clibffi
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/rlib/clibffi.py", line 75, in <module>
[translation:ERROR] link_files = [find_libffi_a()]
[translation:ERROR] File "/dev/shm/portage/dev-python/pypy-1.5/work/pypy-1.5-src/pypy/rlib/clibffi.py", line 70, in find_libffi_a
[translation:ERROR] raise ImportError("'libffi.a' not found in %s" % (dirlist,))
[translation:ERROR] ImportError: 'libffi.a' not found in ['/usr/lib']
The ebuild I put in the tree has amd64 as the only keyword, so yeah, it probably doesn't work for x86. If you can give pointers on how to fix that, let me know. I don't have any boxes I can test it on... You guys rock! :) (In reply to comment #36) > The ebuild I put in the tree has amd64 as the only keyword, so yeah, it > probably doesn't work for x86. If you can give pointers on how to fix that, let > me know. I don't have any boxes I can test it on... There is already patch http://bugs.gentoo.org/attachment.cgi?id=272135 Very excited to see this ebuild land. What is the package's pypy binary, and is it supposed to be on the path? On ~amd64, I have to run it with:
$ /usr/lib64/pypy1.5/pypy-c
Python 2.7.1 (?, Jun 06 2011, 13:46:58)
[PyPy 1.5.0-alpha0 with GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``Not your usual analyses.''
>>>>
Thanks.
I wonder about the same thing, shouldn't pypy-c be symlinked to /usr/bin? Will there be eselect-python support in pypy in the near future? Would be neat to use pypy system wide... Users, who want to install Python modules for PyPy, should see bug #372701. |
