Created attachment 504260 [details] emerge --info Reproducibly, on two RAP systems, I cannot build dev-lang/ghc (on all my other systems, I can). The error message is as follows (it can be reproduced with =ghc-7.10.3 as well): ``` !!! Repository 'local' is missing masters attribute in '/home/hioanas/.local_overlay/metadata/layout.conf' !!! Set 'masters = gentoo' in this file for future compatibility * IMPORTANT: 1 news items need reading for repository 'gentoo'. * Use eselect news read to view new items. These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] dev-lang/ghc-8.0.2:0/8.0.2::gentoo USE="gmp profile -binary -doc -ghcbootstrap -ghcmakebinary" 0 KiB [ebuild N ] app-admin/haskell-updater-1.2.10::gentoo 0 KiB Total: 2 packages (2 new), Size of downloads: 0 KiB Would you like to merge these packages? [Yes/No] y >>> Verifying ebuild manifests >>> Running pre-merge checks for dev-lang/ghc-8.0.2 * Checking for at least 8 GiB disk space at "/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp" ... [ ok ] * Checking for at least 2 GiB disk space at "/home/hioanas/gentoo//usr" .. [ ok ] >>> Emerging (1 of 2) dev-lang/ghc-8.0.2::gentoo * ghc-8.0.2-src.tar.xz SHA256 SHA512 WHIRLPOOL size ;-) ... [ ok ] * ghc-bin-8.0.2-amd64.tbz2 SHA256 SHA512 WHIRLPOOL size ;-) ... [ ok ] * Checking for at least 8 GiB disk space at "/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp" ... [ ok ] * Checking for at least 2 GiB disk space at "/home/hioanas/gentoo//usr" .. [ ok ] >>> Unpacking source... >>> Unpacking ghc-8.0.2-src.tar.xz to /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work >>> Unpacking ghc-bin-8.0.2-amd64.tbz2 to /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work bzip2: /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/distdir/ghc-bin-8.0.2-amd64.tbz2: trailing garbage after EOF ignored >>> Source unpacked in /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work >>> Preparing source in /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/ghc-8.0.2 ... * PT_PAX marking -m /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/usr/lib64/ghc-8.0.2/bin/ghc with scanelf * XATTR_PAX marking -me /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/usr/lib64/ghc-8.0.2/bin/ghc with setfattr * Adjusting to prefix /home/hioanas/gentoo * settings ... [ ok ] /home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/usr/lib64/ghc-8.0.2/bin/ghc-pkg: relocation error: /home/hioanas/gentoo/lib64/libc.so.6: symbol __tunable_get_val, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference * ERROR: dev-lang/ghc-8.0.2::gentoo failed (prepare phase): * failed to update cache after relocation * * Call stack: * ebuild.sh, line 124: Called src_prepare * environment, line 3780: Called relocate_ghc '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work' * environment, line 3543: Called die * The specific snippet of code: * "$gp_back" recache || die "failed to update cache after relocation"; * * If you need support, post the output of `emerge --info '=dev-lang/ghc-8.0.2::gentoo'`, * the complete build log and the output of `emerge -pqv '=dev-lang/ghc-8.0.2::gentoo'`. * The complete build log is located at '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp/build.log'. * The ebuild environment file is located at '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp/environment'. * Working directory: '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/ghc-8.0.2' * S: '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/ghc-8.0.2' >>> Failed to emerge dev-lang/ghc-8.0.2, Log file: >>> '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp/build.log' *** Resuming merge... These are the packages that would be merged, in order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 KiB * emerge --keep-going: app-admin/haskell-updater-1.2.10 dropped because it * requires >=dev-lang/ghc-6.12.1 * Messages for package dev-lang/ghc-8.0.2: * ERROR: dev-lang/ghc-8.0.2::gentoo failed (prepare phase): * failed to update cache after relocation * * Call stack: * ebuild.sh, line 124: Called src_prepare * environment, line 3780: Called relocate_ghc '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work' * environment, line 3543: Called die * The specific snippet of code: * "$gp_back" recache || die "failed to update cache after relocation"; * * If you need support, post the output of `emerge --info '=dev-lang/ghc-8.0.2::gentoo'`, * the complete build log and the output of `emerge -pqv '=dev-lang/ghc-8.0.2::gentoo'`. * The complete build log is located at '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp/build.log'. * The ebuild environment file is located at '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp/environment'. * Working directory: '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/ghc-8.0.2' * S: '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/work/ghc-8.0.2' * Messages for package app-admin/haskell-updater-1.2.10: * emerge --keep-going: app-admin/haskell-updater-1.2.10 dropped because it * requires >=dev-lang/ghc-6.12.1 * * The following 2 packages have failed to build, install, or execute * postinst: * * (dev-lang/ghc-8.0.2:0/8.0.2::gentoo, ebuild scheduled for merge), Log file: * '/home/hioanas/gentoo/var/tmp/portage/dev-lang/ghc-8.0.2/temp/build.log' * (app-admin/haskell-updater-1.2.10:0/0::gentoo, ebuild scheduled for merge) * ```
I think it's caused by the following sed code: if use prefix; then # and insert LD_LIBRARY_PATH entry to EPREFIX dir tree # TODO: add the same for darwin's CHOST and it's DYLD_ local new_ldpath='LD_LIBRARY_PATH="'${EPREFIX}/$(get_libdir):${EPREFIX}/usr/$(get_libdir)'${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH}"\nexport LD_LIBRARY_PATH' sed -i -e '2i'"$new_ldpath" \ "${WORKDIR}/usr/bin/$(cross)ghc-${GHC_PV}" \ "${WORKDIR}/usr/bin/$(cross)ghci-${GHC_PV}" \ "${WORKDIR}/usr/bin/$(cross)ghc-pkg-${GHC_PV}" \ "${WORKDIR}/usr/bin/$(cross)hsc2hs" \ "${WORKDIR}/usr/bin/$(cross)runghc-${GHC_PV}" \ "$gp_back" \ || die "Adding LD_LIBRARY_PATH for wrappers failed" Namely ${EPREFIX}/$(get_libdir) is actively harmful because it tries to use prefix's libc but hot's dynamic laoder. They are not really compatible. Try the following: --- a/dev-lang/ghc/ghc-8.0.2.ebuild +++ b/dev-lang/ghc/ghc-8.0.2.ebuild @@ -320,13 +320,2 @@ relocate_ghc() { if use prefix; then - # and insert LD_LIBRARY_PATH entry to EPREFIX dir tree - # TODO: add the same for darwin's CHOST and it's DYLD_ - local new_ldpath='LD_LIBRARY_PATH="'${EPREFIX}/$(get_libdir):${EPREFIX}/usr/$(get_libdir)'${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH}"\nexport LD_LIBRARY_PATH' - sed -i -e '2i'"$new_ldpath" \ - "${WORKDIR}/usr/bin/$(cross)ghc-${GHC_PV}" \ - "${WORKDIR}/usr/bin/$(cross)ghci-${GHC_PV}" \ - "${WORKDIR}/usr/bin/$(cross)ghc-pkg-${GHC_PV}" \ - "${WORKDIR}/usr/bin/$(cross)hsc2hs" \ - "${WORKDIR}/usr/bin/$(cross)runghc-${GHC_PV}" \ - "$gp_back" \ - || die "Adding LD_LIBRARY_PATH for wrappers failed" hprefixify "${bin_libpath}"/${PN}*/settings If it works I'll commit it as-is.
Hi Horea, What is the glibc version of your host system? @Sergei, as we have discussed in the IRC, the fundamental problem is in the distributed ghc binary packages. They are compiled against new glibc's in the latest Gentoo, making them fail to resolve symbols on old glibc of enterprise linux. IMHO, the only way out is to assume less from the host glibc. Would you please distribute ghc binaries built against older versions of glibc? At least as old as glibc-2.5 on RHEL5 as of year 2017. (In reply to Sergei Trofimovich from comment #1)
Besides, I could only build ghc on RAP and RHEL5 and 6 by manually overriding ELF interpreter to Gentoo with patchelf.
(In reply to Benda Xu from comment #2) > At least as old as glibc-2.5 on RHEL5 as of year 2017. I just noticed that RHEL5 has reached end-of-life. So the glibc supported should be as old as glibc-2.12 on RHEL6.
(In reply to Benda Xu from comment #2) > Hi Horea, > > What is the glibc version of your host system? > > @Sergei, as we have discussed in the IRC, the fundamental problem is in the > distributed ghc binary packages. They are compiled against new glibc's in > the latest Gentoo, making them fail to resolve symbols on old glibc of > enterprise linux. I don't think the failure in #comment1 will disappear even if ghc would be built against the ancient glibc. There libc.so.6 loaded from RAP system by host's ld.so. > IMHO, the only way out is to assume less from the host glibc. Would you > please distribute ghc binaries built against older versions of glibc? At > least as old as glibc-2.5 on RHEL5 as of year 2017. Oldest non-masked available glibc in gentoo is 2.23. I don't think haskell@ can go older than that without major effort.
(In reply to Sergei Trofimovich from comment #5) > (In reply to Benda Xu from comment #2) > > @Sergei, as we have discussed in the IRC, the fundamental problem is in the > > distributed ghc binary packages. They are compiled against new glibc's in > > the latest Gentoo, making them fail to resolve symbols on old glibc of > > enterprise linux. > > I don't think the failure in #comment1 will disappear even if ghc would be > built against the ancient glibc. There libc.so.6 loaded from RAP system by host's ld.so. I agree with this and your patch should be applied. My point is that, the fundamental issue is the backward compatibility on system with old glibc. With your patch, Gentoo Prefix still cannot build ghc on most of the non-Gentoo systems. > > IMHO, the only way out is to assume less from the host glibc. Would you > > please distribute ghc binaries built against older versions of glibc? At > > least as old as glibc-2.5 on RHEL5 as of year 2017. > > Oldest non-masked available glibc in gentoo is 2.23. I don't think haskell@ > can go older than that without major effort. Therefore, in order to support Gentoo Prefix, either we bite the bullet to invest this major effort, or we find away to override ELF interpreter reliably.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a6af937438bbd6d88028a5cda7ff8ba20a16721e commit a6af937438bbd6d88028a5cda7ff8ba20a16721e Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2017-11-24 07:52:12 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2017-11-24 07:52:31 +0000 dev-lang/ghc: drop LD_LIBRARY_PATH hack, bug #637532 LD_LIBRARY_PATH only worked for prefix systems using host's libc. On systems with prefix/libc it causes host's ld.so to load prefix's libc.so. They are incompatible as ld.so relies on presence of certain private symbols libc.so Reported-by: Horea Christian Bug: https://bugs.gentoo.org/637532 Package-Manager: Portage-2.3.16, Repoman-2.3.6 dev-lang/ghc/Manifest | 62 +++++++++++++++++++++--------------------- dev-lang/ghc/ghc-7.10.3.ebuild | 11 -------- dev-lang/ghc/ghc-7.8.4.ebuild | 15 ---------- dev-lang/ghc/ghc-8.0.2.ebuild | 11 -------- dev-lang/ghc/ghc-8.2.1.ebuild | 11 -------- dev-lang/ghc/metadata.xml | 4 +-- 6 files changed, 33 insertions(+), 81 deletions(-)}
This works now.
(In reply to Benda Xu from comment #6) > I agree with this and your patch should be applied. My point is that, the > fundamental issue is the backward compatibility on system with old glibc. > With your patch, Gentoo Prefix still cannot build ghc on most of the > non-Gentoo systems. Agreed. > Therefore, in order to support Gentoo Prefix, either we bite the bullet to > invest this major effort, or we find away to override ELF interpreter > reliably. I would like interpreter change approach first. I guess it's roughly a matter of running 'patchelf --set-interpreter' (and maybe changing R*PATH, we'll try to tweak upstream to make R*PATH not to be absolute at some point). My questions are: 1. Do we have an INTERPRETER string to change to already exposed in an eclass? Or should we factor out something from toolchain-glibc? (Or autodiscover default interpreter from current compiler) 2. Does haskell@ need to build ghc in a special way (any special linker flags) to ease further INTERPRETER changes? 3. How to detect in an eclass fact that interpreter needs to be changed? (How to detect we are targeting prefix/libc and not old-style prefix osing host's libc?)
(In reply to Sergei Trofimovich from comment #9) > > Therefore, in order to support Gentoo Prefix, either we bite the bullet to > > invest this major effort, or we find a way to override ELF interpreter > > reliably. > > I would like interpreter change approach first. I guess it's roughly a > matter of running 'patchelf --set-interpreter' That's true. However, the problem is that patchelf does not work on all the architectures, see bug 591172. > (and maybe changing R*PATH, > we'll try to tweak upstream to make R*PATH not to be absolute at some point). As far as I can see, rpath does not work here. glibc ABI is not forward compatible. If libraries (ffi, ncurses, gmp, etc.) built against new glibc (from Gentoo) and dynamic linker from old host glibc are mixed, prebuilt ghc will fail on different library in different environments. That's the worst of portability. > My questions are: > > 1. Do we have an INTERPRETER string to change to already exposed in an > eclass? No we don't. > Or should we factor out something from toolchain-glibc? I don't think it a bussiness of glibc. The linker takes care of if it, via --dynamic-linker=xxxxx, or equivalently via gcc's -Wl,--dynamic-linker=xxxx. > (Or > autodiscover default interpreter from current compiler) That can only be done by parsing gcc specs, gcc -dumpspecs | grep dynamic-linker Very complex patterns to look for. > 2. Does haskell@ need to build ghc in a special way (any special linker > flags) to ease further INTERPRETER changes? Yes, although not INTERPRETER change: by providing a minimal ghc built statically, not dynamically linking against ffi, gmp, or ncurses fancy libraries. > 3. How to detect in an eclass fact that interpreter needs to be changed? > (How to detect we are targeting prefix/libc and not old-style prefix osing > host's libc?) if use prefix && ! use prefix-guest; then ...; fi prefix-guest is the USE flag for "old-style" prefix-rpath.
Hi Sergei, is it possible to ship intermediate c code instead of ELF binaries? see: https://elephly.net/posts/2017-01-09-bootstrapping-haskell-part-1.html and http://bootstrappable.org/projects.html
(In reply to Benda Xu from comment #11) > Hi Sergei, is it possible to ship intermediate c code instead of ELF > binaries? For GHC not today. GHC does not generate portable C code and final result has many drawbacks: 2-8 times slower compilation times, lack of SMP support, binary incompatibility with GHC built using native code generator. GHC used to have rudimentary support for bootstrapping with other haskell compilers but it was never finished and was eventually removed not only from compiler but also from standard libraries. Now GHC requires recent GHC for bootstrapping.
(In reply to Sergei Trofimovich from comment #12) > (In reply to Benda Xu from comment #11) > > Hi Sergei, is it possible to ship intermediate c code instead of ELF > > binaries? > > For GHC not today. GHC does not generate portable C code and final result > has many drawbacks: 2-8 times slower compilation times, lack of SMP support, > binary incompatibility with GHC built using native code generator. Do you mean the unregistered C here? The performance impact doesn't matter, because it is only for bootstrapping an optimized ghc. The latest ghc document implies unregistered C is portable: https://downloads.haskell.org/~ghc/8.0.2/docs/html/users_guide/codegens.html#unregisterised-compilation It might be: 1. rebuild ghc in unregistered C mode. 2. produce the intermediate unregistered C code to replace the pre-compiled ELF. 3. in an ebuild, compile the unregistered C into a suboptimal ghc. 4. use the suboptimal ghc to bootstrap an optimized one.
(In reply to Benda Xu from comment #13) > (In reply to Sergei Trofimovich from comment #12) > > (In reply to Benda Xu from comment #11) > > > Hi Sergei, is it possible to ship intermediate c code instead of ELF > > > binaries? > > > > For GHC not today. GHC does not generate portable C code and final result > > has many drawbacks: 2-8 times slower compilation times, lack of SMP support, > > binary incompatibility with GHC built using native code generator. > > Do you mean the unregistered C here? The performance impact doesn't matter, > because it is only for bootstrapping an optimized ghc. Yes, I'm talking about --enable-unregisterised mode of GHC. > The latest ghc document implies unregistered C is portable: > https://downloads.haskell.org/~ghc/8.0.2/docs/html/users_guide/codegens. > html#unregisterised-compilation To clarify: doc says not portable but "portable". Generated C code is portable in a sense that you don't need to write platform-specific code to retarget GHC to the new platform (as opposed to write assembly backend for each new target). You just need a C compiler for a target. GHC will infer all the platform specifics from target C compiler at ./configure phase (or will assume worst). For example simple example that shows the platform details of generated C code: -- M.hs module M where a x = x + 42 :: Int when compiled for 64-bit LE platform and 32-bit BE platform produce: $ x86_64-UNREG-linux-gnu-ghc -fforce-recomp -C M.hs && mv M.hc M-x86_64.hc $ m68k-unknown-linux-gnu-ghc -fforce-recomp -C M.hs && mv M.hc M-m68k.hc $ diff -u M-x86_64.hc M-m68k.hc EC_(base_GHCziNum_zdfNumInt_closure); FN_(M_a_entry) { ... -if ((W_)((((W_)Sp+16) - 0x28UL) < (W_)SpLim)) goto _c195; else goto _c196; +if ((W_)((((W_)Sp+8) - 0x14U) < (W_)SpLim)) goto _c195; else goto _c196; ... -_c193 = (W_)Hp-7; +_c19f = (W_)Hp-3; Note how low-level the code is. It checks for haskell stack overflow in bytes precisely knowing how much is needed on 32-bit system and 64-bit system(Sp+<n>). How 64-bit pointers have 3 bits for tags and 32-bit pointers have only 2 bits for tags (Hp-<n>). This C code is not portable in a sense that generated C code can be compiled on any platform. C is used by ghc exactly as an assembler is used by gcc. Even if assembly syntax would be the same on every platform you won't be able to use intermediate assembly files to bootstrap gcc on anything except the targeted system. > It might be: > 1. rebuild ghc in unregistered C mode. > 2. produce the intermediate unregistered C code to replace the pre-compiled > ELF. > 3. in an ebuild, compile the unregistered C into a suboptimal ghc. > 4. use the suboptimal ghc to bootstrap an optimized one. .hc file porting is how ghc was ported to a new platform in the olden days (around version ghc-6.4). It did not work at that time without considerable amount of manual work required. I works nowadays even worse due to bitrot of .hc porting infrastructure. Proper cross-compilation has better chance to succeed.
Hi Sergei, (In reply to Sergei Trofimovich from comment #14) > (In reply to Benda Xu from comment #13) > > To clarify: doc says not portable but "portable". > > Generated C code is portable in a sense that you don't need to write > platform-specific code to retarget GHC to the new platform (as opposed to > write assembly backend for each new target). You just need a C compiler for > a target. GHC will infer all the platform specifics from target C compiler > at ./configure phase (or will assume worst). > > For example simple example that shows the platform details of generated C > code: > -- M.hs > module M where a x = x + 42 :: Int > when compiled for 64-bit LE platform and 32-bit BE platform produce: > $ x86_64-UNREG-linux-gnu-ghc -fforce-recomp -C M.hs && mv M.hc > M-x86_64.hc > $ m68k-unknown-linux-gnu-ghc -fforce-recomp -C M.hs && mv M.hc M-m68k.hc > $ diff -u M-x86_64.hc M-m68k.hc > > EC_(base_GHCziNum_zdfNumInt_closure); > FN_(M_a_entry) { > ... > -if ((W_)((((W_)Sp+16) - 0x28UL) < (W_)SpLim)) goto _c195; else goto > _c196; > +if ((W_)((((W_)Sp+8) - 0x14U) < (W_)SpLim)) goto _c195; else goto _c196; > ... > -_c193 = (W_)Hp-7; > +_c19f = (W_)Hp-3; > > Note how low-level the code is. It checks for haskell stack overflow in > bytes precisely knowing how much is needed on 32-bit system and 64-bit > system(Sp+<n>). How 64-bit pointers have 3 bits for tags and 32-bit pointers > have only 2 bits for tags (Hp-<n>). > > This C code is not portable in a sense that generated C code can be compiled > on any platform. C is used by ghc exactly as an assembler is used by gcc. > Even if assembly syntax would be the same on every platform you won't be > able to use intermediate assembly files to bootstrap gcc on anything except > the targeted system. I did not mean to port ghc to a new architecture, but to make ghc bootstrap on Prefix, where EPREFIX is not a predefined location and so shipping prebuilt binaries does not help. In this regard, even keeping assembly-level code is acceptable, because we only need to flexibility to specify a prefix. What do you think? Benda
I don't think current build system supports "relink-objects" use case. As ghc is a bootstrapping compiler it relies on linker command constructed by ghc.
(In reply to Sergei Trofimovich from comment #16) > I don't think current build system supports "relink-objects" use case. As > ghc is a bootstrapping compiler it relies on linker command constructed by > ghc. If the missing piece is only a set of linker command line arguments, the "relink objects" way of compiling out a seed ghc for bootstrap is doable.
*** Bug 738098 has been marked as a duplicate of this bug. ***
Is this going anywhere? Honestly, I don't think we should bother about supporting "all" Prefix targets. rpath-based Linux Prefixes, may just work, or not. I won't even think about macOS here. If it works on RAP now, then let's consider that a success, and cherish it.
Created attachment 685701 [details, diff] Patch to relocate prebuilt binaries by overriding interpreter path Found this bug as I was about to open a new one to submit this patch. I see that this solution is controversial, but it is simple and worked fine on ppc64le, and is much better than being stuck without ghc and its dependees (pandoc). If solution is not good enough for merging, at least it will be archived here on bugzilla for users to apply the patch and move on (I'd leave the bug open so that it shows up in search results).
Created attachment 685704 [details, diff] Patch to relocate prebuilt binaries by overriding interpreter path (ghc ebuild patch)
