Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 645626 - sys-libs/musl doesn't cross-compile: [[ -e "${D}"/lib/ld-musl-${arch}.so.1 ]] || die;
Summary: sys-libs/musl doesn't cross-compile: [[ -e "${D}"/lib/ld-musl-${arch}.so.1 ]]...
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Anthony Basile
URL:
Whiteboard:
Keywords:
: 696236 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-01-24 21:27 UTC by Stuart Longland
Modified: 2019-10-04 17:44 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build log (build.log,846.49 KB, text/x-log)
2018-01-24 21:27 UTC, Stuart Longland
Details
proposed ebuild (musl-1.1.18-r1.ebuild,3.33 KB, text/plain)
2018-01-25 09:45 UTC, Stuart Longland
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stuart Longland 2018-01-24 21:27:48 UTC
Created attachment 516502 [details]
build log

Seems musl incorrectly generates the /lib/ld-musl-${arch}.so.1 symlink:

./tools/install.sh -D -l /usr/lib/libc.so /tmp/arm-unknown-linux-musleabi/portage/sys-libs/musl-1.1.18-r1/image//lib/ld-musl-arm.so.
1 || true


This causes the build to fail with this cryptic message:
 * ERROR: sys-libs/musl-1.1.18-r1::gentoo failed (install phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_install
 *   environment, line 2330:  Called die
 * The specific snippet of code:
 *           [[ -e "${D}"/lib/ld-musl-${arch}.so.1 ]] || die;
 * 
 * If you need support, post the output of `emerge --info '=sys-libs/musl-1.1.18-r1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=sys-libs/musl-1.1.18-r1::gentoo'`.
 * The complete build log is located at '/tmp/arm-unknown-linux-musleabi/portage/sys-libs/musl-1.1.18-r1/temp/build.log'.
 * The ebuild environment file is located at '/tmp/arm-unknown-linux-musleabi/portage/sys-libs/musl-1.1.18-r1/temp/environment'.
 * Working directory: '/tmp/arm-unknown-linux-musleabi/portage/sys-libs/musl-1.1.18-r1/work/musl-1.1.18'
 * S: '/tmp/arm-unknown-linux-musleabi/portage/sys-libs/musl-1.1.18-r1/work/musl-1.1.18'

The link should be relative; the command should read:
./tools/install.sh -D -l ../usr/lib/libc.so /tmp/arm-unknown-linux-musleabi/portage/sys-libs/musl-1.1.18-r1/image//lib/ld-musl-arm.so.
1 || true

Reproduction steps:
1. layman -a musl
2. crossdev -t arm-unknown-linux-musleabi # This will fail when it gets to gcc.
3. Go to the overlay generated by crossdev, replace the symlink for `gcc` that currently points to /usr/portage, to the equivalent in the musl overlay.
4. emerge cross-arm-unknown-linux-musleabi/gcc  # You may need to play with USE-flag settings to get it to build
5. Set up /usr/cross-arm-unknown-linux-musleabi/etc/portage/make.conf with ELIBC="musl" and other options
6. arm-unknown-linux-musleabi-emerge sys-libs/musl
Comment 1 Stuart Longland 2018-01-24 22:42:29 UTC
Further digging, it seems ${arch} is not set…

Modified ebuild:
        if [[ ${CATEGORY} != cross-* ]] ; then
                local arch=$("${D}"usr/lib/libc.so 2>&1 | sed -n '1s/^musl libc (\(.*\))$/\1/p')
                [[ -e "${D}"/lib/ld-musl-${arch}.so.1 ]] || die "Broken ld-musl-${arch}.so.1"
                cp "${FILESDIR}"/ldconfig.in "${T}" || die
                sed -e "s|@@ARCH@@|${arch}|" "${T}"/ldconfig.in > "${T}"/ldconfig || die
                into /
                dosbin "${T}"/ldconfig
                into /usr
                dobin "${T}"/getconf
                dobin "${T}"/getent
                dobin "${T}"/iconv
                echo 'LDPATH="include ld.so.conf.d/*.conf"' > "${T}"/00musl || die
                doenvd "${T}"/00musl || die
        fi

Result:
 * ERROR: sys-libs/musl-1.1.18-r1::gentoo failed (install phase):
 *   Broken ld-musl-.so.1
Comment 2 Stuart Longland 2018-01-25 09:45:34 UTC
Created attachment 516524 [details]
proposed ebuild

Okay, fixed the problem.  The problem is two-fold.

1. The Makefile symlinks /usr/lib/libc.so to the target, which is wrong because
   when installed in a subdirectory, the actual file we want will be in
   ${ROOT}/usr/lib/libc.so.

   I fix this by patching the Makefile in src_prepare:

src_prepare() {
	# Patch Makefile to fix ld.so symlink
	sed -i -e '/\$(INSTALL) -D -l \$(libdir)\/libc.so/ {
		s:\$(libdir)/libc.so:$(shell realpath --relative-to="$(@D)" $(DESTDIR)&):
	}' "${S}/Makefile" || die "patch failed"
	eapply_user
}

2. The ebuild script tries to set ${arch} by executing ld.so.  An x86-64 host
   is *never* going to natively execute an armv5te binary.  The solution, good
   ol'e `strings`:

	local arch=$( $(tc-getPROG STRINGS strings) "${D}"usr/lib/libc.so \
			| sed -n '/^musl libc / { s/^.*(\(.*\))$/\1/; p }')

   toolchain-funcs lacks a tc-getSTRINGS, but I was able to reverse engineer
   how the others worked to come up with that.
Comment 3 Anthony Basile gentoo-dev 2018-01-25 12:25:47 UTC
(In reply to Stuart Longland from comment #2)
> Created attachment 516524 [details]
> proposed ebuild
> 
> Okay, fixed the problem.  The problem is two-fold.
> 
> 1. The Makefile symlinks /usr/lib/libc.so to the target, which is wrong
> because
>    when installed in a subdirectory, the actual file we want will be in
>    ${ROOT}/usr/lib/libc.so.
> 
>    I fix this by patching the Makefile in src_prepare:
> 
> src_prepare() {
> 	# Patch Makefile to fix ld.so symlink
> 	sed -i -e '/\$(INSTALL) -D -l \$(libdir)\/libc.so/ {
> 		s:\$(libdir)/libc.so:$(shell realpath --relative-to="$(@D)" $(DESTDIR)&):
> 	}' "${S}/Makefile" || die "patch failed"
> 	eapply_user
> }

> 
> 2. The ebuild script tries to set ${arch} by executing ld.so.  An x86-64 host
>    is *never* going to natively execute an armv5te binary.  The solution,
> good
>    ol'e `strings`:
> 
> 	local arch=$( $(tc-getPROG STRINGS strings) "${D}"usr/lib/libc.so \
> 			| sed -n '/^musl libc / { s/^.*(\(.*\))$/\1/; p }')
> 
>    toolchain-funcs lacks a tc-getSTRINGS, but I was able to reverse engineer
>    how the others worked to come up with that.


Have you tested any of these changes on a native compile?  To be honest, I'm not interested in cross compiling. I'm willing to support it but if and only if it doesn't break how things are being done natively.  I suspect change 1 above will cause problems in a catalyst run.  Some suggestions:

1) get tc-getSTRINGS into toolchian.eclass so that its discussed by a broader group of people who can asses your approach.

2) test on a native comile to make sure nothing breaks and protect anything that changes stuff behind `is_crosscompile`.
Comment 4 Stuart Longland 2018-01-25 12:53:05 UTC
(In reply to Anthony Basile from comment #3)
> Have you tested any of these changes on a native compile?  To be honest, I'm
> not interested in cross compiling. I'm willing to support it but if and only
> if it doesn't break how things are being done natively.  I suspect change 1
> above will cause problems in a catalyst run.

Not as yet, the steps I'm doing are:
0. Build a arm-unknown-linux-musleabi toolchain.
1. Get a basic system going in /usr/arm-unknown-linux-musleabi that I can make into a chroot environment on the target device.
2. Throw this on an ARMv5 system, do an `emerge -e @system`
3. Install catalyst
4. Tar up my root as a seed stage and throw that at catalyst.

Not much different to what I did for mips64eln64.  At the moment, I'm stuck at (1)… specifically, compiling sys-devel/gcc to run on the target.  Soon as I'm over that hump though, I will be trying this out with Catalyst.

I'll probably also do this all over again for arm-unknown-linux-musleabihf… as I have a Raspberry Pi here (ARMv6-based) that I'd like to try musl on.

> Some suggestions:
> 
> 1) get tc-getSTRINGS into toolchian.eclass so that its discussed by a
> broader group of people who can asses your approach.
> 
> 2) test on a native comile to make sure nothing breaks and protect anything
> that changes stuff behind `is_crosscompile`.

Yep, will do.
Comment 5 Anthony Basile gentoo-dev 2018-01-25 17:38:03 UTC
(In reply to Stuart Longland from comment #4)
> (In reply to Anthony Basile from comment #3)
> > Have you tested any of these changes on a native compile?  To be honest, I'm
> > not interested in cross compiling. I'm willing to support it but if and only
> > if it doesn't break how things are being done natively.  I suspect change 1
> > above will cause problems in a catalyst run.
> 
> Not as yet, the steps I'm doing are:
> 0. Build a arm-unknown-linux-musleabi toolchain.
> 1. Get a basic system going in /usr/arm-unknown-linux-musleabi that I can
> make into a chroot environment on the target device.
> 2. Throw this on an ARMv5 system, do an `emerge -e @system`
> 3. Install catalyst
> 4. Tar up my root as a seed stage and throw that at catalyst.
> 
> Not much different to what I did for mips64eln64.  At the moment, I'm stuck
> at (1)… specifically, compiling sys-devel/gcc to run on the target.  Soon as
> I'm over that hump though, I will be trying this out with Catalyst.

you don't have to go to exotic systems.  you can test with amd64 and i686 tarballs on the mirrors.

> 
> I'll probably also do this all over again for arm-unknown-linux-musleabihf…
> as I have a Raspberry Pi here (ARMv6-based) that I'd like to try musl on.
> 
> > Some suggestions:
> > 
> > 1) get tc-getSTRINGS into toolchian.eclass so that its discussed by a
> > broader group of people who can asses your approach.
> > 
> > 2) test on a native comile to make sure nothing breaks and protect anything
> > that changes stuff behind `is_crosscompile`.
> 
> Yep, will do.
Comment 6 Stuart Longland 2018-01-25 23:27:00 UTC
Yeah, I realise that… actually my i[456]86 and AMD64 musl builds are out of date.  Been meaning to fix that for a while, so this gives me an excuse.
Comment 7 Stuart Longland 2018-01-26 14:04:01 UTC
I'm in the process of testing this with a native build within Catalyst…

I am presently battling this little gem:

copying chroot-functions.sh to /var/tmp/catalyst/tmp/musl/stage1-amd64-20180126//tmp
Ensure the file has the executable bit set
Running stage1-chroot.sh in chroot /var/tmp/catalyst/tmp/musl/stage1-amd64-20180126/
Adding USE=" build" to make.conf for portage build
emerge --quiet --usepkg --buildpkg --newuse --oneshot --update --newuse sys-apps/portage
Skipping seed stage update...
emerge --quiet --usepkg --buildpkg --newuse --oneshot --nodeps sys-apps/baselayout
emerge --quiet --usepkg --buildpkg --newuse --oneshot app-shells/bash:0 virtual/pkgconfig sys-devel/bison dev-python/pyxattr sys-devel/binutils sys-apps/diffutils app-arch/gzip sys-apps/gawk sys-apps/coreutils sys-apps/less sys-apps/baselayout sys-devel/gcc sys-apps/findutils app-arch/bzip2 virtual/libc app-arch/xz-utils net-misc/wget sys-apps/makedev sys-devel/libtool app-arch/tar sys-devel/patch virtual/package-manager virtual/editor sys-apps/grep sys-apps/file sys-apps/sandbox dev-util/pkgconf net-misc/rsync sys-devel/flex sys-devel/autoconf sys-apps/net-tools virtual/shadow sys-apps/which sys-apps/sed virtual/os-headers sys-devel/gnuconfig sys-devel/gettext sys-apps/attr sys-devel/make sys-devel/automake 

!!! The ebuild selected to satisfy ">=app-text/asciidoc-8.6.3" has unmet requirements.
- app-text/asciidoc-8.6.10::gentoo USE="-examples -graphviz -highlight -test" PYTHON_SINGLE_TARGET="(-pypy) -python2_7" PYTHON_TARGETS="python2_7 (-pypy)"

  The following REQUIRED_USE flag constraints are unsatisfied:
    exactly-one-of ( python_single_target_pypy python_single_target_python2_7 )

  The above constraints are a subset of the following complete expression:
    exactly-one-of ( python_single_target_pypy python_single_target_python2_7 ) python_single_target_pypy? ( python_targets_pypy ) python_single_target_python2_7? ( python_targets_python2_7 )

(dependency required by "sys-apps/paludis-2.6.0::gentoo" [ebuild])
(dependency required by "app-admin/perl-cleaner-2.25::gentoo" [ebuild])
(dependency required by "dev-lang/perl-5.24.3::gentoo" [ebuild])
(dependency required by "sys-devel/autoconf-2.69-r4::gentoo" [ebuild])
(dependency required by "sys-devel/libtool-2.4.6-r3::gentoo" [ebuild])
(dependency required by "sys-devel/libtool" [argument])

The problem affects glibc too… basically unless I manually break into the chroot and tack portage-utils onto that list, perl-cleaner tries to drag in paludis.

Soon as I've worked around that issue, I should be in a better position to exercise these ebuild changes.
Comment 8 Anthony Basile gentoo-dev 2018-01-26 14:06:49 UTC
(In reply to Stuart Longland from comment #7)
> I'm in the process of testing this with a native build within Catalyst…
> 
> I am presently battling this little gem:
> 

upgrade your system to portage-2.3.20.  make sure it also gets into the catalyst tarballs.  i just marked it stable for amd64 and x86 in my local portage tree.  its a really nasty bug in 2.3.19-r1.  also, use catalyst-2.0.18-r4.
Comment 9 Anthony Basile gentoo-dev 2018-01-26 14:31:43 UTC
> make sure it also gets into the
> catalyst tarballs. 

let me explain, when you create the catalyst tarballs using `catalyst -s current` then portage gets in there.  you need to make sure that portage-2.3.20 is marked stable for amd64 and x86 BEFORE you run `catalyst -s current`.
Comment 10 Stuart Longland 2018-01-26 14:34:57 UTC
Ahh, right… wondered if it was something like that.

It'll be a tomorrow job methinks.  Desktop is hibernated so I can get some sleep (a quiet room-mate a Phenom II X6 loaded with fans does not make!), but it sounds like the best course will be for me to unpack my seed tarballs, update portage on those, then re-pack them and try it all again.

If I wait until daylight hours, that should give the mirrors plenty of time to propagate those changes. :-)
Comment 11 Stuart Longland 2018-01-27 11:15:57 UTC
Okay, so after some rejigging /usr/portage to make portage-2.3.20 stable and patching musl… and an own-goal nuking my /dev tree (catalyst didn't unmount the bind mounts properly before I did an rm -fr) … I got as far as stage 3:

emerge --quiet --usepkg --buildpkg --newuse -e @system

emerge: there are no ebuilds to satisfy "x11-misc/shared-mime-info".
(dependency required by "dev-libs/glib-2.52.3::musl[mime]" [ebuild])
(dependency required by "dev-util/pkgconfig-0.29.2::gentoo[-internal-glib]" [ebuild])
(dependency required by "virtual/pkgconfig-0-r1::gentoo" [ebuild])
(dependency required by "sys-apps/kbd-2.0.4::gentoo" [ebuild])
(dependency required by "@system" [argument])

!!! catalyst: run script failed.

The fact that it successfully completed stage 2 suggests to me that native building of musl with this ebuild works.
Comment 12 Stuart Longland 2018-01-28 00:47:37 UTC
(In reply to Anthony Basile from comment #3)
> 1) get tc-getSTRINGS into toolchian.eclass so that its discussed by a
> broader group of people who can asses your approach.

Okay, so I floated the idea in bug #645888 and it was suggested that `strings` is `strings` no matter what you call it.

So that earlier change can be replaced with:

	local arch=$( strings "${D}"usr/lib/libc.so \
			| sed -n '/^musl libc / { s/^.*(\(.*\))$/\1/; p }')

I think SpanKY raises a good point though; why can't we derive ${arch} from the ARCH or CTARGET variables?

I tried to find a way to make it work, without really understanding why the code is there.  Perhaps we need to re-consider this?

> 2) test on a native comile to make sure nothing breaks and protect anything
> that changes stuff behind `is_crosscompile`.

After resolving an unrelated issue with my portage snapshot, my desktop coughed up these:

http://repo.longlandclan.id.au/gentoo/stages/musl/x86/amd64/

So evidently, it works within Catalyst, at least for AMD64.  I've still got to get some things straightened out with my i[456]86 stages.
Comment 13 Stuart Longland 2018-01-28 01:51:34 UTC
Okay, think I've found the mystery of why x86 is broken:

RC=0 stuartl@rikishi /tmp $ cat /usr/portage/profiles/hardened/linux/musl/x86/parent 
../../../../default/linux/uclibc/x86
..
RC=0 stuartl@rikishi /tmp $ cat /usr/portage/profiles/hardened/linux/musl/amd64/parent 
../../../../default/linux/musl/amd64
..
Comment 14 Anthony Basile gentoo-dev 2018-01-28 02:01:51 UTC
(In reply to Stuart Longland from comment #13)
> Okay, think I've found the mystery of why x86 is broken:
> 
> RC=0 stuartl@rikishi /tmp $ cat
> /usr/portage/profiles/hardened/linux/musl/x86/parent 
> ../../../../default/linux/uclibc/x86
> ..
> RC=0 stuartl@rikishi /tmp $ cat
> /usr/portage/profiles/hardened/linux/musl/amd64/parent 
> ../../../../default/linux/musl/amd64
> ..

i was just made aware in bug #645926.  there are new profiles coming down the pipeline, so i'm not going to fix this now, just replace it with the new profiles.
Comment 15 Stuart Longland 2018-01-28 07:08:36 UTC
Right, so having fixed up the profile reference in my local tree, I now have successful i486 and i686 builds:

http://repo.longlandclan.id.au/gentoo/stages/musl/x86/i486/ (uploading at time of writing)
http://repo.longlandclan.id.au/gentoo/stages/musl/x86/i686/

It'd be nice to try the changes out on non-x86 as well, but I'm confident it shouldn't break anything there.
Comment 16 Felix Janda 2018-01-28 13:32:43 UTC
This bug is mainly my fault.

Deriving the ldso name from CTARGET should be possible, but would
involve a lot of case work, and would need to be updated each time we
add a new arch. (The name can depend on whether hard or soft float is
used and on endianness.)

Alpine[1] abuses the Makefile to get the ldso name. Alternatively,
the ldso name is the concatenation of ARCH and SUBARCH from config.mak.

[1]: https://git.alpinelinux.org/cgit/aports/plain/main/musl/APKBUILD
Comment 17 Stuart Longland 2018-01-28 13:43:09 UTC
(In reply to Felix Janda from comment #16)
> This bug is mainly my fault.

Hey, I'm not criticising. :-)

> Deriving the ldso name from CTARGET should be possible, but would
> involve a lot of case work, and would need to be updated each time we
> add a new arch. (The name can depend on whether hard or soft float is
> used and on endianness.)

Just a thought, could we do something like this:

   case ${CTARGET} in
      arm-*-linux-musleabi)  arch="arm" ;;
      x86_64-*-linux-musl)   arch="x86_64" ;;
      # … etc
      *)
         arch=$( strings "${D}"usr/lib/libc.so \
            | sed -n '/^musl libc / { s/^.*(\(.*\))$/\1/; p }')
         ;;
   esac

Then, we only fall through to using `strings` if other methods fail.  We might put an `ewarn` message there to say to the effect of "Unrecognised target ${CTARGET} maps to ${arch}, please file a bug to get this recorded".

The build will still work then, and we get to learn what the different combinations are.

Alternatively, do we just rely on the fact that there's going to be a symlink matching the pattern ld-musl-*.so, and extract ${arch} that way?

> Alpine[1] abuses the Makefile to get the ldso name. Alternatively,
> the ldso name is the concatenation of ARCH and SUBARCH from config.mak.
> 
> [1]: https://git.alpinelinux.org/cgit/aports/plain/main/musl/APKBUILD

That sounds simple enough to extract.
Comment 18 Felix Janda 2018-02-23 00:30:49 UTC
hardcoding all possible dynamic linker names looks like a lot of work.
Note that they depend on softfloat/hardfloat and the endianess.

For the second problem raised in Comment 2, try if the following works:

diff --git a/sys-libs/musl/musl-1.1.18.ebuild b/sys-libs/musl/musl-1.1.18.ebuild
index e36e219945..85ef92c6db 100644
--- a/sys-libs/musl/musl-1.1.18.ebuild
+++ b/sys-libs/musl/musl-1.1.18.ebuild
@@ -92,7 +92,7 @@ src_install() {
 	dosym ${sysroot}/lib/${ldso} ${sysroot}/usr/bin/ldd
 
 	if [[ ${CATEGORY} != cross-* ]] ; then
-		local arch=$("${D}"usr/lib/libc.so 2>&1 | sed -n '1s/^musl libc (\(.*\))$/\1/p')
+		local arch=$(sed -n 's/^ARCH = \(.*\)$/\1/p' config.mak)$(sed -n 's/^SUBARCH = \(.*\)$/\1/p' config.mak)
 		[[ -e "${D}"/lib/ld-musl-${arch}.so.1 ]] || die
 		cp "${FILESDIR}"/ldconfig.in "${T}" || die
 		sed -e "s|@@ARCH@@|${arch}|" "${T}"/ldconfig.in > "${T}"/ldconfig || die


I don't quite understand the first problem in Commment 2. The symlink
for the dynamic linker looks totally fine to me. Note that the dynamic
linker becomes only useful after deploying the crosscompiled system or
after chrooting into it (using something like qemu userland emulation).
Comment 19 hanetzer 2018-08-23 23:57:01 UTC
Not to be a nag, but I just ran into this today myself when using
crossdev to emerge @system into a sdcard for my asus c201 chromebook
I'm doing u-boot dev on/for/with and was wondering if there was
any progress towards a resolution. Quite a lot works well using
cross-emerge --root /mnt --sysroot /mnt @system, but without a libc
its kinda pointless.
Comment 20 tt_1 2019-10-04 17:44:33 UTC
*** Bug 696236 has been marked as a duplicate of this bug. ***