+ ./cmd/dist/dist bootstrap -a Building Go toolchain1 using /usr/lib/go. Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1. Building Go toolchain2 using go_bootstrap and Go toolchain1. Building Go toolchain3 using go_bootstrap and Go toolchain2. Building packages and commands for linux/amd64. # runtime/race/internal/amd64v1 open ../../work/go/src/runtime/race/internal/amd64v1/race_linux.syso: not a directory go tool dist: FAILED: /var/tmp/portage/dev-lang/go-1.21.3/work/go/pkg/tool/linux_amd64/go_bootstrap install std: exit status 1 (Note that "not a directory' means it's a foreign binary blob in my setup; the log snip here is for context of where the blob execution is attempted) 1.21.1 did not have this issue.
(1.21.1 does have the issue now, however)
Correction: this appears to have evaded my check previously because the blob wasn't +x :/ So not a new issue, but an issue nevertheless.
this bug is strange, if I do split the build doing `ebuild go-1.21.3.ebuild `{compile,install,merge} most of the times all files in /usr/lib/go/pkg/tool/linux_amd64/ are of type "ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=..., not stripped" doing a normal emerge instead create files which have no type and exactly the same dimension of the good executable. file /usr/lib/go/pkg/tool/linux_amd64/* | cut -c 1-72 /usr/lib/go/pkg/tool/linux_amd64/addr2line: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/asm: data /usr/lib/go/pkg/tool/linux_amd64/buildid: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/cgo: data /usr/lib/go/pkg/tool/linux_amd64/compile: data /usr/lib/go/pkg/tool/linux_amd64/covdata: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/cover: data /usr/lib/go/pkg/tool/linux_amd64/dist: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/distpack: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/doc: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/fix: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/link: data /usr/lib/go/pkg/tool/linux_amd64/nm: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/objdump: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/pack: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/pprof: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/test2json: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/trace: ELF 64-bit LSB executable, x /usr/lib/go/pkg/tool/linux_amd64/vet: data
Seems like I hit this one too, couldn't compile go-packages suddenly: error obtaining buildID for go tool compile: fork/exec /usr/lib/go/pkg/tool/linux_amd64/compile: exec format error $ file usr/lib/go/pkg/tool/linux_amd64/* | cut -c 1-72 usr/lib/go/pkg/tool/linux_amd64/addr2line: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/asm: data usr/lib/go/pkg/tool/linux_amd64/buildid: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/cgo: data usr/lib/go/pkg/tool/linux_amd64/compile: data usr/lib/go/pkg/tool/linux_amd64/covdata: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/cover: data usr/lib/go/pkg/tool/linux_amd64/dist: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/distpack: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/doc: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/fix: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/link: data usr/lib/go/pkg/tool/linux_amd64/nm: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/objdump: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/pack: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/pprof: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/test2json: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/trace: ELF 64-bit LSB executable, x8 usr/lib/go/pkg/tool/linux_amd64/vet: data hexdumping the "data" files they seem to contain: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000fa0 00 00 00 00 00 00 00 00 00 00 00 00 71 65 35 70 |............ <string of base64-encoded data removed> 00000ff0 00 | .| 00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00aca180 <repetition of the same base64-encoded data> 00aca1d0 30 53 4b 00 00 00 00 00 00 00 00 00 00 00 00 00 |0SK.............| 00aca1e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 0113a3b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............| 0113a3bf decoding the strings gives nothing that makes sense to me.
It'd help if somebody could give some simple way they're stripping binaries to test.
(In reply to Sam James from comment #5) > It'd help if somebody could give some simple way they're stripping binaries > to test. How would I check how they are stripped? Using go-1.21.1 to emerge go-1.21.3 resulted in: # file /usr/lib/go/pkg/tool/linux_amd64/* | cut -c 1-72|grep data$ /usr/lib/go/pkg/tool/linux_amd64/cover: data /usr/lib/go/pkg/tool/linux_amd64/vet: data running execsnoop during the emerge resulted in these commands containing strip: $ grep -i strip execsnoop.log estrip 9557 9307 0 /usr/lib/portage/python3.12/estrip --queue bash 9557 9307 0 /bin/bash /usr/lib/portage/python3.12/estrip --queue estrip 9567 9307 0 /usr/lib/portage/python3.12/estrip --ignore bash 9567 9307 0 /bin/bash /usr/lib/portage/python3.12/estrip --ignore estrip 9577 9307 0 /usr/lib/portage/python3.12/estrip --dequeue bash 9577 9307 0 /bin/bash /usr/lib/portage/python3.12/estrip --dequeue Checking the mentions of the broken files results in: $ grep -E "_amd64/(cover|vet)" execsnoop.log scanelf 9376 9371 0 /usr/bin/scanelf -qyRF %F %n /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/lib/time/update.bash /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/fix /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/test2json /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/objdump /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/addr2line /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/pack /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/pprof /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/link /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/asm /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/dist /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/cover /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/vet /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/trace /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/nm /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/doc /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/buildid /var/tmp/portage/dev-lang/go-1.21.3/image/usr/lib/go/pkg/tool/linux_amd64/distpack
(In reply to Sam James from comment #5) > It'd help if somebody could give some simple way they're stripping binaries > to test. Still working on a more comprehensive audit that doesn't have false positives, but this works for now (in /etc/portage/bashrc): post_src_unpack() { local f einfo "Looking for x86 ELFs..." while read f; do if [ "$(head -c 4 "$f")" = $'\x7fELF' ] && file "$f" | grep -qi 'ELF.*LSB.*86'; then ewarn "$f seems to be an x86 ELF!" rm -v "$f" || die "rm failed" ln -vs "/dev/null/INVALID" "$f" fi done < <(find "${WORKDIR}" -type f) }
By that, I mean this works without false positives, but still may have false negatives (it doesn't look inside tar files, for example). Note that the /dev/null/INVALID link is used to ensure any attempted access becomes a harder error, rather than possibly being ignored.
tought workaround is: emerge -C dev-lang/go FEATURES="-nostrip -binpkg-dostrip -installsources" emerge -1 dev-lang/go emerge -va @golang-rebuild but: seed-desk /srv # file /usr/lib/go/pkg/tool/linux_amd64/* | grep data /usr/lib/go/pkg/tool/linux_amd64/covdata: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=tlbpTPIPIVnTuF1iinte/YRn3gp2U2dpNmqiT_UVn/JtHG5SbBg-XWMJM1FrLY/p6aPYeU5E9eDl5DiqzVk, not stripped /usr/lib/go/pkg/tool/linux_amd64/cover: data /usr/lib/go/pkg/tool/linux_amd64/vet: data
If your name isn't Luke and/or you're not using a Portage hook to strip binaries to avoid binary blobs during builds, you're probably not hitting this bug. You may be hitting bug 917224 instead if you're using ZFS. Let's keep this bug about the binary blob issue, please.
Apparently these blobs are blobs because Go devs don't want to depend on a C++ compiler They come from LLVM, and can be built quite simply. Unfortunately, the llvm.org eclass assumes $PV is always a LLVM version, so it's not quite a simple fix without extracting the entirety of the LLVM code (which slows down my post_src_unpack significantly, note). But this seems to work as a proof-of-concept: --- go-1.21.4.ebuild 2023-11-10 17:50:31.915739657 -0500 +++ go-1.21.4-r1.ebuild 2023-11-11 23:21:08.496994349 -0500 @@ -18,7 +18,9 @@ inherit git-r3 ;; *) - SRC_URI="https://storage.googleapis.com/golang/go${MY_PV}.src.tar.gz " + SRC_URI="https://storage.googleapis.com/golang/go${MY_PV}.src.tar.gz + https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.6/llvm-project-16.0.6.src.tar.xz + " S="${WORKDIR}"/go case ${PV} in *_beta*|*_rc*) ;; @@ -125,7 +127,31 @@ "${FILESDIR}"/go-never-download-newer-toolchains.patch ) +race_blobs=( + race_linux_{arm64,ppc64le,s390x}.syso + race_darwin_arm64.syso + internal/amd64v1/race_{netbsd,windows,freebsd,linux,openbsd,darwin}.syso + internal/amd64v3/race_linux.syso +) + +src_unpack() { + default + + cd go/src/runtime/race && rm -v "${race_blobs[@]}" || die +} + src_compile() { + ebegin "Building race syso objects" + pushd "${WORKDIR}"/llvm*.src/compiler-rt/lib/tsan/go || die + local CRT_BUILD_DIR="$PWD" + ./buildgo.sh || die + popd + local f + for f in "${race_blobs[@]}"; do + ln -v "${CRT_BUILD_DIR}"/race_linux_*.syso "src/runtime/race/$f" || die + done + eend 0 + if has_version -b ">=dev-lang/go-${GO_BOOTSTRAP_MIN}"; then export GOROOT_BOOTSTRAP="${BROOT}/usr/lib/go" elif has_version -b ">=dev-lang/go-bootstrap-${GO_BOOTSTRAP_MIN}"; then Disclaimer: I am unsure if the non-native .syso files are ever used, and if so, the proper way to detect their necessity and build them correctly. It may be better to just identify which one is correct and delete the rest? Disclaimer 2: Apparently there's a boringssl thing that has a similar issue, but the ebuild installed fine without actually fixing it :| (Upstream also says it should be possible to build without the race detector, so maybe a USE flag is in order.)
