Created attachment 556434 [details] gcc-build-logs.tar.bz2 Hello, I'm bootstrapping Gentoo Prefix and on Stage 3 it fails due to gcc 8.2.0-r4 not compiling. The error: /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld: 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument collect2: error: ld returned 2 exit status Makefile:2935: recipe for target 'gcov' failed make[3]: *** [gcov] Error 1 I generated a Docker image with the system state that you can use to inspect the state and try stuff (note: this is Gentoo Prefix bootstrapped in /tmp/gentoo, so you have everything gentoo-related under /tmp/gentoo, the Docker is a Ubuntu 16.04): docker pull awesomebytes/gentoo_prefix_latest_image docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash Attaching gcc-build-logs.tar.bz2
Created attachment 556436 [details] emerge info command
Created attachment 556438 [details] emerge pqv command
Created attachment 556440 [details] Last 500 lines of the build log
I investigated a bit more. If you run that docker image... And you modify /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld to echo the command that is being executed (at the end of collect-ld): *) echo "Executing:" echo $original ${1+"$@"} exec $original ${1+"$@"} ;; esac And you run the same command (I'm not sure if should be from the prev-gcc folder or the gcc folder, but they have the same result): cd /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-gcc /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++ -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/ -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++ -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -isystem /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu -isystem /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include -isystem /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++ -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -no-pie -fno-PIE -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc gcov.o \ > hash-table.o ggc-none.o libcommon.a ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a -o gcov You get the actual command executed: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld --eh-frame-hdr -m elf_x86_64 -dynamic-linker /tmp/gentoo/lib64/ld-linux-x86-64.so.2 -o gcov /tmp/gentoo/usr/lib/../lib64/crt1.o /tmp/gentoo/usr/lib/../lib64/crti.o /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/crtbegin.o -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc -L/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -L/tmp/gentoo/lib/../lib64 -L/tmp/gentoo/usr/lib/../lib64 -L/tmp/gentoo/lib -L/tmp/gentoo/usr/lib gcov.o hash-table.o ggc-none.o libcommon.a ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a -Bstatic -lstdc++ -Bdynamic -lm -lgcc -lgcc_eh -lc -lgcc -lgcc_eh /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/crtend.o /tmp/gentoo/usr/lib/../lib64/crtn.o Which works... I actually ran again the bootstrapping of Gentoo Prefix in that same image (just doing ./bootstrap-prefix.sh again) and gcc-8.2.0-r4 emerged correctly. Hmmm I have no idea what is going on. In summary if I just try to emerge again gcc-8.2.0-r4 after the failed emerge it emerges correctly the second time. But the first time is consistently failing.
I ran the bootstrap of Gentoo Prefix on a 16 core machine with 12GB RAM with -j8 this time (the previous runs were run on a 2 core machine 7GB RAM with -j2) and... this error didn't happen.
Did you have a dmesg output by chance? I wonder if it was an OOM.
I don't have a dmesg output unfortunately. But I'll send the job again adding a dmesg command.
Created attachment 556550 [details] dmesg command just after error
The dmesg output shows on OOM error. The only meaningful error is: 2018-11-28T00:47:42.2545234Z [ 6043.603811] Invalid argument reading file caps for /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld Which doesn't help much.
(In reply to Sammy Pfeiffer from comment #9) > The dmesg output shows on OOM error. The only meaningful error is: > > 2018-11-28T00:47:42.2545234Z [ 6043.603811] Invalid argument reading file > caps for /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld > > Which doesn't help much. The error comes from kernel: https://github.com/torvalds/linux/blob/638820d8da8ededd6dc609beaef02d5396599c03/security/commoncap.c#L679 rc = get_vfs_caps_from_disk(bprm->file->f_path.dentry, &vcaps); if (rc < 0) { if (rc == -EINVAL) printk(KERN_NOTICE "Invalid argument reading file caps for %s\n", bprm->filename); https://github.com/torvalds/linux/blob/638820d8da8ededd6dc609beaef02d5396599c03/security/commoncap.c#L578 int get_vfs_caps_from_disk(const struct dentry *dentry, struct cpu_vfs_cap_data *cpu_caps) { .... Kernel blocks ld execution because it failed to read ld's caps from extended attributes (-EINVAL status). Here we have a bunch of possible places that return -EINVAL. If the failure is indeed memory-related it should be ENOMEM. As you use host debian kernel I suggest reporting the error to Debian. It will likely not solve your problem but should yield clearer error message about memory exhaustion (or something else).
(In reply to Sergei Trofimovich from comment #10) > As you use host debian kernel I suggest reporting the error to Debian. It > will likely not solve your problem but should yield clearer error message > about memory exhaustion (or something else). The kernel of the machine where this is running is (first dmesg line): Linux version 4.15.0-1023-azure (buildd@lcy01-amd64-025) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #24~16.04.1-Ubuntu SMP Wed Aug 29 12:54:36 UTC 2018 (Ubuntu 4.15.0-1023.24~16.04.1-azure 4.15.18) I found this bug report from Ubuntu that "I think" reports a similar error: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1736808 Which they seem to report that it's fixed in a posterior kernel version. I lack the skills to follow this error deeper. I guess there is nothing we can do, it's probably an OOM error but reported in a weird way. I'll give a go on running a job with -j1 to see if that would help.
(In reply to Sammy Pfeiffer from comment #11) > I guess there is nothing we can do, it's probably an OOM error but reported > in a weird way. I'll give a go on running a job with -j1 to see if that > would help. Doing -j1 yielded the same results unfortunately.
The bug you linked is not about OOM condition specifically but about invalid entries after directory move. My guess is that when you don't have a lot of RAM kernel evicts already cached ld inode from RAM and has to reload it. But due to a host kernel bug reload from disk fails with -EINVAL. Looking at the ubuntu bug you linked it's not yet backported anywhere and is planned only for next ubuntu release (if I read status correctly). I'm afraid Gentoo can't do much about it. You might want to ask ubuntu devs to reconsider backporting a fix to stable kernels.
