Only observed 2 packages failing the install phase (while hundreds work fine on hundreds of machines): net-nds/openldap-2.4.48 sys-kernel/genkernel-4.0.2 Unmounting distifiles or having portage without native-extensions makes it working.
Created attachment 603434 [details] genkernel-build.log
Created attachment 603436 [details] openldap-build.log
All files can be read normally meaning this works fine: cat /usr/portage/distfiles/* | md5sum # md5sum /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/* cb40943d2a2cb8ce08d42bc48b0f84f0 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/boost_1_72_0.tar.bz2 4fb3ac57dc4e0afcf723cdf34640179b /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/btrfs-progs-v5.4.1.tar.xz 70913edaf2263a157393af07565c17f0 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/busybox-1.31.1.tar.bz2 2b64ca9ea16e823df359eacf9c02414b /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/cryptsetup-2.2.2.tar.xz 819338fcef98e8e25819f0516722beeb /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/dmraid-1.0.0.rc16-3.tar.bz2 a972c85ed678ad0fdcb7844e1294fb54 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/dropbear-2019.78.tar.bz2 c3776140122465569e58cff55ec2bbf7 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/e2fsprogs-1.45.5.tar.xz cdf54239f892fc7914957f10de1e1c70 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/expat-2.2.8.tar.xz 8000410aadc9231fd48495f7642f3312 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/fuse-2.9.9.tar.gz 92442848df4253da07f3a7acb176ca70 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/genkernel-4.0.2.tar.xz b1df02c73572f27bc859ac05ff2259ab /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/gnupg-1.4.23.tar.bz2 04969ad59cc37bddd83741a08b98f350 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/json-c-0.13.1.tar.gz 1129c243199bdd7db01b55a61aa19601 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/kmod-26.tar.xz 1ba264947d05bd8e0f9dc9a9ed80cf3e /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/libaio-0.3.112.tar.gz 348cc4601ca34307fc6cd6c945467743 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/libgcrypt-1.8.5.tar.bz2 eff437f397e858a9127b76c0d87fa5ed /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/libgpg-error-1.36.tar.bz2 73e24436171f3022c5e80452295ac792 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/LVM2.2.02.186.tgz 39d3f3f9c55c87b1e5d6888e1420f4b5 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/lzo-2.10.tar.gz 51bf3651bd73a06c413a2f964f299598 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/mdadm-4.1.tar.xz abb1b3c3eacbadce062fc31cc4b5f9d0 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/open-iscsi-2.0.878.tar.gz 3743beefa3dd6247a73f8f7a32c14c33 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/popt-1.16.tar.gz b2b58f05eb3c5c0bf9d1e26003b4d698 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/strace-5.4.tar.xz 4ff949e9eef65aaee68d6d923ba0862f /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/thin-provisioning-tools-0.8.5.tar.gz 40411d156ea7fa0e7cd0a8ec6fe60e70 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/unionfs-fuse-2.0.tar.gz a78cbeaed9c39094b96a48ba8f891d50 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/util-linux-2.34.tar.xz 61232b1cc453780517d9b0c12ff1699b /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/xfsprogs-5.4.0.tar.xz 1c9f62f0778697a09d36121ead88e08e /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/zlib-1.2.11.tar.gz 532aa7b3a873e144bbbedd9c0ea87694 /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir/zstd-1.4.4.tar.gz
Only tested on amd64 systems, kernels 4.14/5.4. Portage 2.3.84 (python 3.6.9-final-0, default/linux/amd64/17.1, gcc-9.2.0, glibc-2.30-r3, 5.4.12-gentoo x86_64) ================================================================= System uname: Linux-5.4.12-gentoo-x86_64-Intel-R-_Core-TM-_i5-3570_CPU_@_3.40GHz-with-gentoo-2.6 KiB Mem: 16290656 total, 6456804 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Wed, 15 Jan 2020 00:15:01 +0000 Head commit of repository gentoo: 2cc2ca954f4637a26f14b12a06ad20cb921be337 sh bash 5.0_p11 ld GNU ld (Gentoo 2.32 p2) 2.32.0 app-shells/bash: 5.0_p11::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.17::gentoo, 3.5.9::gentoo, 3.6.9::gentoo, 3.7.5-r1::gentoo, 3.8.0::gentoo dev-util/cmake: 3.16.0::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.18::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r5::gentoo sys-devel/automake: 1.13.4-r2::gentoo, 1.15.1-r2::gentoo, 1.16.1-r2::gentoo sys-devel/binutils: 2.32-r1::gentoo, 2.33.1::gentoo sys-devel/gcc: 9.2.0-r2::gentoo sys-devel/gcc-config: 2.1::gentoo sys-devel/libtool: 2.4.6-r5::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r3::gentoo ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-mtune=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.2/ext-active/ /etc/php/apache2-php7.3/ext-active/ /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cgi-php7.3/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/php/cli-php7.3/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-mtune=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://tux.rainside.sk/gentoo/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="acl amd64 berkdb bzip2 cli crypt cxx dri fortran gdbm iconv ipv6 libtirpc multilib ncurses nptl openmp pam pcre readline seccomp split-usr ssl tcpd unicode xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_core authn_dbm authn_file authz_core authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias watchdog proxy proxy_connect proxy_fcgi proxy_http" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 aes avx f16c popcnt sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 xen pc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" NGINX_MODULES_HTTP="access charset gzip limit_conn limit_req log proxy rewrite gunzip upstream_check stub_status geoip2" NGINX_MODULES_STREAM="geoip2" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6 python3_7 python3_8" RUBY_TARGETS="ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="intel i915 i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Please attach glusterfs volume settings. Would be interesting to know if using strict-readdir makes a difference, which I/O mode is being used...
Seems like strict-readdir isn't present in glusterfs 6.7. It's a default volume, nothing special configured.
Created attachment 603446 [details] distfiles-info.txt
Created attachment 603448 [details] distfiles-get.txt
Please strace it in order to check which syscall fails with errno 77, for example see bug 635002, comment #2. We might want to make it fall back to a regular copy in this case.
Created attachment 603572 [details] strace.txt emerge -v1 sys-kernel/genkernel ... fails ... cd /var/tmp/portage/sys-kernel/genkernel-4.0.2/distdir strace python -c 'from portage.util.file_copy import copyfile; copyfile("boost_1_72_0.tar.bz2", "/tmp/boost_1_72_0.tar.bz2")' ... openat(AT_FDCWD, "boost_1_72_0.tar.bz2", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0664, st_size=106907358, ...}) = 0 ioctl(3, TCGETS, 0x7ffff6588fa0) = -1 ENOSYS (Function not implemented) openat(AT_FDCWD, "/tmp/boost_1_72_0.tar.bz2", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 ioctl(4, TCGETS, 0x7ffff6588fa0) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_DATA) = -1 EBADFD (File descriptor in bad state) close(4) = 0 close(3) = 0 write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last): ) = 35 write(2, " File \"<string>\", line 1, in <m"..., 39 File "<string>", line 1, in <module> ) = 39 openat(AT_FDCWD, "<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib64/python36.zip/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib64/python3.6/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib64/python3.6/lib-dynload/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/lib64/python3.6/site-packages/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) write(2, " File \"/usr/lib64/python3.6/sit"..., 112 File "/usr/lib64/python3.6/site-packages/portage/util/file_copy/__init__.py", line 30, in _optimized_copyfile ) = 112 openat(AT_FDCWD, "/usr/lib64/python3.6/site-packages/portage/util/file_copy/__init__.py", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=871, ...}) = 0 ioctl(3, TCGETS, 0x7ffff6588040) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_CUR) = 0 fcntl(3, F_DUPFD_CLOEXEC, 0) = 4 fcntl(4, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fstat(4, {st_mode=S_IFREG|0644, st_size=871, ...}) = 0 read(4, "# Copyright 2017 Gentoo Foundati"..., 4096) = 871 close(4) = 0 lseek(3, 0, SEEK_SET) = 0 lseek(3, 0, SEEK_CUR) = 0 read(3, "# Copyright 2017 Gentoo Foundati"..., 8192) = 871 close(3) = 0 write(2, " _file_copy(src_file.fileno()"..., 53 _file_copy(src_file.fileno(), dst_file.fileno()) ) = 53 write(2, "OSError: [Errno 77] File descrip"..., 49OSError: [Errno 77] File descriptor in bad state ) = 49 rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fe27a0ec100}, {sa_handler=0x7fe27a419f00, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fe27a0ec100}, 8) = 0 sigaltstack(NULL, {ss_sp=0x5563ff011300, ss_flags=0, ss_size=8192}) = 0 sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}, NULL) = 0 exit_group(1) = ? +++ exited with 1 +++
Created attachment 603634 [details, diff] file_copy: handle lseek SEEK_DATA EBADFD for GlusterFS Please test this patch, which you can apply by dropping into /etc/portage/patches/sys-apps/portage/ and rebuilding portage. NOTE: This change will trigger a fallback to sendfile, and if that fails with something other than EINVAL then we'll have to update the sendfile error handling as well. If it still fails with this patch then please strace it again.
(In reply to Zac Medico from comment #11) > Created attachment 603634 [details, diff] [details, diff] > file_copy: handle lseek SEEK_DATA EBADFD for GlusterFS > > Please test this patch, which you can apply by dropping into > /etc/portage/patches/sys-apps/portage/ and rebuilding portage. > > NOTE: This change will trigger a fallback to sendfile, and if that fails > with something other than EINVAL then we'll have to update the sendfile > error handling as well. > > If it still fails with this patch then please strace it again. Works with portage 2.3.84-r1, thank you.
Thanks for testing! Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/da441b4e97c6de496e6ec14a0dd8a475 https://github.com/gentoo/portage/pull/498
Supposed to be fixed by: https://review.gluster.org/#/c/glusterfs/+/23530/ https://github.com/gluster/glusterfs/commit/9dd57ff963d4d0dde7f3ad0ff230837ff2f4b1df
(In reply to Zac Medico from comment #14) > Supposed to be fixed by: > > https://review.gluster.org/#/c/glusterfs/+/23530/ > https://github.com/gluster/glusterfs/commit/ > 9dd57ff963d4d0dde7f3ad0ff230837ff2f4b1df Yes, it fixes the issue. Thanks for the help Zac!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2a677e4cdb886a25f62f26059c4247b96c7545a4 commit 2a677e4cdb886a25f62f26059c4247b96c7545a4 Author: Tomáš Mózes <hydrapolic@gmail.com> AuthorDate: 2020-01-27 06:54:03 +0000 Commit: Alexys Jacob <ultrabug@gentoo.org> CommitDate: 2020-02-09 16:31:54 +0000 sys-cluster/glusterfs: fix seek Bug: https://bugs.gentoo.org/705536 Package-Manager: Portage-2.3.85, Repoman-2.3.20 Signed-off-by: Tomáš Mózes <hydrapolic@gmail.com> Closes: https://github.com/gentoo/gentoo/pull/14468 Signed-off-by: Alexys Jacob <ultrabug@gentoo.org> .../glusterfs/files/glusterfs-6.7-fix-seek.patch | 45 ++++++++++++++++++++++ ...lusterfs-6.7.ebuild => glusterfs-6.7-r1.ebuild} | 4 ++ ...lusterfs-7.2.ebuild => glusterfs-7.2-r1.ebuild} | 4 ++ 3 files changed, 53 insertions(+)
Thanks, fix in the main repo.
