building firefox 65.x yields: * ERROR: www-client/firefox-65.0.2::gentoo failed (install phase): * !!! newins: /var/tmp/portage/www-client/firefox-65.0.2/distdir/firefox-65.0.2-en-GB.xpi does not exist ls -l /var/tmp/portage/www-client/firefox-65.0.2/distdir/firefox-65.0.2-en-GB.xpi lrwxrwxrwx 1 root root 47 Mar 1 15:55 /var/tmp/portage/www-client/firefox-65.0.2/distdir/firefox-65.0.2-en-GB.xpi -> /usr/portage/distfiles/firefox-65.0.2-en-GB.xpi gentoo64 ~ # ls -l /usr/portage/distfiles/firefox-65.0.2-en-GB.xpi lrwxrwxrwx 1 root root 54 Mar 1 15:55 /usr/portage/distfiles/firefox-65.0.2-en-GB.xpi -> /net/devsrv/portage/distfiles/firefox-65.0.2-en-GB.xpi This is because /usr/portage/distfiles/firefox-65.0.2-en-GB.xpi is a symlink to /net/devsrv/portage/distfiles/firefox-65.0.2-en-GB.xpi which is NFS mounted. if I replace /usr/portage/distfiles/firefox-65.0.2-en-GB.xpi symlink with a copy of firefox-65.0.2-en-GB.xpi it works fine.
The relevant bash code in newins looks like this: > if [[ ! -e $1 ]] ; then > __helpers_die "!!! ${helper}: $1 does not exist" > exit 1 > fi
(In reply to Zac Medico from comment #1) > The relevant bash code in newins looks like this: > > > if [[ ! -e $1 ]] ; then > > __helpers_die "!!! ${helper}: $1 does not exist" > > exit 1 > > fi hmm, does not tell me much ? You think it is correct?
(In reply to Joakim Tjernlund from comment #2) > (In reply to Zac Medico from comment #1) > > The relevant bash code in newins looks like this: > > > > > if [[ ! -e $1 ]] ; then > > > __helpers_die "!!! ${helper}: $1 does not exist" > > > exit 1 > > > fi > > hmm, does not tell me much ? You think it is correct? Then link is pointing outside distdir so maybe a sandbox issue?
Does the "portage" user have permission to read/execute the /net/devsrv/portage/distfiles directory?
(In reply to Mike Gilbert from comment #4) > Does the "portage" user have permission to read/execute the > /net/devsrv/portage/distfiles directory? I looks that its does. Both myself and root can for sure
(In reply to Joakim Tjernlund from comment #5) You can test it by running the following command as root. > setpriv --reuid portage --regid portage --init-groups stat /net/devsrv/portage/distfiles/firefox-65.0.2-en-GB.xpi
(In reply to Mike Gilbert from comment #6) > (In reply to Joakim Tjernlund from comment #5) > > You can test it by running the following command as root. > > > setpriv --reuid portage --regid portage --init-groups stat /net/devsrv/portage/distfiles/firefox-65.0.2-en-GB.xpi setpriv --reuid portage --regid portage --init-groups stat /net/devsrv/portage/distfiles/firefox-65.0.2-en-GB.xpi File: /net/devsrv/portage/distfiles/firefox-65.0.2-en-GB.xpi Size: 441661 Blocks: 864 IO Block: 1048576 regular file Device: 36h/54d Inode: 1618127228 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-03-01 15:55:29.332424540 +0100 Modify: 2019-02-28 16:25:58.000000000 +0100 Change: 2019-03-01 06:36:44.671173122 +0100 Birth: - Nice tool :)
Ok, I'm stumped.
(In reply to Mike Gilbert from comment #8) > Ok, I'm stumped. All firefox files in distfiles are symlinks and accessing them during build/unpacking works. It is only the install phase that fails accessing them so I think portage/sandbox is the problem.
Is this an autofs mount point, by chance?
(In reply to Anthony Parsons from comment #10) > Is this an autofs mount point, by chance? Yes, it is an autofs mount point.
(In reply to Joakim Tjernlund from comment #11) > (In reply to Anthony Parsons from comment #10) > > Is this an autofs mount point, by chance? > > Yes, it is an autofs mount point. Since building firefox takes a long time, I am pretty sure the directory has been unmounted by the time it is about to install.
(In reply to Joakim Tjernlund from comment #12) > (In reply to Joakim Tjernlund from comment #11) > > (In reply to Anthony Parsons from comment #10) > > > Is this an autofs mount point, by chance? > > > > Yes, it is an autofs mount point. > > Since building firefox takes a long time, I am pretty sure the > directory has been unmounted by the time it is about to install. I've been having similar problems with autofs as of late, with multiple packages. It's preventable by setting autofs.conf timeout=0 (infinite), but that seems more like a hack than a fix. It should be remounting automatically on access but something about the way portage interacts with the filesystem is causing that to not happen.
This works: gentoo64 ~ # umount /net/devsrv/portage gentoo64 ~ # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 41922560 25495844 16426716 61% / devtmpfs 10240 0 10240 0% /dev tmpfs 1640020 2752 1637268 1% /run cgroup_root 10240 0 10240 0% /sys/fs/cgroup shm 8200088 168 8199920 1% /dev/shm /dev/sda1 122835 27554 86107 25% /boot /dev/mapper/lvm-home 14632960 98376 14534584 1% /home /dev/mapper/lvm-src 21960216 1061804 20898412 5% /usr/local/src devsrv:/mnt/images 524032000 354109440 169922560 68% /mnt/images gentoo64 ~ # if [[ ! -e /usr/portage/distfiles/firefox-65.0.2.source.tar.xz ]] ; then echo error; fi No error is printed.
(In reply to Joakim Tjernlund from comment #14) > This works: > gentoo64 ~ # umount /net/devsrv/portage > gentoo64 ~ # df > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/root 41922560 25495844 16426716 61% / > devtmpfs 10240 0 10240 0% /dev > tmpfs 1640020 2752 1637268 1% /run > cgroup_root 10240 0 10240 0% /sys/fs/cgroup > shm 8200088 168 8199920 1% /dev/shm > /dev/sda1 122835 27554 86107 25% /boot > /dev/mapper/lvm-home 14632960 98376 14534584 1% /home > /dev/mapper/lvm-src 21960216 1061804 20898412 5% /usr/local/src > devsrv:/mnt/images 524032000 354109440 169922560 68% /mnt/images > gentoo64 ~ # if [[ ! -e /usr/portage/distfiles/firefox-65.0.2.source.tar.xz > ]] ; then echo error; fi > > No error is printed. and df now shows: Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 41922560 25499912 16422648 61% / devtmpfs 10240 0 10240 0% /dev tmpfs 1640020 2760 1637260 1% /run cgroup_root 10240 0 10240 0% /sys/fs/cgroup shm 8200088 168 8199920 1% /dev/shm /dev/sda1 122835 27554 86107 25% /boot /dev/mapper/lvm-home 14632960 98376 14534584 1% /home /dev/mapper/lvm-src 21960216 1061804 20898412 5% /usr/local/src devsrv:/mnt/images 524032000 354109440 169922560 68% /mnt/images devsrv.infinera.com:/portage 419225600 297070592 122155008 71% /net/devsrv/portage
I just got bitten by this bug again, each time I need to build firefox this happens.
Created attachment 578634 [details] sample test ebuild that illustrates attempts to access autofs-mounted NFS filesystems The bug does not only appear in distfiles/ access by the firefox ebuild. Our site uses an overlay with purely local ebuilds that rely on such automounted filesystems, and those ebuilds started breaking with a recent portage upgrade. The attached ebuild provides a test case that illustrates the problem. Two emerge attempts, one after the other, will demonstrate the effect. The first emerge (or "ebuild test-stuff-1.0.ebuild install"), done when the filesystem is not mounted, fails. Even though the ebuild triggers a filesystem automount, the filesystem's contents are not visible to the ebuild even though they are visible outside the ebuild environment at the same time (to another user logged in, for example). It does not seem to be a matter of accessing too soon, either, as the attached sample tries again with a small delay. Oh, and we get these two log messages: 2019-05-29T13:27:22-04:00 automount[7590]: do_mount_indirect: indirect trigger not valid or already mounted /rapp/linux 2019-05-29T13:27:22-04:00 automount[7590]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument After the first emerge fails, however, we immediately try again. This time, since the filesystem is already mounted, the test-stuff ebuild sees the filesystem's contents with no problem at all. For what it's worth, our sample auto.master file is just the single line: /rapp file:/etc/autofs/test.map And the test.map file is similarly one line: linux gallifrey:/vol/lima/app/linux Granted, our practice of accessing NFS-automounted directories from within our local ebuilds is unusual and would not be appropriate for the regular Gentoo repository. But the original bug report's case of shared automounted distfiles/ directories is, I suspect, much more widespread. For that case, you can probably expect a great many emerge failures that will seem mysterious and intermittent. I suppose I should include an "emerge --info" at this point? Portage 2.3.62 (python 2.7.15-final-0, default/linux/amd64/17.0, gcc-8.2.0, glibc-2.28-r5, 4.14.105-gentoo-r1 x86_64) ================================================================= System uname: Linux-4.14.105-gentoo-r1-x86_64-Intel-R-_Core-TM-_i7_CPU_960_@_3.20GHz-with-gentoo-2.6 KiB Mem: 12294816 total, 11424392 free KiB Swap: 16777208 total, 16777208 free Timestamp of repository gentoo: Mon, 08 Apr 2019 00:45:01 +0000 Head commit of repository gentoo: 6a4be948970dd6af01dd9de2c3b7e7ab011c8a94 Head commit of repository acd-local: d5882b333328eb7087cd0d5a4e28c7e0364e5aa0 sh bash 4.4_p23-r1 ld GNU ld (Gentoo 2.31.1 p5) 2.31.1 app-shells/bash: 4.4_p23-r1::gentoo dev-lang/perl: 5.26.2::gentoo dev-lang/python: 2.7.15::gentoo, 3.6.5::gentoo dev-util/cmake: 3.9.6::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.38.3-r1::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.16.1-r1::gentoo sys-devel/binutils: 2.31.1-r4::gentoo sys-devel/gcc: 8.2.0-r6::gentoo sys-devel/gcc-config: 2.0::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.28-r5::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://gallifrey.<elided> priority: -1000 sync-rsync-verify-jobs: 1 sync-rsync-verify-metamanifest: yes sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 acd-local location: /usr/local/portage/acd-local sync-type: git sync-uri: <elided>gallifrey.<elided> masters: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA dlj-1.1 Broadcom Oracle-BCLA-JavaSE googleearth" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/revisions /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://gallifrey.<elided> http://distfiles.gentoo.org http://www.ibiblio.org/pub/Linux/distributions/gentoo" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en_US" MAKEOPTS="-j9 -l8" PKGDIR="/usr/portage/packages" PORTAGE_BINHOST="http://gallifrey<elided>" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac aalib accessibility acl acpi alsa amd64 audit bash-completion berkdb bidi bzip2 cairo caps cdparanoia cdr cli cracklib crypt cups curl cxx dbus dri dvd dvdr dvdread eds emacs encode fam ffmpeg fontconfig fortran gd gdbm gif glib gmp gnome gnome-keyring gpm graphviz gs gstreamer gtk iconv idn imagemagick imlib ipv6 jbig jpeg kde lame lcms libtirpc lzma lzo mad mbox mng motif mp3 mpeg multilib nautilus ncurses nfs nls nptl ogg opengl openmp oss pam pcre pdf perl png ppds python qt3support qt4 quicktime readline sasl sdl seccomp semantic-desktop session spell spl ssl svg szip tcl tcpd theora tiff tk truetype unicode v4l v4l2 vhosts vorbis win32codecs wmf wxwindows x264 xattr xinerama xml xpm xscreensaver xulrunner xv zlib" ABI_X86="64 32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="emu efi-32 efi-64 pc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" L10N="en-US" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23 ruby24" USERLAND="GNU" VIDEO_CARDS="fbdev v4l vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
This could be an issue with mount namespaces. Are you able to reproduce the problem with FEATURES="-mount-sandbox -pid-sandbox"?
I actually had this same problem while building GCC, my distfiles also being an automounted NFS share. I assumed it is an autofs problem though. For long builds the share is simply unmounted, as comment #c12 https://bugs.gentoo.org/687090
Yes, doing the emerge with FEATURES="-mount-sandbox -pid-sandbox" makes the problem go away. If I understand this correctly, I suppose it's not surprising that the sandbox isolation would work both ways: not only are mounts in the ebuild environment not visible to the outside world, but mounts/umounts in the outside world are not visible to the ebuild either. Even if they are autofs mounts triggered within the ebuild.
That's probably the gist of it, though I am not familiar enough with autofs to discuss specifics. I do suspect that this might work better on systems where mounts are shared between namespaces by default, as would be configured by systemd.
FWIW, I can live with the workaround as it is because I want to share the distfiles with all (dev) machines in use. Perhaps if Portage could warn about distfiles being automounted being a problem this would help other people in the future to resolve this problem?
Here's some discussion of autofs in relation to mount namespaces. https://www.spinics.net/lists/autofs/msg01694.html
