bug #69763 describes a setup in wich /usr/portage/distfiles is a NFS share mounted read only. This kind of setup is required for me as im trying to implement secure "thin" clients where /usr/portage and the distfiles must reside on remote NFS server. Using a special FETHCOMMAND I'm a able to place files there ( the nfs server does the downloading and the write to the distfiles ) In the NFS client portage checks /usr/portatge/distfiles for being writable, and then fails. According to the activity in #69763 the bug was fix in >=portage-2.0.51.20, however I'm using 2.1.4.4 and still getting the same behavior. Portage does not even try to run my FETCHCOMMAND here is my emerge --info Portage 2.1.4.4 (default-linux/x86/2007.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.24-gentoo-r4 i686) ================================================================= System uname: 2.6.24-gentoo-r4 i686 Intel(R) Pentium(R) M processor 1.73GHz Timestamp of tree: Mon, 05 May 2008 23:15:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] app-shells/bash: 3.2_p17-r1 dev-java/java-config: 1.3.7, 2.1.4 dev-lang/python: 2.4.4-r9 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.61-r1 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.18-r1 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=pentium-m -O2 -m32 -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-O2 -mcpu=i686 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--nospinner" FEATURES="distlocks metadata-transfer sandbox sfperms skiprotest strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" LINGUAS="he en" MAKEOPTS="-j4" PKGDIR="/usr/portage//packages/Pm" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage/" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://mirror.hamakor.org.il/gentoo-portage" USE="X acl acpi alsa apci avahi avi bash-completion berkdb bidi bluetooth bzip2 cdparanoia cdr cli cracklib crypt cups divx4linux dri dvd dvdr encode exif fam fftw flash font-server fortran gdbm gif gpm gtk2 iconv ieee1394 innodb isdnlog jpeg jpeg2k kde maildir midi mime mmx mozilla mozsvg mp3 mpeg mudflap ncurses network-cron nls nptl nptlonly ogg oggvorbis opengl openmp pam pcre perl png pppd python qt3support qt4 quicktime radius readline reflection session shared sharedmem shorten skype spell spl sse sse2 ssl svg tcpd tiff truetype unicode usb utf8 vhosts wmf x86 xine xinerama xml xml2 xorg zip zlib" ALSA_CARDS="snd-intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="ptp ptp2 canon casio kodak template directory" ELIBC="glibc" INPUT_DEVICES="mouse keyboard synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="he en" LIRC_DEVICES="sir" USERLAND="GNU" VIDEO_CARDS="i810 i915" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS Reproducible: Always Steps to Reproduce: 1. mount nfsserver:/usr/portage /usr/portage -o ro 2. emerge -f mozilla-firefox Actual Results: Calculating dependencies ... done! >>> Emerging (1 of 1) www-client/mozilla-firefox-2.0.0.14 to / !!! No write access to '/usr/portage/distfiles' !!! File mozilla-firefox-2.0.0.14-patches-0.1.tar.bz2 isn't fetched but unable to get it. !!! Fetch for /usr/portage/www-client/mozilla-firefox/mozilla-firefox-2.0.0.14.ebuild failed, continuing... !!! Some fetch errors were encountered. Please see above for details. www-client/mozilla-firefox-2.0.0.14 Expected Results: Invoke my FETCHCOMMAND and wait for result, the file will be in /usr/portage/distfiles.
It's undocumented, but if you set FEATURES="skiprocheck -distlocks" in make.conf then it should work. We should document this in the make.conf.5 man page.
This is fixed in 2.1.5.
According to bug 175612, comment #12, this feature was broken by this commit: https://gitweb.gentoo.org/proj/portage.git/commit/?id=ebbde237d33e783c562cc6c70987969ac7228b96 commit ebbde237d33e783c562cc6c70987969ac7228b96 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-04-27 21:59:57 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-04-30 02:28:21 +0000 fetch: atomic downloads (bug 175612) Direct FETCHCOMMAND/RESUMECOMMAND output to a temporary file with a constant .__download__ suffix, and atomically rename the file to remove the suffix only after the download has completed successfully (includes digest verification when applicable). Also add unit tests to cover most fetch cases. Bug: https://bugs.gentoo.org/175612 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/BinpkgVerifier.py | 4 +- lib/portage/package/ebuild/fetch.py | 105 ++++++++++----- lib/portage/tests/ebuild/test_fetch.py | 230 +++++++++++++++++++++++++++++++++ 3 files changed, 303 insertions(+), 36 deletions(-)
I suppose we could make FEATURES="skiprocheck" disable the atomic download behavior that solves bug 175612.
(In reply to Zac Medico from comment #4) > I suppose we could make FEATURES="skiprocheck" disable the atomic download > behavior that solves bug 175612. Or introduce a RENAMECOMMAND that defaults to "mv \"\${DISTDIR}/\${TMPFILE}\" \"\${DISTDIR}/\${FILE}\"" but that can be overridden in make.conf to allow performing the renames on a remote host. If you do overload the "skiprocheck" feature, you ought to rename it, perhaps to "rodistdir". Alternatively you could introduce a new feature, "atomicfetch", enabled by default, that could be disabled to suppress the atomic behavior.
(In reply to Matt Whitlock from comment #5) > (In reply to Zac Medico from comment #4) > > I suppose we could make FEATURES="skiprocheck" disable the atomic download > > behavior that solves bug 175612. > > Or introduce a RENAMECOMMAND that defaults to "mv > \"\${DISTDIR}/\${TMPFILE}\" \"\${DISTDIR}/\${FILE}\"" but that can be > overridden in make.conf to allow performing the renames on a remote host. We'd also need a DELETECOMMAND since there are cases where we want to delete invalid files. Looking at the code, I see a couple of unlink calls where failures are silently allowed. Also, the _checksum_failure_temp_file function uses mkstemp, so it would be useful to have a command for creating temp files. Also, we currently ignore errors for chown/chmod calls, so maybe we could avoid that somehow. > If you do overload the "skiprocheck" feature, you ought to rename it, > perhaps to "rodistdir". Alternatively you could introduce a new feature, > "atomicfetch", enabled by default, that could be disabled to suppress the > atomic behavior. Is there any reason to disable the atomic fetch behavior, aside from the skiprocheck feature though?
Here are some minimal fixes for the old skiprocheck behavior: https://archives.gentoo.org/gentoo-portage-dev/message/57f93a88dff0d2a179916f96f22fc0bd https://github.com/gentoo/portage/pull/467
(In reply to Zac Medico from comment #6) > Is there any reason to disable the atomic fetch behavior, aside from the > skiprocheck feature though? Not that I know of. I'd only want to disable it because it breaks the read-only DISTDIR use case.
We can make it possible to pass digests and other information to FETCHCOMMAND, so that it can perform digest verification and atomic rename itself.
The changes for bug 646898 are now an issue since mirror file layout data is cached in ${DISTDIR}/.mirror-cache.json.
Also bug 697566 complicates things if we use FETCHCOMMAND to fetch mirror layout.conf which don't really want to save in DISTDIR.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=6b5889afb1e80bc673ce782e65fc0f49ee7d0908 commit 6b5889afb1e80bc673ce782e65fc0f49ee7d0908 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-13 22:13:18 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-14 19:46:16 +0000 fetch: minimal skiprocheck fixes (bug 220533) Fix cases here fetch assumes that DISTDIR is writable when it's actually read-only. This preserves old behavior which allowed users to override FETCHCOMMAND to fetch files on a remote system, even though DISTDIR is locally mounted in read-only mode. Bug: https://bugs.gentoo.org/220533 Fixes: ebbde237d33e ("fetch: atomic downloads (bug 175612)") Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/fetch.py | 38 ++++++++++++++++++++-------------- lib/portage/tests/ebuild/test_fetch.py | 22 ++++++++++++++++++++ 2 files changed, 45 insertions(+), 15 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e18292c54af86bee8299b6a925d25eb6c55fd77b commit e18292c54af86bee8299b6a925d25eb6c55fd77b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-14 22:53:35 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-14 23:01:06 +0000 sys-apps/portage: Bump to version 2.3.77 #220533 Fix FEATURES=skiprocheck read-only DISTDIR support #646898 Support GLEP 75 mirror structure #658648 Disable emerge --autounmask by default, except for package.use and package.license changes #695870 Improvements to ebuild(5) man page Bug: https://bugs.gentoo.org/697734 Bug: https://bugs.gentoo.org/220533 Bug: https://bugs.gentoo.org/646898 Bug: https://bugs.gentoo.org/658648 Bug: https://bugs.gentoo.org/695870 Package-Manager: Portage-2.3.77, Repoman-2.3.17 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.77.ebuild | 261 +++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+)