Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bugzilla DB migration completed. Please report issues to Infra team via email via infra@gentoo.org or IRC
Bug 220533 - Read only DISTDIR is not possible (FEATURES=skiprocheck)
Summary: Read only DISTDIR is not possible (FEATURES=skiprocheck)
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal major with 1 vote (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, REGRESSION
Depends on:
Blocks: 377365 697566 697734
  Show dependency tree
 
Reported: 2008-05-06 07:22 UTC by orishi
Modified: 2020-01-17 22:22 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description orishi 2008-05-06 07:22:14 UTC
bug #69763 describes a setup in wich /usr/portage/distfiles is a NFS share mounted read only. 
This kind of setup is required for me as im trying to implement secure "thin" clients where /usr/portage and the distfiles must reside on remote NFS server.
Using a special FETHCOMMAND I'm a able to place files there ( the nfs server does the downloading and the write to the distfiles )
In the NFS client portage checks /usr/portatge/distfiles for being writable, and then fails.
According to the activity in #69763 the bug was fix in >=portage-2.0.51.20, however I'm using 2.1.4.4 and still getting the same behavior. 
Portage does not even try to run my FETCHCOMMAND
here is my emerge --info
Portage 2.1.4.4 (default-linux/x86/2007.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.24-gentoo-r4 i686)
=================================================================
System uname: 2.6.24-gentoo-r4 i686 Intel(R) Pentium(R) M processor 1.73GHz
Timestamp of tree: Mon, 05 May 2008 23:15:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
app-shells/bash:     3.2_p17-r1
dev-java/java-config: 1.3.7, 2.1.4
dev-lang/python:     2.4.4-r9
dev-python/pycrypto: 2.0.1-r6
sys-apps/baselayout: 1.12.11.1
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.13, 2.61-r1
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.18-r1
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   1.5.26
virtual/os-headers:  2.6.23-r3
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=pentium-m -O2 -m32 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-O2 -mcpu=i686 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--nospinner"
FEATURES="distlocks metadata-transfer sandbox sfperms skiprotest strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"
LANG="en_US.UTF-8"
LC_ALL="en_US.UTF-8"
LINGUAS="he en"
MAKEOPTS="-j4"
PKGDIR="/usr/portage//packages/Pm"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage/"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://mirror.hamakor.org.il/gentoo-portage"
USE="X acl acpi alsa apci avahi avi bash-completion berkdb bidi bluetooth bzip2 cdparanoia cdr cli cracklib crypt cups divx4linux dri dvd dvdr encode exif fam fftw flash font-server fortran gdbm gif gpm gtk2 iconv ieee1394 innodb isdnlog jpeg jpeg2k kde maildir midi mime mmx mozilla mozsvg mp3 mpeg mudflap ncurses network-cron nls nptl nptlonly ogg oggvorbis opengl openmp pam pcre perl png pppd python qt3support qt4 quicktime radius readline reflection session shared sharedmem shorten skype spell spl sse sse2 ssl svg tcpd tiff truetype unicode usb utf8 vhosts wmf x86 xine xinerama xml xml2 xorg zip zlib" ALSA_CARDS="snd-intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="ptp ptp2 canon casio kodak template directory" ELIBC="glibc" INPUT_DEVICES="mouse keyboard synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="he en" LIRC_DEVICES="sir" USERLAND="GNU" VIDEO_CARDS="i810 i915"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS


Reproducible: Always

Steps to Reproduce:
1. mount nfsserver:/usr/portage /usr/portage -o ro
2. emerge -f mozilla-firefox
Actual Results:  
Calculating dependencies ... done!

>>> Emerging (1 of 1) www-client/mozilla-firefox-2.0.0.14 to /
!!! No write access to '/usr/portage/distfiles'
!!! File mozilla-firefox-2.0.0.14-patches-0.1.tar.bz2 isn't fetched but unable to get it.

!!! Fetch for /usr/portage/www-client/mozilla-firefox/mozilla-firefox-2.0.0.14.ebuild failed, continuing...



!!! Some fetch errors were encountered.  Please see above for details.

   www-client/mozilla-firefox-2.0.0.14


Expected Results:  
Invoke my FETCHCOMMAND and wait for result, the file will be in /usr/portage/distfiles.
Comment 1 Zac Medico gentoo-dev 2008-05-13 02:19:27 UTC
It's undocumented, but if you set FEATURES="skiprocheck -distlocks" in make.conf then it should work. We should document this in the make.conf.5 man page.
Comment 2 Zac Medico gentoo-dev 2008-05-14 21:38:12 UTC
This is fixed in 2.1.5.
Comment 3 Zac Medico gentoo-dev 2019-09-12 20:02:52 UTC
According to bug 175612, comment #12, this feature was broken by this commit:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ebbde237d33e783c562cc6c70987969ac7228b96

commit ebbde237d33e783c562cc6c70987969ac7228b96
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-04-27 21:59:57 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-04-30 02:28:21 +0000

    fetch: atomic downloads (bug 175612)
    
    Direct FETCHCOMMAND/RESUMECOMMAND output to a temporary file with
    a constant .__download__ suffix, and atomically rename the file
    to remove the suffix only after the download has completed
    successfully (includes digest verification when applicable).
    Also add unit tests to cover most fetch cases.
    
    Bug: https://bugs.gentoo.org/175612
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/BinpkgVerifier.py          |   4 +-
 lib/portage/package/ebuild/fetch.py    | 105 ++++++++++-----
 lib/portage/tests/ebuild/test_fetch.py | 230 +++++++++++++++++++++++++++++++++
 3 files changed, 303 insertions(+), 36 deletions(-)
Comment 4 Zac Medico gentoo-dev 2019-09-13 07:06:42 UTC
I suppose we could make FEATURES="skiprocheck" disable the atomic download behavior that solves bug 175612.
Comment 5 Matt Whitlock 2019-09-13 08:24:03 UTC
(In reply to Zac Medico from comment #4)
> I suppose we could make FEATURES="skiprocheck" disable the atomic download
> behavior that solves bug 175612.

Or introduce a RENAMECOMMAND that defaults to "mv \"\${DISTDIR}/\${TMPFILE}\" \"\${DISTDIR}/\${FILE}\"" but that can be overridden in make.conf to allow performing the renames on a remote host.

If you do overload the "skiprocheck" feature, you ought to rename it, perhaps to "rodistdir". Alternatively you could introduce a new feature, "atomicfetch", enabled by default, that could be disabled to suppress the atomic behavior.
Comment 6 Zac Medico gentoo-dev 2019-10-13 22:12:51 UTC
(In reply to Matt Whitlock from comment #5)
> (In reply to Zac Medico from comment #4)
> > I suppose we could make FEATURES="skiprocheck" disable the atomic download
> > behavior that solves bug 175612.
> 
> Or introduce a RENAMECOMMAND that defaults to "mv
> \"\${DISTDIR}/\${TMPFILE}\" \"\${DISTDIR}/\${FILE}\"" but that can be
> overridden in make.conf to allow performing the renames on a remote host.

We'd also need a DELETECOMMAND since there are cases where we want to delete invalid files. Looking at the code, I see a couple of unlink calls where failures are silently allowed.

Also, the _checksum_failure_temp_file function uses mkstemp, so it would be useful to have a command for creating temp files.

Also, we currently ignore errors for chown/chmod calls, so maybe we could avoid that somehow.

> If you do overload the "skiprocheck" feature, you ought to rename it,
> perhaps to "rodistdir". Alternatively you could introduce a new feature,
> "atomicfetch", enabled by default, that could be disabled to suppress the
> atomic behavior.

Is there any reason to disable the atomic fetch behavior, aside from the skiprocheck feature though?
Comment 7 Zac Medico gentoo-dev 2019-10-13 22:47:31 UTC
Here are some minimal fixes for the old skiprocheck behavior:

https://archives.gentoo.org/gentoo-portage-dev/message/57f93a88dff0d2a179916f96f22fc0bd
https://github.com/gentoo/portage/pull/467
Comment 8 Matt Whitlock 2019-10-14 01:12:04 UTC
(In reply to Zac Medico from comment #6)
> Is there any reason to disable the atomic fetch behavior, aside from the
> skiprocheck feature though?

Not that I know of. I'd only want to disable it because it breaks the read-only DISTDIR use case.
Comment 9 Zac Medico gentoo-dev 2019-10-14 06:27:53 UTC
We can make it possible to pass digests and other information to FETCHCOMMAND, so that it can perform digest verification and atomic rename itself.
Comment 10 Zac Medico gentoo-dev 2019-10-14 17:57:51 UTC
The changes for bug 646898 are now an issue since mirror file layout data is cached in ${DISTDIR}/.mirror-cache.json.
Comment 11 Zac Medico gentoo-dev 2019-10-14 19:53:05 UTC
Also bug 697566 complicates things if we use FETCHCOMMAND to fetch mirror layout.conf which don't really want to save in DISTDIR.
Comment 12 Larry the Git Cow gentoo-dev 2019-10-14 20:06:44 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6b5889afb1e80bc673ce782e65fc0f49ee7d0908

commit 6b5889afb1e80bc673ce782e65fc0f49ee7d0908
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-10-13 22:13:18 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-10-14 19:46:16 +0000

    fetch: minimal skiprocheck fixes (bug 220533)
    
    Fix cases here fetch assumes that DISTDIR is writable when it's actually
    read-only. This preserves old behavior which allowed users to override
    FETCHCOMMAND to fetch files on a remote system, even though DISTDIR is
    locally mounted in read-only mode.
    
    Bug: https://bugs.gentoo.org/220533
    Fixes: ebbde237d33e ("fetch: atomic downloads (bug 175612)")
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/package/ebuild/fetch.py    | 38 ++++++++++++++++++++--------------
 lib/portage/tests/ebuild/test_fetch.py | 22 ++++++++++++++++++++
 2 files changed, 45 insertions(+), 15 deletions(-)
Comment 13 Larry the Git Cow gentoo-dev 2019-10-14 23:01:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e18292c54af86bee8299b6a925d25eb6c55fd77b

commit e18292c54af86bee8299b6a925d25eb6c55fd77b
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-10-14 22:53:35 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-10-14 23:01:06 +0000

    sys-apps/portage: Bump to version 2.3.77
    
     #220533 Fix FEATURES=skiprocheck read-only DISTDIR support
     #646898 Support GLEP 75 mirror structure
     #658648 Disable emerge --autounmask by default, except for package.use
             and package.license changes
     #695870 Improvements to ebuild(5) man page
    
    Bug: https://bugs.gentoo.org/697734
    Bug: https://bugs.gentoo.org/220533
    Bug: https://bugs.gentoo.org/646898
    Bug: https://bugs.gentoo.org/658648
    Bug: https://bugs.gentoo.org/695870
    Package-Manager: Portage-2.3.77, Repoman-2.3.17
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.77.ebuild | 261 +++++++++++++++++++++++++++++++++
 2 files changed, 262 insertions(+)