Bug 69763 – read only /usr/portage does not imply files cannot be fetched

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Jaco Kroon 2004-11-01 13:18:38 UTC]

It seems that portage believes that if /usr/portage is read-only then it is impossible to fetch files into /usr/portage/distfiles.  This is not the case, take the following setup:

nfs server called pug that exports /usr/portage ro to whoever wants it.
nfs client called slowcoach that mounts pug:/usr/portage onto /usr/portage

now, in my setup it is possible for slowcoach to ask pug to fetch a package that it wants, specifically I'm using torpage (http://www.kroon.co.za/torpage.php) to achieve this.  Another method would also be to make use of certificates to allow slowcoach to ssh to pug to issue the wget on pug itself instead of on slowcoach.

Reproducible: Always
Steps to Reproduce:
1.  Make sure that /usr/portage is ro
2.  Ensure that FETCHCOMMAND has some way of actually getting files onto this ro file system
3.  Attemp to merge a package for which the distfiles is not yet available.

Actual Results:  
Portage fails with an error stating that /usr/portage/distfiles is ro:

slowcoach root # emerge -f minicom
Calculating dependencies ...done!
>>> emerge (1 of 1) net-dialup/minicom-2.1-r1 to /
!!! No write access to /usr/portage/distfiles/
!!! File minicom-2.1.tar.gz isn't fetched but unable to get it.

!!! Fetch for /usr/portage/net-dialup/minicom/minicom-2.1-r1.ebuild failed,
continuing...



!!! Some fetch errors were encountered.  Please see above for details.


slowcoach root # 

This whilst torpage is perfectly capable of fetching the file.


pug:

Portage 2.0.51-r2 (default-x86-1.4, gcc-3.3.4, glibc-2.3.4.20040808-r1, 2.6.9 i686)
=================================================================
System uname: 2.6.9 i686 Pentium III (Coppermine)
Gentoo Base System version 1.4.16
distcc 2.16 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
Autoconf: sys-devel/autoconf-2.59-r5
Automake: sys-devel/automake-1.8.5-r1
Binutils: sys-devel/binutils-2.14.90.0.8-r1
Headers:  sys-kernel/linux26-headers-2.6.8.1-r1
Libtools: sys-devel/libtool-1.5.2-r5
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-march=pentium3 -O2 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
COMPILER=""
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3.2/share/config /usr/kde/3/share/config /usr/lib/mozilla/defaults/pref
/usr/share/config /usr/share/texmf/dvipdfm/config/
/usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/
/usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/alias
/var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/init.d /etc/terminfo /etc/env.d"
CXXFLAGS="-march=pentium3 -O2 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache distlocks fixpackages sandbox"
GENTOO_MIRRORS="http://gentoo.osuosl.org
http://distro.ibiblio.org/pub/Linux/distributions/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://localhost/gentoo-portage"
USE="X alsa apache2 apm avi berkdb bitmap-fonts cdr crypt cups directfb encode
f77 foomaticdb gdbm gif gpm gtk imlib java jpeg ldap libg++ libwww mad mikmod
mmx motif mozilla mpeg mysql ncurses nls nptl oggvorbis opengl oss pam pdflib
perl png python qt quicktime readline sdl slang spell ssl svga tcltk tcpd tetex
truetype x86 xml2 xmms xv xvid zlib"

slowcoach:

Gentoo Base System version 1.4.16
Portage 2.0.51-r2 (default-x86-1.4, gcc-3.3.4, glibc-2.3.4.20040808-r1, 2.6.9 i586)
=================================================================
System uname: 2.6.9 i586 Pentium 75 - 200
distcc 2.16 i586-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
Autoconf: sys-devel/autoconf-2.59-r5
Automake: sys-devel/automake-1.8.5-r1
Binutils: sys-devel/binutils-2.14.90.0.8-r1
Headers:  sys-kernel/linux26-headers-2.6.8.1-r1
Libtools: sys-devel/libtool-1.5.2-r5
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-march=i586 -Os -pipe -fomit-frame-pointer"
CHOST="i586-pc-linux-gnu"
COMPILER=""
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config
/usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/init.d /etc/env.d"
CXXFLAGS="-march=i586 -Os -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache distlocks sandbox"
GENTOO_MIRRORS="ftp://ftp.is.co.za/linux/distributions/gentoo
http://gentoo.oregonstate.edu/
http://www.ibiblio.org/pub/Linux/distributions/gentoo"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://ftp.up.ac.za/gentoo-portage"
USE="apache2 apm avi berkdb bitmap-fonts crypt cups encode f77 foomaticdb gdbm
gif gpm imlib java jpeg ldap libg++ libwww mad mikmod motif mpeg mysql ncurses
nptl oggvorbis opengl oss pam pdflib perl png python quicktime readline slang
spell ssl svga truetype x86 xml2 xmms xv zlib"

[Comment 1 Jaco Kroon 2004-11-03 12:08:16 UTC]

Right people, this one is a real show stopper for me at home.  It's really anoying, could I please get some feedback on this or should I go rtfs?  I would really like to push out the next version of torpage, which I suppose I could do in spite of this but I'd really like some way of still doing the read-only mount of /usr/portage via nfs from one machine to another.

[Comment 2 Jason Stubbs (RETIRED) 2004-11-04 05:15:38 UTC]

The fact that this bug hasn't been closed indicates that it is valid. The fix is not just a case of removing an "if readonly then stop the fetch" as checks need to be done later. It'll get fixed at some stage.

[Comment 3 Jaco Kroon 2004-11-04 05:21:19 UTC]

Any way I can help?  I managed to locate the few lines of code that does this but I don't know enough python to simply "fix" it.  I could just change it for me but that is kinda the wrong way.  I have two ideas, but have no idea of how to implement either one.

1)  Add a "feature", ie, another keyword for use in the FEATURES environment variable in /etc/make.conf
2)  Command line option to disable checking of ro file system.

I think option 1 is the better one and I'll rtfs a bit more as soon as I can and make an attempt at a patch as soon as possible (hopefully this weekend).

[Comment 4 Jaco Kroon 2004-11-04 13:41:41 UTC]

Created attachment 43311 [details, diff]
skiprocheck addition to FEATURES

Ok, so it hapened before the weekend.  The patch basically adds a feature
called skiprocheck which disables checking for a readonly filesystem before
attempting to download.

It also enforces use_locks = 0 (perhaps this should only be done if DISTDIR is
actually located on a ro filesystem?).

In addition it prevents portage from attempting to perform chown and chmod
operations in case of a ro filesystem.

The patch was created using portage-2.0.51-r2.

Anything I missed?

[Comment 5 Jaco Kroon 2005-03-28 10:59:41 UTC]

Created attachment 54698 [details, diff]
A much improved skiprocheck patch for portage.py

Right, this version is much better, and should cover all patchs within the
fetch function.  I've tested it rather thoroughly and am confident that it will
do the trick.

I use this along with torpage on top of a read-only nfs export to fetch files.

[Comment 6 Jason Stubbs (RETIRED) 2005-04-14 06:30:41 UTC]

It's good. Release in a few days...

[Comment 7 Steven Elling 2005-05-15 22:01:29 UTC]

I have the same "No write access to $DISTDIR" problem.  However, my configuration is a little different.

I have a central server with NFS shares---one being portage and another being a data share where distfiles are stored---and all other machines mount server:/usr/portage read-only and server:/usr/local/share/data read/write.

I have sudo set up to run portage as root only for certain users and those users are a member of the portage group, plus, that group has rwx permissions on $DISTDIR.

Running 'sudo emerge somepackage' used to work as expected on client machines but recently emerge started returning the error above.

If I run 'emerge -f somepackage' as a portage group member, it downloads the packages just fine.  If I then run 'sudo emerge somepackage' as a portage group member, it still returns the error but does continue as normal.

Here is the output of 'emerge info':

Portage 2.0.51.19 (default-linux/x86/2005.0, gcc-3.3.5-20050130, glibc-2.3.4.20041102-r1, 2.6.11-gentoo-r6 i686)
=================================================================
System uname: 2.6.11-gentoo-r6 i686 Unknown CPU Type
Gentoo Base System version 1.4.16
Python:              dev-lang/python-2.3.5 [2.3.5 (#1, May  5 2005, 00:12:17)]
distcc 2.16 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled]
ccache version 2.3 [enabled]
dev-lang/python:     2.3.5
sys-apps/sandbox:    [Not Present]
sys-devel/autoconf:  2.59-r6, 2.13
sys-devel/automake:  1.7.9-r1, 1.8.5-r3, 1.5, 1.4_p6, 1.6.3, 1.9.4
sys-devel/binutils:  2.15.92.0.2-r7
sys-devel/libtool:   1.5.16
virtual/os-headers:  2.6.8.1-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-march=athlon-xp -m3dnow -mmmx -msse -O2 -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/X11/gdm /etc/X11/rstart/rstartd.real /etc/X11/serverconfig /etc/X11/starthere /etc/X11/sysconfig /etc/X11/xdm/chooser /etc/X11/xorg.conf.example /etc/gconf /etc/gnome-vfs-2.0/modules /etc/hotplug /etc/init.d /etc/openldap/schema /etc/sound/event /etc/terminfo /usr/X11R6/lib/X11/xkb /etc/env.d"
CXXFLAGS="-march=athlon-xp -m3dnow -mmmx -msse -O2 -fomit-frame-pointer"
DISTDIR="/usr/local/share/data/pub/linux/gentoo/distfiles"
FEATURES="autoaddcvs autoconfig buildpkg ccache distcc distlocks sandbox sfperms strict userpriv usersandbox"
GENTOO_MIRRORS="http://gentoo.oregonstate.edu/ http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LANG="en_US"
LINGUAS="en"
MAKEOPTS="-j4"
PKGDIR="/usr/local/share/data/pub/linux/gentoo/packages/athlon-xp"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/share/data/pub/linux/gentoo/portage"
SYNC="rsync://charge.electrostatic.org/gentoo-portage"
USE="x86 3dnow 3dnowext X Xaw3d acl alsa apm arts audiofile avi berkdb bitmap-fonts bonobo cdparanoia cdr crypt cups curl dga divx4linux dv dvd dvdr dvdread eds emboss encode esd evms2 evo fam flac foomaticdb gd gdbm gif gnome2 gpm gstreamer gtk gtk2 gtkhtml guile imagemagick imlib ipv6 java jpeg kde kdeenablefinal ldap libg++ libwww live mad mmx mmx2 mmxext motif mozilla mp3 mpeg mysql nas ncurses network nvidia offensive ofx ogg oggvorbis opengl pam pdflib perl png python qt quicktime quotes readline samba scanner sdl spell sse ssl svga tcltk tcpd tetex tiff truetype truetype-fonts type1-fonts usb userlocales v4l v4l2 vorbis xine xml xml2 xmms xscreensaver xv xvid xvmc zlib linguas_en userland_GNU kernel_linux elibc_glibc"
Unset:  ASFLAGS, CBUILD, CTARGET, LC_ALL, LDFLAGS
-----

[Comment 8 Jason Stubbs (RETIRED) 2005-05-16 06:05:53 UTC]

Steven: Completely unrelated. This bug is fixed and released in >=portage-2.0.51.20. Just waiting for one of those versions to go stable for this bug to be closed.

[Comment 9 Jaco Kroon 2005-07-10 00:11:35 UTC]

2.0.51.22 has been released stable for amd64 but not for x86 ... ?