Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 252821 - Exporting /usr/portage read-only and lockfile failure.
Summary: Exporting /usr/portage read-only and lockfile failure.
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: High minor (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-12-28 15:05 UTC by C W Rose
Modified: 2009-01-14 18:00 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description C W Rose 2008-12-28 15:05:51 UTC
If /usr/portage is exported with a Samba share which is set read-only
(a convenient and secure approach) a build using the exported directory
fails. This is because portage uses the /usr/portage/distfiles/.locks for
lock files.

It is suggested that /usr/lib/portage/pym/portage.py be changed (around
line 3157) to use PORT_LOCKDIR instead of DISTFILES, and that PORT_LOCKDIR
be defined in /etc/make.conf as /var/lock/portage.  A .lock directory can
then be placed in the standard lock tree. (Why is the .lock directory
hidden anyway?).    

Reproducible: Always
Comment 1 Marius Mauch (RETIRED) gentoo-dev 2008-12-28 17:24:22 UTC
It's preferrable to keep the locks in the same directory tree as the files that have to be locked for technical reasons (as locks must be on the same physical filesystem). Also moving distfile locks to /var is defeating the purpose of the locks, unless you share /var as well (all systems accessing the distfiles must also use the same locks).
As a workaround for your problem I suggest you change $DISTDIR to a writable location or set FEATURES="-distlocks" in make.conf.
Comment 2 C W Rose 2008-12-28 21:49:37 UTC
Well, if you have to have a lock on the same filesystem, why is /var/lock
so widely used?  In fact, you can build locks on a different FS, as long
as everyone adheres to the same lock convention (as is done with UUCP);
and if they don't, using the same FS isn't going to help.

In this case portage has sole control, so there's no problem.

As far as different people accessing the file system is concerned, it's being
exported _read_only_; no lock is in fact needed, it's just that portage sets
one for reasons of its own.

And no, I don't want to make the directory writeable, since I don't want the
general public to mess with it.

Comment 3 Marius Mauch (RETIRED) gentoo-dev 2008-12-31 02:46:03 UTC
(In reply to comment #2)
> Well, if you have to have a lock on the same filesystem, why is /var/lock
> so widely used?

Is it? I'm not aware of many things actually using /var/lock for file-locks.

> In fact, you can build locks on a different FS, as long
> as everyone adheres to the same lock convention (as is done with UUCP);
> and if they don't, using the same FS isn't going to help.

Actually, the FHS specifies the format for lockfiles in /var/lock:
"The naming convention which must be used is "LCK.." followed by the base name of the device"
(http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLOCKLOCKFILES)

Note that it says "device", not "file", which strongly suggests that the location should not be used for file locks. And by referring to the basename you can't safely use it for file-locks anyway as you'd have to use the same lockfile for different files.

> As far as different people accessing the file system is concerned, it's being
> exported _read_only_; no lock is in fact needed, it's just that portage sets
> one for reasons of its own.

It's exported read-only in _your_ case, so _you_ don't need the locks. But by default we have to play it safe and assume that different systems have concurrent write access (after all, the locking was added because someone had a need for it, not because it was considered a cool feature).
And if you don't need the locking you can disable it with FEATURES="-distlocks" in make.conf as I already suggested.

@Zac: maybe distlocks could be automatically disabled if we don't have write access to $DISTDIR, so I'll leave this open in case you want to look into that.
Comment 4 C W Rose 2009-01-03 16:00:10 UTC
In fact, /var/locks is the successor to the old /usr/spool/uucp, which
was used for file, device, and process locks.  However, to recap, the
original problem was to export a portage filesystem via Samba, read-only
so that the remote users couldn't tamper with it.  I found that this
wasn't possible, since portage wanted to write lockfiles to the filesystem
even when reading it.  This led to three questions, which I've now managed
to answer:

Q.  Why did portage's lock extend even to reading the filesystem?
A.  It's easier to program that way.

Q.  Why was the lockfile directory hidden?
A.  To add to the bravado.

Q.  Why was the lockfile not installed somewhere under /var/lock?
A.  Before version 2.6 Linux' open() call didn't have the O_EXCL flag.
    (Which really suprised me.)  Possibly creating the lockfile and
    linking to it was thought to be too slow.

Anyhow, portage works with the transplanted lock directory, (though this
must be done on each client, not the server) whether because the lock
succeeds, or because portage ignores lock failure, I don't know.  Since
the main block of portage code is 10K lines of python, it will be a while
before I dig deeper into the matter.

Investigating distlocks, I find that together with clean_locks it is
documented in the make.conf man page, not in emerge.  It might be worth
putting a pointer in the emerge man page.

Comment 5 Alec Warner (RETIRED) archtester gentoo-dev Security 2009-01-03 20:39:14 UTC
(In reply to comment #4)
> In fact, /var/locks is the successor to the old /usr/spool/uucp, which
> was used for file, device, and process locks.  However, to recap, the
> original problem was to export a portage filesystem via Samba, read-only
> so that the remote users couldn't tamper with it.  I found that this
> wasn't possible, since portage wanted to write lockfiles to the filesystem
> even when reading it.  This led to three questions, which I've now managed
> to answer:
> 
> Q.  Why did portage's lock extend even to reading the filesystem?
> A.  It's easier to program that way.

What about a use case where Alice starts reading a file (without locking it) and then Bob comes along and locks the file writing and then modifies it?  Alice could end up with bad data.  Locking on reads is useful to avoid this corner case.

> 
> Q.  Why was the lockfile directory hidden?
> A.  To add to the bravado.
> 
> Q.  Why was the lockfile not installed somewhere under /var/lock?
> A.  Before version 2.6 Linux' open() call didn't have the O_EXCL flag.
>     (Which really suprised me.)  Possibly creating the lockfile and
>     linking to it was thought to be too slow.

I think pure laziness wins out here, certainly we don't want to limit use to O_EXCL (even the manpage says that portable programs should symlink to the lockfile).  A patch to lock in some user specified directly should be fairly trivial.

> 
> Anyhow, portage works with the transplanted lock directory, (though this
> must be done on each client, not the server) whether because the lock
> succeeds, or because portage ignores lock failure, I don't know.  Since
> the main block of portage code is 10K lines of python, it will be a while
> before I dig deeper into the matter.
> 
> Investigating distlocks, I find that together with clean_locks it is
> documented in the make.conf man page, not in emerge.  It might be worth
> putting a pointer in the emerge man page.
> 

Yeah manpages should have a search feature over the entire corpus :/
Comment 6 Pascal Poudrier 2009-01-14 16:16:11 UTC
I also have the same problem. Even trying with the FEATURES="-distlocks" doesn't help, I still get the same error, readonly filesystem.

A little difference is that I use shared binary packages (/usr/portage/packages). emerge try to create a lock in that folder (which is readonly).

Thank's
Comment 7 Zac Medico gentoo-dev 2009-01-14 16:51:27 UTC
(In reply to comment #6)
> I also have the same problem. Even trying with the FEATURES="-distlocks"
> doesn't help, I still get the same error, readonly filesystem.

Current versions of portage use FEATURES=distlocks to enable locking in $PKGDIR. Also, locks aren't used unless you have emerge's --getbinpkg option enabled, or have FEATURES=getbinpkg enabled. If the behavior that you observe differs, then maybe you need to update to the latest version of sys-apps/portage.
Comment 8 Pascal Poudrier 2009-01-14 17:25:41 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > I also have the same problem. Even trying with the FEATURES="-distlocks"
> > doesn't help, I still get the same error, readonly filesystem.
> 
> Current versions of portage use FEATURES=distlocks to enable locking in
> $PKGDIR. Also, locks aren't used unless you have emerge's --getbinpkg option
> enabled, or have FEATURES=getbinpkg enabled. If the behavior that you observe
> differs, then maybe you need to update to the latest version of
> sys-apps/portage.
> 

I use portage 2.2_rc20 and I emerge using -K (getbinpkgonly). I fetch binary package from a readonly samba share.
Comment 9 Zac Medico gentoo-dev 2009-01-14 17:47:54 UTC
You don't need --getbinpkg/--getbinpkgonly or the related FEATURES unless you are fetching remote packages with PORTAGE_BINHOST.
Comment 10 Pascal Poudrier 2009-01-14 18:00:04 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > I also have the same problem. Even trying with the FEATURES="-distlocks"
> > doesn't help, I still get the same error, readonly filesystem.
> 
> Current versions of portage use FEATURES=distlocks to enable locking in
> $PKGDIR. Also, locks aren't used unless you have emerge's --getbinpkg option
> enabled, or have FEATURES=getbinpkg enabled. If the behavior that you observe
> differs, then maybe you need to update to the latest version of
> sys-apps/portage.
> 

I use portage 2.2_rc20 and I emerge using -K (getbinpkgonly). I fetch binary package from a readonly samba share.(In reply to comment #9)
> You don't need --getbinpkg/--getbinpkgonly or the related FEATURES unless you
> are fetching remote packages with PORTAGE_BINHOST.
> 

Ok, maybe I missuse the whole thing.

What I did is I mapped the whole /usr/portage to a samba share and I mount it on client machine. Binary packages are located in /usr/portage/packages/All (I think it's the default place for binary).

If I want to emerge a package, I simply do an "emerge -Kav package_name" and everything work. I use this way since at least 4 years and I started recently getting problem. I patched it by putting the share RW instead and it fixed the problem (at least for now) but I prefer keeping the whole thing RO.

I don't want all my machine to emerge --sync and binpkg are still accessible within the same share.

Is it better to use a BIN_HOST instead ?

Thanks !