Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 774384 - sys-apps/portage-prefix: Add option to disable locking or use symlinks for systems that do not support hardlinks
Summary: sys-apps/portage-prefix: Add option to disable locking or use symlinks for sy...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: Normal enhancement with 1 vote (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-06 04:55 UTC by matoro
Modified: 2021-03-17 19:36 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
_get_lock_fn: test fcntl.flock and hardlink_lockfile (_get_lock_fn-test-fcntl.flock-and-hardlink_lockfile.patch,1.24 KB, patch)
2021-03-06 08:36 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description matoro archtester 2021-03-06 04:55:18 UTC
Android does not allow the creation of hardlinks by applications.  See e.g. https://github.com/termux/termux-app/issues/837  Could there be an option added to sys-apps/portage-prefix to either a) use symlinks instead of hardlinks for locking or b) disable the use of locks entirely?

Example traceback of what this looks like:

* Bootstrapping Gentoo prefixed portage installation using
* host:   aarch64-unknown-linux-gnu
* prefix: /data/data/com.termux/files/home/gentoo
* ready to bootstrap stage2_log
* Triggering Linux RAP bootstrap
USE=-acl -berkdb -fortran -gdbm -git -libcxx -nls -pcre -python -qmanifest -qtegrity -readline bootstrap clang internal-glib PKG=sys-devel/gnuconfig

These are the packages that would be merged, in order:

[ebuild  N     ] sys-devel/gnuconfig-20210107::gentoo  65 KiB

Total: 1 package (1 new), Size of downloads: 65 KiB

>>> Verifying ebuild manifests


>>> Emerging (1 of 1) sys-devel/gnuconfig-20210107::gentoo
Exception in callback AsynchronousTask._exit_listener_cb(<bound method...0x7554fd0c70>>)
handle: <Handle AsynchronousTask._exit_listener_cb(<bound method...0x7554fd0c70>>)>
Traceback (most recent call last):
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/portage/locks.py", line 382, in _lockfile_was_removed
    os.link(lock_path, hardlink_path)
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/portage/__init__.py", line 243, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
PermissionError: [Errno 13] Permission denied: b'/data/data/com.termux/files/home/gentoo/tmp/var/tmp/portage/.sys-devel.portage_lockfile' -> b'/data/data/com.termux/files/home/gentoo/tmp/var/tmp/portage/..sys-devel.portage_lockfile.hardlock-localhost-11094'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/_emerge/AsynchronousTask.py", line 201, in _exit_listener_cb
    listener(self)
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/_emerge/EbuildBuild.py", line 109, in _start_with_metadata
    self._prefetch_exit(prefetcher)
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/_emerge/EbuildBuild.py", line 178, in _prefetch_exit
    AsyncTaskFuture(future=self._build_dir.async_lock()),
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/_emerge/EbuildBuildDir.py", line 113, in async_lock
    catdir_lock.start()
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/_emerge/AsynchronousTask.py", line 30, in start
    self._start()
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/_emerge/AsynchronousLock.py", line 49, in _start
    self._imp = lockfile(self.path,
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/portage/locks.py", line 141, in lockfile
    lock = _lockfile_iteration(mypath, wantnewlockfile=wantnewlockfile,
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/portage/locks.py", line 330, in _lockfile_iteration
    (removed, fstat_result) = _lockfile_was_removed(myfd, lockfilename)
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/portage/locks.py", line 385, in _lockfile_was_removed
    _raise_exc(e)
  File "/data/data/com.termux/files/home/gentoo/tmp/usr/lib/python3.8/portage/util/install_mask.py", line 150, in _raise_exc
    raise wrapper
portage.exception.PermissionDenied: [Errno 13] Permission denied: b'/data/data/com.termux/files/home/gentoo/tmp/var/tmp/portage/.sys-devel.portage_lockfile' -> b'/data/data/com.termux/files/home/gentoo/tmp/var/tmp/portage/..sys-devel.portage_lockfile.hardlock-localhost-11094'
Terminated

Reproducible: Always
Comment 1 Zac Medico gentoo-dev 2021-03-06 08:36:49 UTC
Created attachment 689604 [details, diff]
_get_lock_fn: test fcntl.flock and hardlink_lockfile

This patch shows how we can test hardlink_lockfile in the _get_lock_fn, and possibly add a fallback.
Comment 2 Nathan Shearer 2021-03-17 17:55:41 UTC
I have run into this problem as well. For me, portage appears to be creating a lockfile at var/db/.pkg.portage_lockfile and then crashing because it can't do something with that file.

Adding FEATURES="-distlocks" does not seem to help.

Can I suggest something like FEATURES="-hardlink" to remove hardlink usage entirely from portage since the underlying filesystem does not allow hardlinks?

https://forums.gentoo.org/viewtopic-p-8588094.html#8588094
Comment 3 Nathan Shearer 2021-03-17 18:13:04 UTC
I tried applying the patch to /data/local/tmp/gentoo/var/db/.pkg.portage_lockfile in my prefix install, yet I am still receiving the following error:

/data/local/tmp/gentoo $ emerge -pv vim

These are the packages that would be merged, in order:

Calculating dependencies... done!
Permission denied: '[Errno 13] Permission denied: b'/data/local/tmp/gentoo/var/db/.pkg.portage_lockfile' -> b'/data/local/tmp/gentoo/var/db/..pkg.portage_lockfile.hardlock-localhost-4719''
Comment 4 Zac Medico gentoo-dev 2021-03-17 19:20:36 UTC
(In reply to Nathan Shearer from comment #2)
> I have run into this problem as well. For me, portage appears to be creating
> a lockfile at var/db/.pkg.portage_lockfile and then crashing because it
> can't do something with that file.
> 
> Adding FEATURES="-distlocks" does not seem to help.
> 
> Can I suggest something like FEATURES="-hardlink" to remove hardlink usage
> entirely from portage since the underlying filesystem does not allow
> hardlinks?
> 
> https://forums.gentoo.org/viewtopic-p-8588094.html#8588094

I'd prefer to perform a runtime test for hardlink support, so that the user won't have to do anything.

(In reply to Nathan Shearer from comment #3)
> I tried applying the patch to
> /data/local/tmp/gentoo/var/db/.pkg.portage_lockfile in my prefix install,
> yet I am still receiving the following error:

We'll have to implement some alternative locking mechanism before it will be useful.
Comment 5 Nathan Shearer 2021-03-17 19:36:31 UTC
(In reply to Zac Medico from comment #4)
> (In reply to Nathan Shearer from comment #2)
> > I have run into this problem as well. For me, portage appears to be creating
> > a lockfile at var/db/.pkg.portage_lockfile and then crashing because it
> > can't do something with that file.
> > 
> > Adding FEATURES="-distlocks" does not seem to help.
> > 
> > Can I suggest something like FEATURES="-hardlink" to remove hardlink usage
> > entirely from portage since the underlying filesystem does not allow
> > hardlinks?
> > 
> > https://forums.gentoo.org/viewtopic-p-8588094.html#8588094
> 
> I'd prefer to perform a runtime test for hardlink support, so that the user
> won't have to do anything.

That sounds good to me.

> (In reply to Nathan Shearer from comment #3)
> > I tried applying the patch to
> > /data/local/tmp/gentoo/var/db/.pkg.portage_lockfile in my prefix install,
> > yet I am still receiving the following error:
> 
> We'll have to implement some alternative locking mechanism before it will be
> useful.

I would be happy to test them out on my prefix install. It took over 24 hours to compile the prefix on a raspberry pi 4, so I have it built and working on my up-to-date stock (not rooted) Note 20 and ready to test.

I can also provide a copy of the prefix if that will help. It's vanilla unmodified and works on android without root.