Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 576888 - sys-apps/portage: removeFromContents FileNotFoundError with FEATURES=parallel-install
Summary: sys-apps/portage: removeFromContents FileNotFoundError with FEATURES=parallel...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: portage-2.3.0
  Show dependency tree
 
Reported: 2016-03-09 17:11 UTC by Zac Medico
Modified: 2017-03-06 22:00 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2016-03-09 17:11:58 UTC
This involving soft-blocker file collision resolution for bind-tools-9.10.3_p2 and net-dns/bind-9.10.3_p2 was reported in bug 576786, comment #2:

>>> Emerging binary (278 of 410) net-dns/bind-tools-9.10.3_p2::gentoo
>>> Emerging binary (279 of 410) net-dns/bind-9.10.3_p2::gentoo
>>> Installing (279 of 410) net-dns/bind-9.10.3_p2::gentoo
>>> Installing (278 of 410) net-dns/bind-tools-9.10.3_p2::gentoo

.....

>>> Jobs: 284 of 410 complete, 1 failed             Load avg: 2.90, 2.90, 1.97
>>> Extracting info

 * Package:    net-dns/bind-tools-9.10.3_p2
 * Repository: gentoo
 * USE:        userland_GNU elibc_glibc readline amd64 ssl abi_x86_64 seccomp urandom kernel_linux
 * FEATURES:   preserve-libs sandbox userpriv usersandbox
>>> Extracting net-dns/bind-tools-9.10.3_p2

 * checking 29 files for package collisions
>>> Merging net-dns/bind-tools-9.10.3_p2 to /

--- /usr/
--- /usr/share/
--- /usr/share/man/
--- /usr/share/man/man8/
>>> /usr/share/man/man8/dnssec-revoke.8.bz2
>>> /usr/share/man/man8/dnssec-verify.8.bz2
>>> /usr/share/man/man8/dnssec-settime.8.bz2
>>> /usr/share/man/man8/dnssec-keygen.8.bz2
>>> /usr/share/man/man8/dnssec-keyfromlabel.8.bz2
>>> /usr/share/man/man8/dnssec-importkey.8.bz2
>>> /usr/share/man/man8/dnssec-dsfromkey.8.bz2
>>> /usr/share/man/man8/dnssec-signzone.8.bz2

--- /usr/share/man/man1/
>>> /usr/share/man/man1/host.1.bz2
>>> /usr/share/man/man1/nsupdate.1.bz2
>>> /usr/share/man/man1/dig.1.bz2
>>> /usr/share/man/man1/delv.1.bz2
>>> /usr/share/man/man1/nslookup.1.bz2

--- /usr/share/doc/
>>> /usr/share/doc/bind-tools-9.10.3_p2/
>>> /usr/share/doc/bind-tools-9.10.3_p2/CHANGES.bz2
>>> /usr/share/doc/bind-tools-9.10.3_p2/README.bz2
>>> /usr/share/doc/bind-tools-9.10.3_p2/FAQ.bz2

--- /usr/bin/
>>> /usr/bin/dnssec-keyfromlabel
>>> /usr/bin/nsupdate
>>> /usr/bin/dnssec-signzone
>>> /usr/bin/nslookup
>>> /usr/bin/dnssec-dsfromkey
>>> /usr/bin/dnssec-settime
>>> /usr/bin/dnssec-verify
>>> /usr/bin/dnssec-revoke
>>> /usr/bin/dnssec-keygen
>>> /usr/bin/dnssec-importkey
>>> /usr/bin/host
>>> /usr/bin/delv
>>> /usr/bin/dig
>>> Safely unmerging already-installed instance...
>>> Original instance of package unmerged safely.

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/_MergeProcess.py", line 235, in _spawn
    prev_mtimes=self.prev_mtimes, counter=counter)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 5013, in merge
    counter=counter)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 4297, in treewalk
    relative_paths=False)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 1063, in removeFromContents
    self.writeContentsToContentsFile(pkg, new_contents)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 1075, in writeContentsToContentsFile
    f = atomic_ofstream(os.path.join(pkg.dbdir, "CONTENTS"))
  File "/usr/lib64/python3.4/site-packages/portage/util/__init__.py", line 1288, in __init__
    mode=mode, **portage._native_kwargs(kargs)))
FileNotFoundError: [Errno 2] No such file or directory: b'/var/db/pkg/net-dns/bind-9.9.5-r3/CONTENTS.12168'
Comment 1 Zac Medico gentoo-dev 2016-03-09 17:42:24 UTC
The list of blocked packages is evaluated inside the MergeProcess._spawn method. With parallel-install, the list of blocked packages can be stale by the time dblink.treewalk operates on the list with removeFromContents.

This can be fixed by making dblink.treewalk detect stale items in the list of blocked packages, and skip removeFromContents calls for these items.
Comment 2 Zac Medico gentoo-dev 2016-03-09 20:44:17 UTC
(In reply to Zac Medico from comment #1)
> This can be fixed by making dblink.treewalk detect stale items in the list
> of blocked packages, and skip removeFromContents calls for these items.

The blocked packages are also used in the _collision_protect call before the files are merged, so we should acquire a lock for _collision_protect, but only when there are blocked packages to account for.
Comment 3 Zac Medico gentoo-dev 2016-03-09 21:59:01 UTC
There's a patch in the following branch:

https://github.com/zmedico/portage/tree/bug_576888
Comment 4 Zac Medico gentoo-dev 2016-03-09 22:29:53 UTC
(In reply to Zac Medico from comment #3)
> There's a patch in the following branch:
> 
> https://github.com/zmedico/portage/tree/bug_576888

Withdrawing this patch for now, since I'm afraid that concurrent uninstall of one of the blocked packages could unmerge files that were just installed by the current package.
Comment 5 Zac Medico gentoo-dev 2016-03-10 06:51:50 UTC
We can make the Scheduler avoid these kinds of problems by placing conflicting packages in a queue to be processed serially.
Comment 6 Zac Medico gentoo-dev 2016-03-13 06:37:40 UTC
(In reply to Zac Medico from comment #5)
> We can make the Scheduler avoid these kinds of problems by placing
> conflicting packages in a queue to be processed serially.

The scheduler won't be able to account for concurrent emerge/ebuild invocations, whereas locks can handle that very well.

If we introduce locking at the package-slot level, then a package with blockers can trigger locking of all the blocked package-slots. This should minimize the amount of time that packages spend waiting on one another, and will allow for cooperation with concurrent emerge/ebuild invocations.
Comment 7 Zac Medico gentoo-dev 2016-03-14 00:39:36 UTC
(In reply to Zac Medico from comment #3)
> There's a patch in the following branch:
> 
> https://github.com/zmedico/portage/tree/bug_576888

There's a new patch in that branch, posted for review here:

https://archives.gentoo.org/gentoo-portage-dev/message/8687ecc13cc170a7b75c1da34a15d91d
Comment 9 marco 2016-06-21 12:28:34 UTC
Can i include the patch in sys-apps/portage-2.2.28 ?

wget -O /etc/portage/patches/sys-apps/portage/bug_576888.patch https://gitweb.gentoo.org/proj/portage.git/patch/?id=dccd4687999aed1788dfe5c3461523f2b23ca79f

and use it ?
Comment 10 Zac Medico gentoo-dev 2016-06-21 14:15:11 UTC
(In reply to marco from comment #9)
> Can i include the patch in sys-apps/portage-2.2.28 ?
> 
> wget -O /etc/portage/patches/sys-apps/portage/bug_576888.patch
> https://gitweb.gentoo.org/proj/portage.git/patch/
> ?id=dccd4687999aed1788dfe5c3461523f2b23ca79f
> 
> and use it ?

Yes, that's fine.