Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 576888

Summary: sys-apps/portage: removeFromContents FileNotFoundError with FEATURES=parallel-install
Product: Portage Development Reporter: Zac Medico <zmedico>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal Keywords: InVCS
Priority: Normal    
Version: 2.2   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 573774, 576786    

Description Zac Medico gentoo-dev 2016-03-09 17:11:58 UTC 
This involving soft-blocker file collision resolution for bind-tools-9.10.3_p2 and net-dns/bind-9.10.3_p2 was reported in bug 576786, comment #2:

>>> Emerging binary (278 of 410) net-dns/bind-tools-9.10.3_p2::gentoo
>>> Emerging binary (279 of 410) net-dns/bind-9.10.3_p2::gentoo
>>> Installing (279 of 410) net-dns/bind-9.10.3_p2::gentoo
>>> Installing (278 of 410) net-dns/bind-tools-9.10.3_p2::gentoo

.....

>>> Jobs: 284 of 410 complete, 1 failed             Load avg: 2.90, 2.90, 1.97
>>> Extracting info

 * Package:    net-dns/bind-tools-9.10.3_p2
 * Repository: gentoo
 * USE:        userland_GNU elibc_glibc readline amd64 ssl abi_x86_64 seccomp urandom kernel_linux
 * FEATURES:   preserve-libs sandbox userpriv usersandbox
>>> Extracting net-dns/bind-tools-9.10.3_p2

 * checking 29 files for package collisions
>>> Merging net-dns/bind-tools-9.10.3_p2 to /

--- /usr/
--- /usr/share/
--- /usr/share/man/
--- /usr/share/man/man8/
>>> /usr/share/man/man8/dnssec-revoke.8.bz2
>>> /usr/share/man/man8/dnssec-verify.8.bz2
>>> /usr/share/man/man8/dnssec-settime.8.bz2
>>> /usr/share/man/man8/dnssec-keygen.8.bz2
>>> /usr/share/man/man8/dnssec-keyfromlabel.8.bz2
>>> /usr/share/man/man8/dnssec-importkey.8.bz2
>>> /usr/share/man/man8/dnssec-dsfromkey.8.bz2
>>> /usr/share/man/man8/dnssec-signzone.8.bz2

--- /usr/share/man/man1/
>>> /usr/share/man/man1/host.1.bz2
>>> /usr/share/man/man1/nsupdate.1.bz2
>>> /usr/share/man/man1/dig.1.bz2
>>> /usr/share/man/man1/delv.1.bz2
>>> /usr/share/man/man1/nslookup.1.bz2

--- /usr/share/doc/
>>> /usr/share/doc/bind-tools-9.10.3_p2/
>>> /usr/share/doc/bind-tools-9.10.3_p2/CHANGES.bz2
>>> /usr/share/doc/bind-tools-9.10.3_p2/README.bz2
>>> /usr/share/doc/bind-tools-9.10.3_p2/FAQ.bz2

--- /usr/bin/
>>> /usr/bin/dnssec-keyfromlabel
>>> /usr/bin/nsupdate
>>> /usr/bin/dnssec-signzone
>>> /usr/bin/nslookup
>>> /usr/bin/dnssec-dsfromkey
>>> /usr/bin/dnssec-settime
>>> /usr/bin/dnssec-verify
>>> /usr/bin/dnssec-revoke
>>> /usr/bin/dnssec-keygen
>>> /usr/bin/dnssec-importkey
>>> /usr/bin/host
>>> /usr/bin/delv
>>> /usr/bin/dig
>>> Safely unmerging already-installed instance...
>>> Original instance of package unmerged safely.

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/_MergeProcess.py", line 235, in _spawn
    prev_mtimes=self.prev_mtimes, counter=counter)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 5013, in merge
    counter=counter)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 4297, in treewalk
    relative_paths=False)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 1063, in removeFromContents
    self.writeContentsToContentsFile(pkg, new_contents)
  File "/usr/lib64/python3.4/site-packages/portage/dbapi/vartree.py", line 1075, in writeContentsToContentsFile
    f = atomic_ofstream(os.path.join(pkg.dbdir, "CONTENTS"))
  File "/usr/lib64/python3.4/site-packages/portage/util/__init__.py", line 1288, in __init__
    mode=mode, **portage._native_kwargs(kargs)))
FileNotFoundError: [Errno 2] No such file or directory: b'/var/db/pkg/net-dns/bind-9.9.5-r3/CONTENTS.12168'
Comment 1 Zac Medico gentoo-dev 2016-03-09 17:42:24 UTC 
The list of blocked packages is evaluated inside the MergeProcess._spawn method. With parallel-install, the list of blocked packages can be stale by the time dblink.treewalk operates on the list with removeFromContents.

This can be fixed by making dblink.treewalk detect stale items in the list of blocked packages, and skip removeFromContents calls for these items.
Comment 2 Zac Medico gentoo-dev 2016-03-09 20:44:17 UTC 
(In reply to Zac Medico from comment #1)
> This can be fixed by making dblink.treewalk detect stale items in the list
> of blocked packages, and skip removeFromContents calls for these items.

The blocked packages are also used in the _collision_protect call before the files are merged, so we should acquire a lock for _collision_protect, but only when there are blocked packages to account for.
Comment 3 Zac Medico gentoo-dev 2016-03-09 21:59:01 UTC 
There's a patch in the following branch:

https://github.com/zmedico/portage/tree/bug_576888
Comment 4 Zac Medico gentoo-dev 2016-03-09 22:29:53 UTC 
(In reply to Zac Medico from comment #3)
> There's a patch in the following branch:
> 
> https://github.com/zmedico/portage/tree/bug_576888

Withdrawing this patch for now, since I'm afraid that concurrent uninstall of one of the blocked packages could unmerge files that were just installed by the current package.
Comment 5 Zac Medico gentoo-dev 2016-03-10 06:51:50 UTC 
We can make the Scheduler avoid these kinds of problems by placing conflicting packages in a queue to be processed serially.
Comment 6 Zac Medico gentoo-dev 2016-03-13 06:37:40 UTC 
(In reply to Zac Medico from comment #5)
> We can make the Scheduler avoid these kinds of problems by placing
> conflicting packages in a queue to be processed serially.

The scheduler won't be able to account for concurrent emerge/ebuild invocations, whereas locks can handle that very well.

If we introduce locking at the package-slot level, then a package with blockers can trigger locking of all the blocked package-slots. This should minimize the amount of time that packages spend waiting on one another, and will allow for cooperation with concurrent emerge/ebuild invocations.
Comment 7 Zac Medico gentoo-dev 2016-03-14 00:39:36 UTC 
(In reply to Zac Medico from comment #3)
> There's a patch in the following branch:
> 
> https://github.com/zmedico/portage/tree/bug_576888

There's a new patch in that branch, posted for review here:

https://archives.gentoo.org/gentoo-portage-dev/message/8687ecc13cc170a7b75c1da34a15d91d
Comment 9 marco 2016-06-21 12:28:34 UTC 
Can i include the patch in sys-apps/portage-2.2.28 ?

wget -O /etc/portage/patches/sys-apps/portage/bug_576888.patch https://gitweb.gentoo.org/proj/portage.git/patch/?id=dccd4687999aed1788dfe5c3461523f2b23ca79f

and use it ?
Comment 10 Zac Medico gentoo-dev 2016-06-21 14:15:11 UTC 
(In reply to marco from comment #9)
> Can i include the patch in sys-apps/portage-2.2.28 ?
> 
> wget -O /etc/portage/patches/sys-apps/portage/bug_576888.patch
> https://gitweb.gentoo.org/proj/portage.git/patch/
> ?id=dccd4687999aed1788dfe5c3461523f2b23ca79f
> 
> and use it ?

Yes, that's fine.