Binhost error during fetching/extracting stage. Use of a file:/// local location either in make.conf PORTAGE_BINHOST or binrepos.conf sync-uri = will result in an error. binary N ] dev-python/asn1crypto-1.5.1_p20231012-1::gentoo USE="-test" PYTHON_TARGETS="python3_11 (-pypy3) -python3_10 (-python3_12)" 333 KiB >>> Emerging binary (1 of 2) dev-python/asn1crypto-1.5.1_p20231012::gentoo * /var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-2.xpak.partial >>> Extracting info Error reading binpkg '/var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-1.xpak': [Errno 2] No such file or directory: '/var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-1.xpak'!!! Error Extracting '/var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-1.xpak', ('dev-python/asn1crypto-1.5.1_p20231012', 1, 340016, 1703705004, 1703705004) >>> Failed to emerge dev-python/asn1crypto-1.5.1_p20231012, Log file: Reproducible: Always Steps to Reproduce: 1. setting make.conf PORTAGE_BINHOST to a file:/// location And/or similarly bugged if using binrepos.conf sync-uri = /some/path or file:///some/path 2. emerge -g (--getbinpkg) will find the ID -1 binpkg saying it will be installed 3. when it actually goes to extract the -1 id, it fails because the -1 that was fetched turned into a -2 locally. Actual Results: Remote side: (normal) # ls /mnt/crucialp1/systemd-verified-cow-1-binpkgs/dev-python/asn1crypto asn1crypto-1.5.1_p20231012-1.xpak Local side after the error. # ls /var/cache/binpkgs/dev-python/asn1crypto asn1crypto-1.5.1_p20231012-2.xpak Expected Results: Local side has no of this package and emaint binhost --fix was run on both PKGDIRs and /var/cache/edb/binhost was purged entirely. The -1 ID package from the remote SHOULD download fetch as -1 locally, and extract itself normally as -1. Incrementing the value should not happen in the case when the remote package lives on a file:/// type PKGDIR. The detection of a local file behavior is likely triggering the duplicate-ID avoidance mechanism with the host PKG by auto-incrementing to not clobber. But that cant apply here. The two PKGDIRS are not the same even though they are both local. another user has come forward to alert me to this, named YmrDtnJu and it seems to be a widespread issue, I was able to reproduce this and submit the bug for them.
Created attachment 881269 [details] Build ID is auto-incremented for no reason but it breaks everything
Created attachment 881270 [details] emerge --info Of note they report it being broken in Portage 3.0.59 and working in 3.0.57 - to narrow it down.
(In reply to genBTC from comment #0) > Expected Results: > Local side has no of this package and emaint binhost --fix was run on both > PKGDIRs and /var/cache/edb/binhost was purged entirely. The -1 ID package > from the remote SHOULD download fetch as -1 locally, and extract itself > normally as -1. > Incrementing the value should not happen in the case when the remote package > lives on a file:/// type PKGDIR. It should increment if there is an existing file, but it seems that you don't have one, so it's odd that it incremented. Maybe it's some kind of flaw in the _allocate_filename_multi logic from https://github.com/gentoo/portage/pull/893. > The detection of a local file behavior is likely triggering the duplicate-ID > avoidance mechanism with the host PKG by auto-incrementing to not clobber. > But that cant apply here. The two PKGDIRS are not the same even though they > are both local. Allocating a new file name should be supported if there is an existing file, however, this code in the Binpkg _unpack_metadata method uses self.settings.mycpv where it is stale: await self._bintree.dbapi.unpack_metadata( self.settings, infloc, loop=self.scheduler ) There's an earlier self._bintree.inject call that returns a regenerated cpv that would be valid to use instead of self.settings.mycpv: if self._fetched_pkg: pkg_path = self._bintree.getname( self._bintree.inject( pkg.cpv, current_pkg_path=self._fetched_pkg, allocated_pkg_path=self._pkg_allocated_path, ), allocate_new=False, )
The reason _allocate_filename_multi increments is that it calls _max_build_id which counts the remote instance.
We can make _allocate_filename_multi simply re-use the remote build_id if the corresponding file does not exist locally. The call to _max_build_id can be skipped in this case.
We should probably update the Package cpv attribute with the result of bintree.inject inside both BinpkgPrefetcher and Binpkg, and also in the Scheduler _run_pkg_pretend method.
(In reply to Zac Medico from comment #6) > We should probably update the Package cpv attribute with the result of > bintree.inject inside both BinpkgPrefetcher and Binpkg, and also in the > Scheduler _run_pkg_pretend method. All of these inject calls also need error handling like EbuildBinpkg has since bug 921089.
(In reply to genBTC from comment #0) > >>> Emerging binary (1 of 2) dev-python/asn1crypto-1.5.1_p20231012::gentoo > * > /var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-2.xpak. > partial > > >>> Extracting info > Error reading binpkg > '/var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-1.xpak': > [Errno 2] No such file or directory: > '/var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-1.xpak'! > !! Error Extracting > '/var/cache/binpkgs/dev-python/asn1crypto/asn1crypto-1.5.1_p20231012-1.xpak', > ('dev-python/asn1crypto-1.5.1_p20231012', 1, 340016, 1703705004, 1703705004) It seems like the binarytree inject failed, since the downloaded package should have had the same instance key, but the messages seem to indicate that the key is missing (which explains why the path is incorrect). Error handling around the binarytree inject call will help if there are any other bugs lurking here.
There's a regression from bug 921089 because FileCopier does not preserve timestamps. This fixed lib/portage/tests/emerge/test_binpkg_fetch.py in https://github.com/gentoo/portage/pull/1227: --- a/lib/_emerge/BinpkgFetcher.py +++ b/lib/_emerge/BinpkgFetcher.py @@ -106,7 +106,8 @@ class BinpkgFetcher(CompositeTask): finally: if copier.isAlive(): copier.cancel() - + if copier.returncode == os.EX_OK: + fetcher.sync_timestamp() else: fetcher.start() try:
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=6c915f3ded10c23127c33b771453d54e3f0d4915 commit 6c915f3ded10c23127c33b771453d54e3f0d4915 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-01-03 04:26:36 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-01-03 05:56:59 +0000 binarytree: Avoid unnecessary build id incrementation When allocating a local file name for a remote package, prefer re-using the existing build id if the file name is unused locally, in order to avoid unnecessary build id incrementation triggered when the _max_build_id method counts remote build ids. Use the same file name reservation via open with O_EXCL which has been used during build_id incrementation since fa901a6510c. Bug: https://bugs.gentoo.org/921208 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/bintree.py | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=d8285bad5815f0f2db64657dd245efc3fe948210 commit d8285bad5815f0f2db64657dd245efc3fe948210 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-03 06:08:53 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-03 19:59:51 +0000 BinpkgFetcher: preserve mtime The FileCopier from 1db44d18578a7aee58449cb97e1991cb06c915c3 doesn't preserve its timestamp so there's a mismatch between the expected mtime in the index and the binpkg. Bug: https://bugs.gentoo.org/921208 Thanks-to: Zac Medico <zmedico@gentoo.org> Signed-off-by: Sam James <sam@gentoo.org> lib/_emerge/BinpkgFetcher.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=14ec6c259ca5bd7439ede37216bbb0d160e909e9 commit 14ec6c259ca5bd7439ede37216bbb0d160e909e9 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-03 04:22:24 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-03 19:59:51 +0000 tests: add getbinpkg file:// sync testcase for BUILD_ID We do a bit of a strange dance here where we: 1) create a binpkg; 2) shift the PKGDIR so that it becomes a "remote" one for use with file:///; 3) try to merge the binpkg but defer checking the exit status; 4) check if the downloaded filename was unnecessarily incremented (-2). if it is, fail. 5) check the deferred exit status from 3) so that we fail if the merge failed for another reason (e.g. failed injection). Bug: https://bugs.gentoo.org/921208 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/tests/emerge/meson.build | 1 + lib/portage/tests/emerge/test_binpkg_fetch.py | 226 ++++++++++++++++++++++++++ 2 files changed, 227 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b37b3b5340a35656651134154b193dbda7cf13c6 commit b37b3b5340a35656651134154b193dbda7cf13c6 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-05 08:23:30 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-05 08:23:30 +0000 sys-apps/portage: add 3.0.61 Note that bug #921208 is only partly fixed but I wanted to get these fixes out, in part because we've got more people trying binpkgs right now with the binhost announcement. Bug: https://bugs.gentoo.org/921208 Closes: https://bugs.gentoo.org/921177 Closes: https://bugs.gentoo.org/920180 Closes: https://bugs.gentoo.org/921181 Closes: https://bugs.gentoo.org/921327 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.61.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69 commit 77c44c46194922509bc4f2b5cfc099412a560a69 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-02-22 07:23:40 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-02-22 07:23:50 +0000 sys-apps/portage: add 3.0.62 Closes: https://bugs.gentoo.org/663324 Closes: https://bugs.gentoo.org/728046 Closes: https://bugs.gentoo.org/891137 Closes: https://bugs.gentoo.org/906368 Closes: https://bugs.gentoo.org/916566 Closes: https://bugs.gentoo.org/921170 Closes: https://bugs.gentoo.org/921208 Closes: https://bugs.gentoo.org/921400 Closes: https://bugs.gentoo.org/922038 Closes: https://bugs.gentoo.org/922142 Closes: https://bugs.gentoo.org/923368 Closes: https://bugs.gentoo.org/923750 Closes: https://bugs.gentoo.org/923841 Closes: https://bugs.gentoo.org/923852 Closes: https://bugs.gentoo.org/923854 Closes: https://bugs.gentoo.org/924192 Closes: https://bugs.gentoo.org/924273 Closes: https://bugs.gentoo.org/924585 Closes: https://bugs.gentoo.org/921380 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+)
