I have a package server that is building binary packages for client computers. Both the client and the server is using portage 3.0.36. The server is distributing the packages to the clients via `lighttpd`. When installing for example the package dosfstools or metalog, the client computer gets an extra binary package file. The server only has one binary package that is denoted by `-1` in the package name. However the client gets both the `-1` package and a `-2` package in the `/var/cache/binpkgs` directory. I can see that it seems like the client is downloading two files as it first states that downloads the `-1` file but it is saving it to `-2` and `-2.gpkg.tar.partial` The package is installed correctly but a invalid `-2` file is left behind and portage complains that it is a "Invalid binary package". This didn't seem to happen when I was using xpak. Reproducible: Always Steps to Reproduce: 1. Setup a binary package server and have it package metalog or dosfsutils 2. Install one of those package on a client 3. Notice that portage complains about the invalid binary packages. Actual Results: The client gets an extra binary package `-2` in the cache folder even though that file doesn't exist on the server. Expected Results: No ´-2´ files should be created as they do not exist on the server
Created attachment 813667 [details] Server make.conf
Created attachment 813691 [details] Client make.conf
Note that this doesn't happen on every binary package. So I'm a bit confused of what triggers it. I don't know if I have supplied enough information to debug this properly, so please let me know if you need any more information.
I forgot to mention that it is consistent though. I have reinstalled both the server and the client from scratch and it is the same binary packages that has issues.
This could be the bug https://bugs.gentoo.org/870283 . Can you try this patch and see if it fixed? https://github.com/gentoo/portage/commit/095b7c615abca297bfa6e912a4d83001c43ed055.patch
If I patch the client with that patch it partially solves it. It solves it for non live packages. However for live `9999` packages (in this case blender) it still has the same issue. In addition to that, it seems like it still saves the binary package with its number incremented. So for metalog for example, on the server the package is `metalog-20220214-1.gpkg.tar` but when the client downloads it, it is renamed to `metalog-20220214-2.gpkg.tar` even if there is no `-1` file on the client. Same with the blender live packages, the server has -3, but the client creates both -3 and -4 (where -4 is invalid). So I guess there are different code patch for live and non live packages?
I did some more tests and have some more results. If I do a clean install of the client and apply the portage patch after stage3 has been extracted and chrooted into, the only binary package that becomes invalid is the `linux-firmware` package. (I install a full system with KDE plasma and Blender) It is also reproducible by removing the binary package in the cache and running `emaint binhost --fix` and then re-emerging `linux-firmware` Strangely enough, the live Blender package does not have an issue on first install. However if I remove the binary cache package and do the `emaint` clean up, new emerges do create invalid binary package files. So that one seems a bit flaky.
(In reply to Sebastian Parborg from comment #6) > In addition to that, it seems like it still saves the binary package with > its number incremented. > > So for metalog for example, on the server the package is > `metalog-20220214-1.gpkg.tar` but when the client downloads it, it is > renamed to `metalog-20220214-2.gpkg.tar` even if there is no `-1` file on > the client. The number is no longer meaningful after this update. Because if the package is signed, it cannot be updated locally. So its only purpose is to prevent filename conflicts from different compiles/sources. Since '-1' is in the binpkg database, it will try to use the next available number when download remote package which will be save as '-2'.
(In reply to Sheng Yu from comment #8) > > The number is no longer meaningful after this update. Because if the package > is signed, it cannot be updated locally. So its only purpose is to prevent > filename conflicts from different compiles/sources. Since '-1' is in the > binpkg database, it will try to use the next available number when download > remote package which will be save as '-2'. I see. Is there any easy way to query available cached binary package slots on the system? I struggled a bit because I thought at first that if I did `emerge =<package>-2` I would get the `-2` tar file. But it instead uses the slot number. So from what I can see I would have to parse `Packages` myself to be able to figure out which slots are available. And preferably some information about them like the build date etc. Is there a built in way to query this through portage? I also tried `eix` but that doesn't seem to have anything for this either (but I might have missed something obvious of course). I also saw the pull request you created. I will not be able to test it until Monday. But I guess you are already quite certain that this will fix it. Will you do a `-r1` release of portage 3.0.36 with these fixes? Or will they be in 3.0.38?
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=7dc66f4f5635ce4abb97b3c6698cda3396cce820 commit 7dc66f4f5635ce4abb97b3c6698cda3396cce820 Author: Sheng Yu <syu.os@protonmail.com> AuthorDate: 2022-09-23 20:32:01 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-09-25 01:36:11 +0000 _emerge: Scheduler: Use pkg_allocated_path in Scheduler We previously forgot to plumb in allocated_pkg_path to use pkg_allocated_path which meant that Scheduler allocated a new one rather than the existing one. Bug: https://bugs.gentoo.org/872392 Signed-off-by: Sheng Yu <syu.os@protonmail.com> Closes: https://github.com/gentoo/portage/pull/906 Signed-off-by: Sam James <sam@gentoo.org> lib/_emerge/Scheduler.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
(In reply to Sebastian Parborg from comment #9) > Will you do a `-r1` release of portage 3.0.36 with these fixes? > Or will they be in 3.0.38? 3.0.38 I imagine with the fix for this and bug 864259. Please let us know either way if the patch works or not. Thanks!
I tested it out and it seems to be fixed. :) At least I don't get any bad packages from `linux-firmware` now which was the one that was easy to reproduce.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=00f62c1578506cb2d94b3ccf922cb40fa128387a commit 00f62c1578506cb2d94b3ccf922cb40fa128387a Author: Sam James <sam@gentoo.org> AuthorDate: 2022-10-01 02:59:09 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-10-01 02:59:59 +0000 sys-apps/portage: add 3.0.38 Closes: https://bugs.gentoo.org/827974 Closes: https://bugs.gentoo.org/864259 Closes: https://bugs.gentoo.org/865115 Closes: https://bugs.gentoo.org/871570 Closes: https://bugs.gentoo.org/872392 Closes: https://bugs.gentoo.org/872440 Closes: https://bugs.gentoo.org/873088 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.38.ebuild | 273 +++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+)
