| Summary: | Binary package global update changes BINPKG_FORMAT extension | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Nowa Ammerlaan <nowa> |
| Component: | Binary packages support | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | gentoo, mgorny, sam, syu.os |
| Priority: | Normal | Keywords: | InVCS |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | 873757 | ||
| Bug Blocks: | |||
|
Description
Nowa Ammerlaan
2022-10-17 10:14:12 UTC
Hmmm, so we cannot trust binpkg_format in the index. Use file extension to check format? (In reply to Sheng Yu from comment #1) > Hmmm, so we cannot trust binpkg_format in the index. > Use file extension to check format? But isn't the index correct prior to the update? Or if Portage writes wrong format into the index in the first place, that seems to be the actual bug. (In reply to Michał Górny from comment #2) > (In reply to Sheng Yu from comment #1) > > Hmmm, so we cannot trust binpkg_format in the index. > > Use file extension to check format? > > But isn't the index correct prior to the update? Or if Portage writes wrong > format into the index in the first place, that seems to be the actual bug. For what I understand, metadata like BINPKG_FORMAT in Package Index got deduplicate. And when user change the BINPKG_FORMAT, it get overwritten by the new value and portage think the old untracked package is gpkg. Another possible solution is we exclude BINPKG_FORMAT from global record and only add per package. (In reply to Sheng Yu from comment #3) > Another possible solution is we exclude BINPKG_FORMAT from global record and > only add per package. I think it should be per-package, tbh. It's not worth the risk of the confusion. Some people might want to keep old binpkgs for a very long time. Ah, sorry, I didn't realize it was global. In that case, perhaps detecting on filename (or header) is the best, and we shouldn't really be storing BINPKG_FORMAT in the first place. (In reply to Michał Górny from comment #5) > Ah, sorry, I didn't realize it was global. In that case, perhaps detecting > on filename (or header) is the best, and we shouldn't really be storing > BINPKG_FORMAT in the first place. BINPKG_FORMAT exists because the binary package "path" may not exists in the index. I don't know in what case this could happen, but portage code shows if the path not recorded, it will guess the path. Maybe old behaviour? Sort of related to this:
I have noticed that when a package is renamed portage appears to *copy* the binpkgs instead of moving them (and then does this with the wrong file extension). This makes emerge --sync incredibly slow when there are a bunch of binpkgs for that package. My PKGDIR is network mounted so it can be shared between different computers. However, copying all these binpkgs over the network is really really slow (global updates can take 15 to 20 minutes).
Why don't we just keep it simple and move the binpkgs instead? That would be so much faster. Besides, there is no point in keeping the binpkgs with the old name, we won't be using them anyway.
I.e. when we have 'move dev-python/zeroconf dev-python/python-zeroconf' just do the equivalent of 'mv ${PKGDIR}/dev-python/zeroconf ${PKGDIR}/dev-python/python-zeroconf'. This would:
- Speed things up significantly,
- Make things simpler and,
- Fix this bug (where we get the wrong file extension on the moved binpkg)
(In reply to Sheng Yu from comment #6) > BINPKG_FORMAT exists because the binary package "path" may not exists in the > index. I don't know in what case this could happen, but portage code shows > if the path not recorded, it will guess the path. Maybe old behaviour? Yes, might be just backwards compatibility. If the current version of Portage always includes PATH, then BINPKG_FORMAT is probably entirely redundant, since old versions of Portage don't feature GPKG support, so if it's missing, you can just assume XPAK (that said, you'd still have to deal with two possible XPAK suffixes). (In reply to Andrew Ammerlaan from comment #7) > I.e. when we have 'move dev-python/zeroconf dev-python/python-zeroconf' just > do the equivalent of 'mv ${PKGDIR}/dev-python/zeroconf > ${PKGDIR}/dev-python/python-zeroconf'. This would: > - Speed things up significantly, > - Make things simpler and, > - Fix this bug (where we get the wrong file extension on the moved binpkg) It would also fix Bug 877271 and Bug 791856 (In reply to Andrew Ammerlaan from comment #7) > Sort of related to this: > > I have noticed that when a package is renamed portage appears to *copy* the > binpkgs instead of moving them (and then does this with the wrong file > extension). This makes emerge --sync incredibly slow when there are a bunch > of binpkgs for that package. My PKGDIR is network mounted so it can be > shared between different computers. However, copying all these binpkgs over > the network is really really slow (global updates can take 15 to 20 > minutes). The binpkg included metadata of that package, so if you 'move dev-python/zeroconf dev-python/python-zeroconf-new' The old binpkg will not be recognized because the package name is dev-python/zeroconf. All the move / update is to use old package contents and update some info in the memory and write to new file, include the old xpak format. I generally think move operations are relatively rare, and how many binary packages of that package you have? > > Why don't we just keep it simple and move the binpkgs instead? That would be > so much faster. Besides, there is no point in keeping the binpkgs with the > old name, we won't be using them anyway. The code said ignore old binary packages, they should be cleaned by other tools. So I did not touch them. The bug has been closed via the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=445f10f4214c673f8fe0a9cc518c12767be4f159 commit 445f10f4214c673f8fe0a9cc518c12767be4f159 Author: Sheng Yu <syu.os@protonmail.com> AuthorDate: 2022-11-02 18:46:03 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-02 22:58:15 +0000 Use binpkg extensions and header to get format This deprecates the BINPKG_FORMAT in the metadata and Package index. Also fixed a bug that obsolete packages that have been moved may be attempted to be read during package searching. Closes: https://bugs.gentoo.org/877357 Closes: https://bugs.gentoo.org/877419 Signed-off-by: Sheng Yu <syu.os@protonmail.com> Closes: https://github.com/gentoo/portage/pull/928 Signed-off-by: Sam James <sam@gentoo.org> lib/_emerge/BinpkgExtractorAsync.py | 4 +- lib/_emerge/BinpkgFetcher.py | 32 ++------ lib/_emerge/Package.py | 1 - lib/_emerge/actions.py | 15 +++- lib/_emerge/depgraph.py | 15 +++- lib/_emerge/search.py | 6 +- lib/portage/binpkg.py | 23 ++++-- lib/portage/dbapi/bintree.py | 142 +++++++++++++++--------------------- lib/portage/gpkg.py | 4 + lib/portage/versions.py | 16 ---- lib/portage/xpak.py | 18 +++++ 11 files changed, 141 insertions(+), 135 deletions(-) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=a7dd39c1ae4a5ea4e3252ae8129fbd671c95d5f7 commit a7dd39c1ae4a5ea4e3252ae8129fbd671c95d5f7 Author: Sheng Yu <syu.os@protonmail.com> AuthorDate: 2022-11-08 22:52:46 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-08 23:07:46 +0000 Do not drop default PATH in the package index (avoid Packages regeneration) The new Packages index (introduced in 445f10f4214c673f8fe0a9cc518c12767be4f159) needs PATH but it got dropped later on when processing, so we would try to rebuild the index on every emerge call. This fixes regenerating Packages loop. (Note that this didn't affect a released version.) Fixes: 445f10f4214c673f8fe0a9cc518c12767be4f159 Bug: https://bugs.gentoo.org/877357 Bug: https://bugs.gentoo.org/877419 Signed-off-by: Sheng Yu <syu.os@protonmail.com> Closes: https://github.com/gentoo/portage/pull/934 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/dbapi/bintree.py | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=203e0d0083b5e7d8098785ec8862c02f6baf5534 commit 203e0d0083b5e7d8098785ec8862c02f6baf5534 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-11-20 03:04:40 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-20 03:05:40 +0000 sys-apps/portage: add 3.0.39 Closes: https://bugs.gentoo.org/365655 Closes: https://bugs.gentoo.org/757525 Closes: https://bugs.gentoo.org/871573 Closes: https://bugs.gentoo.org/875812 Closes: https://bugs.gentoo.org/875860 Closes: https://bugs.gentoo.org/877215 Closes: https://bugs.gentoo.org/877271 Closes: https://bugs.gentoo.org/877357 Closes: https://bugs.gentoo.org/877419 Closes: https://bugs.gentoo.org/873757 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.39.ebuild | 273 +++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+) |
