Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 877419 - Binary package global update changes BINPKG_FORMAT extension
Summary: Binary package global update changes BINPKG_FORMAT extension
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 873757
Blocks:
  Show dependency tree
 
Reported: 2022-10-17 10:14 UTC by Andrew Ammerlaan
Modified: 2022-11-20 03:06 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Ammerlaan gentoo-dev 2022-10-17 10:14:12 UTC
After setting:

BINPKG_FORMAT="gpkg"

Portage will rename .xpak binpkgs to .gpkg.tar when renaming/moving them, this results in warnings about binpkgs with an incorrect extension:

Performing Global Updates
(Could take a couple of minutes if you have a lot of binary packages.)
.='update pass'  *='binary update'  #='/var/db update'  @='/var/db move'
s='/var/db SLOT move'  %='binary move'  S='binary SLOT move'
p='update /etc/portage/package.*'
/var/db/repos/gentoo/profiles/updates/4Q-2022.....
@ * waiting for lock on /home/andrew/NAS/binpkgs/.Packages.portage_lockfile ...                                                                                                                                                             [ ok ]


%%%%%%%%%%%%%@%%%%%%%%%%%%%%%%%%%%%%%%%pp########*********************************File /home/andrew/NAS/binpkgs/media-libs/libva/libva-2.10.0-1.gpkg.tar binpkg format mismatch, actual format: xpak*File 
<snip>
/home/andrew/NAS/binpkgs/media-libs/libva-intel-media-driver/libva-intel-media-driver-22.4.4-1.gpkg.tar binpkg format mismatch,actual format: xpak


Portage shouldn't change the extension of existing binpkgs.
Comment 1 Sheng Yu 2022-10-26 06:43:54 UTC
Hmmm, so we cannot trust binpkg_format in the index.
Use file extension to check format?
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2022-10-26 11:25:14 UTC
(In reply to Sheng Yu from comment #1)
> Hmmm, so we cannot trust binpkg_format in the index.
> Use file extension to check format?

But isn't the index correct prior to the update?  Or if Portage writes wrong format into the index in the first place, that seems to be the actual bug.
Comment 3 Sheng Yu 2022-10-26 23:38:55 UTC
(In reply to Michał Górny from comment #2)
> (In reply to Sheng Yu from comment #1)
> > Hmmm, so we cannot trust binpkg_format in the index.
> > Use file extension to check format?
> 
> But isn't the index correct prior to the update?  Or if Portage writes wrong
> format into the index in the first place, that seems to be the actual bug.

For what I understand, metadata like BINPKG_FORMAT in Package Index got deduplicate.
And when user change the BINPKG_FORMAT, it get overwritten by the new value and portage think the old untracked package is gpkg.

Another possible solution is we exclude BINPKG_FORMAT from global record and only add per package.
Comment 4 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-10-26 23:50:46 UTC
(In reply to Sheng Yu from comment #3)
> Another possible solution is we exclude BINPKG_FORMAT from global record and
> only add per package.

I think it should be per-package, tbh. It's not worth the risk of the confusion. Some people might want to keep old binpkgs for a very long time.
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2022-10-27 04:40:51 UTC
Ah, sorry, I didn't realize it was global.  In that case, perhaps detecting on filename (or header) is the best, and we shouldn't really be storing BINPKG_FORMAT in the first place.
Comment 6 Sheng Yu 2022-10-27 05:35:35 UTC
(In reply to Michał Górny from comment #5)
> Ah, sorry, I didn't realize it was global.  In that case, perhaps detecting
> on filename (or header) is the best, and we shouldn't really be storing
> BINPKG_FORMAT in the first place.

BINPKG_FORMAT exists because the binary package "path" may not exists in the index. I don't know in what case this could happen, but portage code shows if the path not recorded, it will guess the path. Maybe old behaviour?
Comment 7 Andrew Ammerlaan gentoo-dev 2022-10-27 11:24:14 UTC
Sort of related to this:

I have noticed that when a package is renamed portage appears to *copy* the binpkgs instead of moving them (and then does this with the wrong file extension). This makes emerge --sync incredibly slow when there are a bunch of binpkgs for that package. My PKGDIR is network mounted so it can be shared between different computers. However, copying all these binpkgs over the network is really really slow (global updates can take 15 to 20 minutes). 

Why don't we just keep it simple and move the binpkgs instead? That would be so much faster. Besides, there is no point in keeping the binpkgs with the old name, we won't be using them anyway.

I.e. when we have 'move dev-python/zeroconf dev-python/python-zeroconf' just do the equivalent of 'mv ${PKGDIR}/dev-python/zeroconf ${PKGDIR}/dev-python/python-zeroconf'. This would:
- Speed things up significantly,
- Make things simpler and,
- Fix this bug (where we get the wrong file extension on the moved binpkg)
Comment 8 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2022-10-27 11:24:22 UTC
(In reply to Sheng Yu from comment #6)
> BINPKG_FORMAT exists because the binary package "path" may not exists in the
> index. I don't know in what case this could happen, but portage code shows
> if the path not recorded, it will guess the path. Maybe old behaviour?

Yes, might be just backwards compatibility.  If the current version of Portage always includes PATH, then BINPKG_FORMAT is probably entirely redundant, since old versions of Portage don't feature GPKG support, so if it's missing, you can just assume XPAK (that said, you'd still have to deal with two possible XPAK suffixes).
Comment 9 Andrew Ammerlaan gentoo-dev 2022-10-27 11:27:58 UTC
(In reply to Andrew Ammerlaan from comment #7)
> I.e. when we have 'move dev-python/zeroconf dev-python/python-zeroconf' just
> do the equivalent of 'mv ${PKGDIR}/dev-python/zeroconf
> ${PKGDIR}/dev-python/python-zeroconf'. This would:
> - Speed things up significantly,
> - Make things simpler and,
> - Fix this bug (where we get the wrong file extension on the moved binpkg)

It would also fix Bug 877271 and Bug 791856
Comment 10 Sheng Yu 2022-10-27 21:01:52 UTC
(In reply to Andrew Ammerlaan from comment #7)
> Sort of related to this:
> 
> I have noticed that when a package is renamed portage appears to *copy* the
> binpkgs instead of moving them (and then does this with the wrong file
> extension). This makes emerge --sync incredibly slow when there are a bunch
> of binpkgs for that package. My PKGDIR is network mounted so it can be
> shared between different computers. However, copying all these binpkgs over
> the network is really really slow (global updates can take 15 to 20
> minutes). 

The binpkg included metadata of that package, so if you 
'move dev-python/zeroconf dev-python/python-zeroconf-new'
The old binpkg will not be recognized because the package name is dev-python/zeroconf.

All the move / update is to use old package contents and update some info in the memory and write to new file, include the old xpak format.

I generally think move operations are relatively rare, and how many binary packages of that package you have?

> 
> Why don't we just keep it simple and move the binpkgs instead? That would be
> so much faster. Besides, there is no point in keeping the binpkgs with the
> old name, we won't be using them anyway.

The code said ignore old binary packages, they should be cleaned by other tools.
So I did not touch them.
Comment 11 Larry the Git Cow gentoo-dev 2022-11-02 22:58:23 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=445f10f4214c673f8fe0a9cc518c12767be4f159

commit 445f10f4214c673f8fe0a9cc518c12767be4f159
Author:     Sheng Yu <syu.os@protonmail.com>
AuthorDate: 2022-11-02 18:46:03 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-11-02 22:58:15 +0000

    Use binpkg extensions and header to get format
    
    This deprecates the BINPKG_FORMAT in the metadata and Package index.
    
    Also fixed a bug that obsolete packages that have been moved may be
    attempted to be read during package searching.
    
    Closes: https://bugs.gentoo.org/877357
    Closes: https://bugs.gentoo.org/877419
    Signed-off-by: Sheng Yu <syu.os@protonmail.com>
    Closes: https://github.com/gentoo/portage/pull/928
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/_emerge/BinpkgExtractorAsync.py |   4 +-
 lib/_emerge/BinpkgFetcher.py        |  32 ++------
 lib/_emerge/Package.py              |   1 -
 lib/_emerge/actions.py              |  15 +++-
 lib/_emerge/depgraph.py             |  15 +++-
 lib/_emerge/search.py               |   6 +-
 lib/portage/binpkg.py               |  23 ++++--
 lib/portage/dbapi/bintree.py        | 142 +++++++++++++++---------------------
 lib/portage/gpkg.py                 |   4 +
 lib/portage/versions.py             |  16 ----
 lib/portage/xpak.py                 |  18 +++++
 11 files changed, 141 insertions(+), 135 deletions(-)
Comment 12 Larry the Git Cow gentoo-dev 2022-11-08 23:07:52 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=a7dd39c1ae4a5ea4e3252ae8129fbd671c95d5f7

commit a7dd39c1ae4a5ea4e3252ae8129fbd671c95d5f7
Author:     Sheng Yu <syu.os@protonmail.com>
AuthorDate: 2022-11-08 22:52:46 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-11-08 23:07:46 +0000

    Do not drop default PATH in the package index (avoid Packages regeneration)
    
    The new Packages index (introduced in 445f10f4214c673f8fe0a9cc518c12767be4f159)
    needs PATH but it got dropped later on when processing, so we would try
    to rebuild the index on every emerge call.
    
    This fixes regenerating Packages loop.
    
    (Note that this didn't affect a released version.)
    
    Fixes: 445f10f4214c673f8fe0a9cc518c12767be4f159
    Bug: https://bugs.gentoo.org/877357
    Bug: https://bugs.gentoo.org/877419
    Signed-off-by: Sheng Yu <syu.os@protonmail.com>
    Closes: https://github.com/gentoo/portage/pull/934
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/dbapi/bintree.py | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)
Comment 13 Larry the Git Cow gentoo-dev 2022-11-20 03:06:03 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=203e0d0083b5e7d8098785ec8862c02f6baf5534

commit 203e0d0083b5e7d8098785ec8862c02f6baf5534
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2022-11-20 03:04:40 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-11-20 03:05:40 +0000

    sys-apps/portage: add 3.0.39
    
    Closes: https://bugs.gentoo.org/365655
    Closes: https://bugs.gentoo.org/757525
    Closes: https://bugs.gentoo.org/871573
    Closes: https://bugs.gentoo.org/875812
    Closes: https://bugs.gentoo.org/875860
    Closes: https://bugs.gentoo.org/877215
    Closes: https://bugs.gentoo.org/877271
    Closes: https://bugs.gentoo.org/877357
    Closes: https://bugs.gentoo.org/877419
    Closes: https://bugs.gentoo.org/873757
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.39.ebuild | 273 +++++++++++++++++++++++++++++++++
 2 files changed, 274 insertions(+)