The documentation on FEATURES=binpkg-multi-instance appears very minimal. make.conf(5) has some information about it, and documents that it suddenly generates files ending with .xpak suffix. This is also implicitly documented on https://wiki.gentoo.org/wiki/Binary_package_guide#The_PKGDIR_layout, be it without notion. However, on that same page, a description is made for gentoo binpkgs that they end with .tbz2, and that .xpak is the metadata stuffed inside them. The xpak(5) man-page (from sys-apps/portage) also documents that each binary package has an xpak attached to it. This clearly isn't the case any more. The reference to qxpak, qtbz2 etc. is strong and embedded in many places, yet the new fileformat .xpak is confusing. (I don't understand why there wasn't chosen for a name like .epkg, I've raised that concern long time ago.) I think that the documentation of xpak(5) to the very least should document the confusing situation of binary packages as well as only the xpak metadata now being referred to as .xpak. Further it seems that a notion should be made about whether or not the xpak metadata contains a reference as to what compressor is used to the main tar (assuming it is, given the description in make.conf(5)), and thus whether the compression type should be guessed (as is done currently). Finally, I'd like to reach a concensus on how the q-tools are going to deal with this. qxpak(1) certainly sounds like a misnomer, yet the functionality to extract metadata is still necessary. Perhaps something like folding qtbz2 and qxpak into one (already existing) applet qpkg, to try and avoid the terminology confusion? Reproducible: Always
There's also the GLEP 78 gpkg format that we'll hopefully be adopting soon: https://github.com/gentoo/portage/pull/562 Rename xpak to epkg is feasible. Could also rename it back to tbz2. ;-)
Wouldn't renaming at this point break for people who use this?
The file extension (tbz2 vs xpak vs epkg) can be changed without breaking existing binhost clients, because ${PKGDIR}/Packages includes the file extension in the PATH metadata for each package. For the same reason, FEATURES=binpkg-multi-instance is usable with clients that lack FEATURES=binpkg-multi-instance support, since they will simply save the files locally with tbz2 file extensions. So, we can change the local file layout and file extensions as much as necessary without breaking existing binhost clients, thanks to PATH metadata and the fact that the xpak file format is identical to the tbz2 file format when bz2 compression is used. The BINPKG_COMPRESS=bz2 setting is sticky for existing installs, which prevents surprise compression changes that would break unprepared binhost clients. The only reason that clients will need explicit support for gpkg is that it's a different file format. Since tbz2 and xpak packages are identical from the perspective of binhost clients, we're free to change the extension to epkg or anything else. (In reply to Fabian Groffen from comment #0) > Further it seems that a notion should be made about whether or not the xpak > metadata contains a reference as to what compressor is used to the main tar > (assuming it is, given the description in make.conf(5)), and thus whether > the compression type should be guessed (as is done currently). I think the portage.util.compression_probe function is a reasonable means to determine the compression. If we created another way to specify compression, then it would be redundant.
(In reply to Fabian Groffen from comment #0) > Finally, I'd like to reach a concensus on how the q-tools are going to deal > with this. qxpak(1) certainly sounds like a misnomer, yet the functionality > to extract metadata is still necessary. Perhaps something like folding > qtbz2 and qxpak into one (already existing) applet qpkg, to try and avoid > the terminology confusion? Both of these tools work with the tbz2/xpak format. If you want to rename tbz2/xpak to something else, then you can consider that when renaming tools.
For gpkg, compression of internal members is specified by file extensions: https://www.gentoo.org/glep/glep-0078.html#archive-member-compression
(In reply to Zac Medico from comment #3) > The only reason that clients will need explicit support for gpkg is that > it's a different file format. Since tbz2 and xpak packages are identical > from the perspective of binhost clients, we're free to change the extension > to epkg or anything else. The main costs associated with a file extension change are: 1) Any tools that make assumptions about local file extensions will have to adapt. Since eclean-pkg does not make any assumptions about file extensions, it will not require any changes. 2) The change has to be well documented and communicated, and the old layout should be sticky for existing installs, like FEATURES=-binpkg-multi-instance since bug 772785.
Ok, so it seems the least we can do here is decide whether or not we want to change the .xpak extension for multi-binpkg to something else, as not to confuse predating documentation and behaviour for xpak meta archives. From GLEP 78 it seems that xpak is used when compression isn't bzip2, while if it would be bzip2, the old tbz2 is used. Is this really true/how it is implemented currently? Given that we want to replace this with gpkg.tar format, it seems little worth now to change this and even make a bigger mess/confusion. qxpak/qtbz2 will simply be deprecated tools as soon as the new format is out. As for GLEP 78, I'm assuming this opens the road to add "deltas" to the same gpkg in order to create something like a single package containing USE=X and a delta for USE=X+n or something? Or perhaps versions? Or is the package container format considered completely a black box, and will the multi-instance repo remain the same, but with a different storage container? Thanks
(In reply to Zac Medico from comment #3) > (In reply to Fabian Groffen from comment #0) > > Further it seems that a notion should be made about whether or not the xpak > > metadata contains a reference as to what compressor is used to the main tar > > (assuming it is, given the description in make.conf(5)), and thus whether > > the compression type should be guessed (as is done currently). > > I think the portage.util.compression_probe function is a reasonable means to > determine the compression. If we created another way to specify compression, > then it would be redundant. Yes, qmerge probes the compression header too, yet not all compressors use a magic identification (but then, who uses brotli anymore?)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=b094ba47368bb9b16fb17ee47a54644a09860823 commit b094ba47368bb9b16fb17ee47a54644a09860823 Author: Rin Cat (鈴猫) <dev@rincat.ch> AuthorDate: 2022-01-23 23:23:14 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-01-24 01:17:10 +0000 */*: GPKG (new binpkg format) Support (GLEP 78, provisionally) Bug: https://bugs.gentoo.org/500630 Bug: https://bugs.gentoo.org/659864 Bug: https://bugs.gentoo.org/672672 Bug: https://bugs.gentoo.org/773259 Bug: https://bugs.gentoo.org/820578 Signed-off-by: Rin Cat (鈴猫) <dev@rincat.ch> Closes: https://github.com/gentoo/portage/pull/562 Signed-off-by: Sam James <sam@gentoo.org> MANIFEST.in | 3 + bin/gpkg-helper.py | 71 + bin/misc-functions.sh | 67 +- bin/quickpkg | 108 +- cnf/make.conf.example | 36 + cnf/make.globals | 21 + lib/_emerge/Binpkg.py | 13 +- lib/_emerge/BinpkgExtractorAsync.py | 9 + lib/_emerge/BinpkgFetcher.py | 31 +- lib/_emerge/EbuildBinpkg.py | 16 +- lib/_emerge/EbuildPhase.py | 33 +- lib/_emerge/Package.py | 1 + lib/_emerge/actions.py | 44 +- lib/_emerge/depgraph.py | 30 +- lib/portage/__init__.py | 1 + lib/portage/binpkg.py | 56 + lib/portage/const.py | 12 + lib/portage/dbapi/bintree.py | 445 ++++- lib/portage/dbapi/vartree.py | 77 +- lib/portage/exception.py | 20 + lib/portage/gpg.py | 106 + lib/portage/gpkg.py | 2015 ++++++++++++++++++++ .../package/ebuild/_config/special_env_vars.py | 1 + lib/portage/package/ebuild/config.py | 10 + lib/portage/package/ebuild/doebuild.py | 13 + .../06B3A311BD775C280D22A9305D90EA06352177F6.rev | 37 + .../8DEDA2CDED49C8809287B89D8812797DDF1DD192.rev | 37 + .../273B030399E7BEA66A9AD42216DE7CA17BA5D42E.key | Bin 0 -> 2055 bytes .../C99796FB85B0C3DF03314A11B5850C51167D6282.key | Bin 0 -> 2055 bytes lib/portage/tests/.gnupg/pubring.kbx | Bin 0 -> 2774 bytes lib/portage/tests/.gnupg/trustdb.gpg | Bin 0 -> 1360 bytes lib/portage/tests/__init__.py | 37 +- lib/portage/tests/emerge/test_simple.py | 47 +- lib/portage/tests/gpkg/__init__.py | 2 + lib/portage/tests/gpkg/__test__.py | 0 lib/portage/tests/gpkg/test_gpkg_checksum.py | 396 ++++ lib/portage/tests/gpkg/test_gpkg_gpg.py | 398 ++++ .../tests/gpkg/test_gpkg_metadata_update.py | 59 + lib/portage/tests/gpkg/test_gpkg_metadata_url.py | 173 ++ lib/portage/tests/gpkg/test_gpkg_path.py | 390 ++++ lib/portage/tests/gpkg/test_gpkg_size.py | 58 + lib/portage/tests/gpkg/test_gpkg_stream.py | 112 ++ lib/portage/tests/resolver/ResolverPlayground.py | 55 +- .../test_build_id_profile_format.py | 50 +- .../binpkg_multi_instance/test_rebuilt_binaries.py | 44 +- .../tests/resolver/soname/test_autounmask.py | 38 +- .../tests/resolver/soname/test_downgrade.py | 84 +- .../tests/resolver/soname/test_or_choices.py | 39 +- .../tests/resolver/soname/test_reinstall.py | 40 +- .../tests/resolver/soname/test_skip_update.py | 39 +- .../soname/test_slot_conflict_reinstall.py | 135 +- .../resolver/soname/test_slot_conflict_update.py | 38 +- .../tests/resolver/soname/test_soname_provided.py | 45 +- .../tests/resolver/soname/test_unsatisfiable.py | 40 +- .../tests/resolver/soname/test_unsatisfied.py | 40 +- .../tests/resolver/test_autounmask_binpkg_use.py | 38 +- lib/portage/tests/resolver/test_bdeps.py | 44 +- .../resolver/test_binary_pkg_ebuild_visibility.py | 35 +- lib/portage/tests/resolver/test_changed_deps.py | 41 +- ...test_complete_if_new_subslot_without_revbump.py | 40 +- .../resolver/test_disjunctive_depend_order.py | 34 +- lib/portage/tests/resolver/test_multirepo.py | 62 +- .../test_regular_slot_change_without_revbump.py | 41 +- lib/portage/tests/resolver/test_simple.py | 34 +- lib/portage/tests/resolver/test_slot_abi.py | 113 +- .../tests/resolver/test_slot_abi_downgrade.py | 77 +- .../resolver/test_slot_change_without_revbump.py | 40 +- .../resolver/test_slot_operator_autounmask.py | 40 +- .../tests/resolver/test_slot_operator_bdeps.py | 74 +- .../tests/resolver/test_slot_operator_rebuild.py | 40 +- .../tests/resolver/test_slot_operator_unsolved.py | 41 +- lib/portage/tests/resolver/test_useflags.py | 37 +- lib/portage/tests/runTests.py | 12 + lib/portage/tests/update/test_move_ent.py | 129 +- lib/portage/tests/update/test_move_slot_ent.py | 139 +- lib/portage/tests/update/test_update_dbentry.py | 182 +- lib/portage/util/_urlopen.py | 4 +- lib/portage/versions.py | 16 + man/make.conf.5 | 80 + .../06B3A311BD775C280D22A9305D90EA06352177F6.rev | 37 + .../8DEDA2CDED49C8809287B89D8812797DDF1DD192.rev | 37 + .../273B030399E7BEA66A9AD42216DE7CA17BA5D42E.key | Bin 0 -> 2055 bytes .../C99796FB85B0C3DF03314A11B5850C51167D6282.key | Bin 0 -> 2055 bytes repoman/lib/repoman/tests/.gnupg/pubring.kbx | Bin 0 -> 2774 bytes repoman/lib/repoman/tests/.gnupg/trustdb.gpg | Bin 0 -> 1360 bytes repoman/lib/repoman/tests/runTests.py | 10 + repoman/setup.py | 12 +- setup.py | 14 +- 88 files changed, 6272 insertions(+), 903 deletions(-)
