Summary: | portage binary packages, option to use gzip or xz instead of bzip2 | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Ross Capdeville <ross> |
Component: | Binary packages support | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | enhancement | CC: | alexander, dberkholz, dschridde+gentoobugs, flameeyes, fturco, gart, gentoo-bugs, gentoo-bugs, karsten.elfenbein, kfm, marco, nikoli, pacho, paul-gentoo, paul, pva, wschlich, zerochaos |
Priority: | High | Keywords: | InVCS |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
URL: | http://archives.gentoo.org/gentoo-portage-dev/msg_6ead086db61f438bfbac01c97d3da390.xml | ||
See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=219129 https://bugs.gentoo.org/show_bug.cgi?id=593916 https://bugs.gentoo.org/show_bug.cgi?id=349582 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | 150031 | ||
Bug Blocks: | 627566 | ||
Attachments: | Patch to support compression option gzip in portage 2.1.2.2 |
Description
Ross Capdeville
2006-08-02 13:35:31 UTC
*** Bug 142581 has been marked as a duplicate of this bug. *** (In reply to comment #0) > What was the reasoning behind this decision to hardcode the compression > utility? It's probably just shortsightedness. > Is there an easy way to change it? Is there a more complicated way to > change it? My company is possibly willing to do the programming if we had some > guidance. Do you think this could be useful to others? There probably aren't very many code changes required to fix this. We'd only have to fix any compression/decompressin code as well as code that relies on the tbz2 extension . Lets start with the files that would need to be changed. First, the mechanism to create packages should be able to use gzip or bzip2. Would you want to feed this off an environment varaible, defaulting to bzip2? If not how would we know what format to make/fetch packages? bin/misc-functions.sh Function dyn_package() pym/getbinpkg.py Hardcoded .tbz2 pym/portage.py Hardcoded .tbz2, also spawns bzip2 xpak.py - may magically work after the above items are changed? (In reply to comment #3) > xpak.py - may magically work after the above items are changed? Yes, xpak does not need to be concerned with the compression because the xpak segment is appended to the tail of the file and is uncompressed. Should portage only look for the extension corresponding the the currently selected compression type? That way it won't have to go scanning for both tbz2 and tgz at the same time (potentially finding two versions of the same package). We can create a tool that converts packages from one type to another. do we want to allow people to select different compression types for everything or just have 1 (PORTAGE_COMPRESSION) ? this would affect documentation as well ... *** Bug 166027 has been marked as a duplicate of this bug. *** Created attachment 125913 [details, diff]
Patch to support compression option gzip in portage 2.1.2.2
This patch applied to the root directory tree of a gentoo installation patches portage to accept the PORTAGE_COMPRESS="gzip" option for packages. if this option is set, it creates packages compressed with gzip and also installs packages compressed with gzip. Apply patch in your / directory as root against portage 2.1.2.2 with the command:
patch -p1 < 142579.patch
erp, that doesnt really scale (and it has some whitespace damage). what about the next guy who wants to use a different compression scheme ? in fact, going by your original demands (pure decompression speed), i think using lzma would make installation times even faster. some tips: - abstract the "binary package extension" into a function so everything can be centralized there - instead of parsing ${PORTAGE_COMPRESS} directly to figure out the extension, just use `ecompress --suffix` - instead of adding custom decompression routines, reuse the unpack() function i'm pretty sure with these three fundamental changes, you would basically get complete flexibility. want .tlzma ? no problem, merely set: PORTAGE_COMPRESS="lzma" want .tbz2 ? no problem, merely set: PORTAGE_COMPRESS="bzip2" want .tgz ? no problem, merely set: PORTAGE_COMPRESS="gzip" i think you get the idea :) has anyone made any further progress on this? A few months ago we had a discussion about ways to handle it in terms of PORTAGE_BINHOST support: http://archives.gentoo.org/gentoo-portage-dev/msg_6ead086db61f438bfbac01c97d3da390.xml Has there been any movement on this recently? It would be great to have the option to build .gz or .xz packages even just for local use (bzip2 is really slow and xz -2 or xz -3 outperforms it on compression time, compressed size, AND decompression time iirc). *** Bug 292302 has been marked as a duplicate of this bug. *** I have a forward-compatibility patch for gzip and xz compression in the following branch: https://github.com/zmedico/portage/tree/binpkg-multi-decompression I've posted it for review here: http://thread.gmane.org/gmane.linux.gentoo.portage.devel/5074 (In reply to Zac Medico from comment #13) > I have a forward-compatibility patch for gzip and xz compression in the > following branch: > > https://github.com/zmedico/portage/tree/binpkg-multi-decompression Decompression support is in the master branch now: https://github.com/gentoo/portage/commit/6b3d262e6316073a2a3be81086c05891d970ae2a I'm planning to add support for compression selection after support for a compression-independent file layout has been merged (see bug #150031). All that's left to do is to provide a way to select the compression type. Since portage-2.2.16, portage probes the compression type and maps that to a decompressor. (In reply to Zac Medico from comment #15) > All that's left to do is to provide a way to select the compression type. > Since portage-2.2.16, portage probes the compression type and maps that to a > decompressor. There is any option to enable creating binpkg with gzip or lzma ? *** Bug 380925 has been marked as a duplicate of this bug. *** Hi, any update regarding how to select the compression type of binpkgs ? I'm missing how to do that or it is a feature still in development? It's not implemented yet, but it's high on my todo list. In the master branch, we now have support to configure binary package compression: https://gitweb.gentoo.org/proj/portage.git/commit/?id=cff2c0149142843316e1851c2e73bcec30f08471 The relevant variables are now called BINPKG_COMPRESS AND BINPKG_COMPRESS_FLAGS: https://gitweb.gentoo.org/proj/portage.git/commit/?id=2c7d38b9512609f6828cbba1066f2b3b2d9144bf |