Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 264258 - Support of custom compressor (or at least support pigz/pbzip2)
Summary: Support of custom compressor (or at least support pigz/pbzip2)
Status: RESOLVED DUPLICATE of bug 335340
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-03-30 08:57 UTC by Timothy Redaelli (RETIRED)
Modified: 2010-09-06 05:55 UTC (History)
14 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
custom-bzip-command (0001-Bug-264258-Allow-for-a-custom-bzip-command.patch,6.22 KB, patch)
2010-04-16 19:42 UTC, Jeremy Olexa (darkside) (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Timothy Redaelli (RETIRED) gentoo-dev 2009-03-30 08:57:23 UTC
Since now SMP/multicore systems are commons I suggest to allow user to set custom unpacks commands.

So user can use pbzip2 / pigz instead of the (default) monothread version.

Now user must hack /usr/lib/portage/bin/ebuild.sh or he have to create the fake symlinks in /usr/local/bin to do it
Comment 1 Zac Medico gentoo-dev 2009-03-31 08:08:48 UTC
How about some variables that can be defined in make.conf, like PORTAGE_BZIP_COMMAND and PORTAGE_GZIP_COMMAND? That will allow you to choose the command and also pass some options to it.
Comment 2 Timothy Redaelli (RETIRED) gentoo-dev 2009-03-31 08:16:34 UTC
Ok, but maybe it's better to support some more cases:
PORTAGE_TAR_COMMAND (for star, bsdtar)
PORTAGE_BZIP2_COMMAND (for bzip2, pbzip2)
PORTAGE_GZIP_COMMAND (for gzip, pigz)
PORTAGE_LZMA_COMMAND (lzma-utils, xc... where is enough usable to put it in the tree)
Comment 3 Eric Thibodeau 2009-06-17 18:13:44 UTC
I vote for this (and thought I had actually put in a request for pbzip2 in portage :P ).

The variable approach seems decent enough.
Comment 4 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2009-07-12 19:02:26 UTC
In my opinion, creating variables for each possible compressor would be some kind of overkill.

I think better solution would be to enable extending unpack() through some kind of script within /etc/portage/. Such script would be called by unpack(), and it would either recognize the file and unpack it or return nonzero status to let unpack() try to handle the thing.

What is more important, such method could not only let user replace default compressors but also add new extensions/formats (but the second thing would be more usable with eclasses).
Comment 5 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-07-12 19:22:14 UTC
Portage could support an associative array PORTAGE_DECOMPRESSOR_COMMANDS whose indices would be names of standard decompressors.
Comment 6 boris64 2009-08-06 14:13:21 UTC
You have my vote.

Pbzip2 is much faster on a multicore cpu. I'm using
a simple "alias bzip2='pbzip2'" in my /etc/profile
for some time now without any problems or errors.
Comment 7 Eric Thibodeau 2009-08-06 15:15:48 UTC
ice little(In reply to comment #6)

>I'm using
> a simple "alias bzip2='pbzip2'" in my /etc/profile
> for some time now without any problems or errors.

Nice little hack now that its command prompt compatible (drop-in replacement). Anyone else care to comment, especially now that the bzip2 incompatibility in large files seems to have been taken care of.
Comment 8 Eric Thibodeau 2009-08-06 15:22:36 UTC
(In reply to comment #7)
> ice little(In reply to comment #6)
> 
> >I'm using
> > a simple "alias bzip2='pbzip2'" in my /etc/profile
> > for some time now without any problems or errors.
> 
> Nice little hack now that its command prompt compatible (drop-in replacement).
> Anyone else care to comment, especially now that the bzip2 incompatibility in
> large files seems to have been taken care of.
> 

Actually, as a reply to this naïve implementation I was about to do, I tend to keep all such modifications in /etc/profile.d/defaults.sh (sourced automatically by /etc/profile)

Here is what I added with some minimal sanity check (I copy this script to many machines)...as a temporary hack:

if [ -f `which pbzip2` ]
then
        alias bunzip2='pbunzip2'
        alias bzip2='pbzip2'
fi
Comment 9 boris64 2009-08-06 15:43:32 UTC
(In reply to comment #8)
>> ...
> 
> Actually, as a reply to this naïve implementation I was about to do, I tend to
> keep all such modifications in /etc/profile.d/defaults.sh (sourced
> automatically by /etc/profile)
> 
> Here is what I added with some minimal sanity check (I copy this script to many
> machines)...as a temporary hack:
> 
> if [ -f `which pbzip2` ]
> then
>         alias bunzip2='pbunzip2'
>         alias bzip2='pbzip2'
> fi
> 

Yeah, that's nice(r) ;)

I slightly changed this (i think "[ -x ];..." would be better here)
and copied it to /etc/profile.d/pbzip2.sh.

[code]
## pbzip2 instead of bzip2
if [ -x $(which pbzip2) ]; then
        alias bunzip2='pbunzip2'
        alias bzip2='pbzip2'
fi
[/code]

Can pigz be used the same way as substitute 
for gzip? Did anyone already try out?

Regards
Comment 10 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2009-08-06 18:10:07 UTC
I personally would prefer replacing the bz2 compressor through /usr/bin/{,un}bzip2 symlinks which I proposed in bug 277539. But to make it work with portage, bug 277537 (PATH order change) is needed too. And basically to allow portage to use pbzip2 instead of original bzip2, bug 277544 needs to be fixed first (command-line incompatibility).
Comment 11 Stephan Friedrichs 2009-09-24 12:11:50 UTC
Wouldn't it make sense to introduce virtual packages (one for p?bzip2 and one for pigz/gzip) and set a symlink via an eselect module? This way, all software (not only portage, but also the gnome archive manager and others) can depend on the virtual packages.
Comment 12 Holger Hoffstätte 2009-10-25 13:48:36 UTC
(In reply to comment #9)
> Can pigz be used the same way as substitute 
> for gzip? Did anyone already try out?

Replacing (or overriding via symlink) /bin/gzip with a symlink to /usr/bin/pigz works fine for regular usage since the other commands are just scripts that invoke gzip with the appropriate options (which is unfortunately not the case when replacing bzip2 with pbzip2).
Regular packing/unpacking with tar works fine; the showstopper is that emerge breaks when trying to uncompress the /var/tmp/.../distfile/ symlink to the real tarball in /usr/portage/distfiles, because (I think) pigz barfs on the symlink. :-(
Comment 13 Holger Hoffstätte 2009-10-25 15:15:55 UTC
(In reply to comment #12)
> Regular packing/unpacking with tar works fine; the showstopper is that emerge
> breaks when trying to uncompress the /var/tmp/.../distfile/ symlink to the real
> tarball in /usr/portage/distfiles, because (I think) pigz barfs on the symlink.

It was indeed a bug/difference in behaviour by pigz. I sent a patch upstream to mimic the exact gzip behaviour when uncompressing symlinks to stdout.
Comment 14 Holger Hoffstätte 2009-11-26 20:21:46 UTC
(In reply to comment #13)
> It was indeed a bug/difference in behaviour by pigz. I sent a patch upstream to
> mimic the exact gzip behaviour when uncompressing symlinks to stdout.

Fixes for pigz-2.1.5 can be found in bug#294752 - please test with USE=symlink. Me and a friend are using this "live", so far without any bad side effects; it requires /usr/bin to be on the $PATH before /bin, which I believe is the default for Gentoo.
Comment 15 Dror Levin (RETIRED) gentoo-dev 2009-12-31 16:25:15 UTC
Both pbzip2 and pigz now have a symlink USE flag which makes them replace bzip2/gzip. I guess that's at least a step forward.
Comment 16 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2010-04-16 19:42:28 UTC
Created attachment 228063 [details, diff]
custom-bzip-command

(In reply to comment #1)
> How about some variables that can be defined in make.conf, like
> PORTAGE_BZIP_COMMAND and PORTAGE_GZIP_COMMAND? That will allow you to choose
> the command and also pass some options to it.
> 

I took this approach. PORTAGE_BZIP_COMMAND
Comment 17 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2010-04-16 20:13:40 UTC
(In reply to comment #16)
> Created an attachment (id=228063) [details]
> custom-bzip-command

An unintentional side-effect to this patch is that unpack() can fail now. It is probably more correct to a) just use pbzip2[symlink] or b) create a virtual/bzip package. Anyway, it was a fun exercise to implement this but in practice..meh, now I know why this bug has been open for so long.
Comment 18 Brian Harring gentoo-dev 2010-04-16 21:11:00 UTC
(In reply to comment #17)
> An unintentional side-effect to this patch is that unpack() can fail now. It is
> probably more correct to a) just use pbzip2[symlink] or b) create a
> virtual/bzip package. Anyway, it was a fun exercise to implement this but in
> practice..meh, now I know why this bug has been open for so long.

Actually, virtual/bzip gets my vote- same for virtual/gzip.  I'm working on some code w/ the explicit intention of folding pbzip2/pigz/parallel xz/few others into one binary.  It's not done yet, but I definitely was intending on dropping compatible bzip2 links in place...
Comment 19 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2010-04-17 07:34:35 UTC
(In reply to comment #18)
> Actually, virtual/bzip gets my vote- same for virtual/gzip.  I'm working on
> some code w/ the explicit intention of folding pbzip2/pigz/parallel xz/few
> others into one binary.  It's not done yet, but I definitely was intending on
> dropping compatible bzip2 links in place...

I would like to notice that I've chosen /usr/bin as the pbzip2 symlink location for an additional reason - it relies on libbz2, which is installed in /usr/lib. Thus, it should be impossible to have only this 'additional' implementation of bzip2 installed without the basic one (or else we would have bzip2 available only after mounting /usr).

Moreover, even if we move all these files into / (and separate libbz2.so from bzip2 ebuild), I still wouldn't like to see only pbzip2 or some other implementation installed without the fallback basic one (especially that on my notebook I had issues with pbzip2 on high load).

This way, we get to the point that the basic implementation (at least of the most basic compressors) should always be installed, and then the virtual starts being useless -- as one of possible solutions would always be installed.
Comment 20 Matt Turner gentoo-dev 2010-07-07 05:17:41 UTC
(In reply to comment #11)
> Wouldn't it make sense to introduce virtual packages (one for p?bzip2 and one
> for pigz/gzip) and set a symlink via an eselect module? This way, all software
> (not only portage, but also the gnome archive manager and others) can depend on
> the virtual packages.

IMO, this is by far the best and most forward-looking idea, ... and no one has commented on it? Please provide feedback for this idea.
Comment 21 Eric Thibodeau 2010-07-08 18:32:49 UTC
(In reply to comment #20)
> IMO, this is by far the best and most forward-looking idea, ... and no one has
> commented on it? Please provide feedback for this idea.

Problem is that pbzip2 (and maybe pigz) isn't a drop-in replacement library but a drop-in replacement _program_ that _uses_ the bzip2 libs itself. One would have to provide a library interface to pbzip2 for such a thing to be even considered.
Comment 22 Zac Medico gentoo-dev 2010-09-06 05:55:03 UTC
This is fixed in portage-2.1.9. See bug 335340.

*** This bug has been marked as a duplicate of bug 335340 ***