Since now SMP/multicore systems are commons I suggest to allow user to set custom unpacks commands. So user can use pbzip2 / pigz instead of the (default) monothread version. Now user must hack /usr/lib/portage/bin/ebuild.sh or he have to create the fake symlinks in /usr/local/bin to do it
How about some variables that can be defined in make.conf, like PORTAGE_BZIP_COMMAND and PORTAGE_GZIP_COMMAND? That will allow you to choose the command and also pass some options to it.
Ok, but maybe it's better to support some more cases: PORTAGE_TAR_COMMAND (for star, bsdtar) PORTAGE_BZIP2_COMMAND (for bzip2, pbzip2) PORTAGE_GZIP_COMMAND (for gzip, pigz) PORTAGE_LZMA_COMMAND (lzma-utils, xc... where is enough usable to put it in the tree)
I vote for this (and thought I had actually put in a request for pbzip2 in portage :P ). The variable approach seems decent enough.
In my opinion, creating variables for each possible compressor would be some kind of overkill. I think better solution would be to enable extending unpack() through some kind of script within /etc/portage/. Such script would be called by unpack(), and it would either recognize the file and unpack it or return nonzero status to let unpack() try to handle the thing. What is more important, such method could not only let user replace default compressors but also add new extensions/formats (but the second thing would be more usable with eclasses).
Portage could support an associative array PORTAGE_DECOMPRESSOR_COMMANDS whose indices would be names of standard decompressors.
You have my vote. Pbzip2 is much faster on a multicore cpu. I'm using a simple "alias bzip2='pbzip2'" in my /etc/profile for some time now without any problems or errors.
ice little(In reply to comment #6) >I'm using > a simple "alias bzip2='pbzip2'" in my /etc/profile > for some time now without any problems or errors. Nice little hack now that its command prompt compatible (drop-in replacement). Anyone else care to comment, especially now that the bzip2 incompatibility in large files seems to have been taken care of.
(In reply to comment #7) > ice little(In reply to comment #6) > > >I'm using > > a simple "alias bzip2='pbzip2'" in my /etc/profile > > for some time now without any problems or errors. > > Nice little hack now that its command prompt compatible (drop-in replacement). > Anyone else care to comment, especially now that the bzip2 incompatibility in > large files seems to have been taken care of. > Actually, as a reply to this naïve implementation I was about to do, I tend to keep all such modifications in /etc/profile.d/defaults.sh (sourced automatically by /etc/profile) Here is what I added with some minimal sanity check (I copy this script to many machines)...as a temporary hack: if [ -f `which pbzip2` ] then alias bunzip2='pbunzip2' alias bzip2='pbzip2' fi
(In reply to comment #8) >> ... > > Actually, as a reply to this naïve implementation I was about to do, I tend to > keep all such modifications in /etc/profile.d/defaults.sh (sourced > automatically by /etc/profile) > > Here is what I added with some minimal sanity check (I copy this script to many > machines)...as a temporary hack: > > if [ -f `which pbzip2` ] > then > alias bunzip2='pbunzip2' > alias bzip2='pbzip2' > fi > Yeah, that's nice(r) ;) I slightly changed this (i think "[ -x ];..." would be better here) and copied it to /etc/profile.d/pbzip2.sh. [code] ## pbzip2 instead of bzip2 if [ -x $(which pbzip2) ]; then alias bunzip2='pbunzip2' alias bzip2='pbzip2' fi [/code] Can pigz be used the same way as substitute for gzip? Did anyone already try out? Regards
I personally would prefer replacing the bz2 compressor through /usr/bin/{,un}bzip2 symlinks which I proposed in bug 277539. But to make it work with portage, bug 277537 (PATH order change) is needed too. And basically to allow portage to use pbzip2 instead of original bzip2, bug 277544 needs to be fixed first (command-line incompatibility).
Wouldn't it make sense to introduce virtual packages (one for p?bzip2 and one for pigz/gzip) and set a symlink via an eselect module? This way, all software (not only portage, but also the gnome archive manager and others) can depend on the virtual packages.
(In reply to comment #9) > Can pigz be used the same way as substitute > for gzip? Did anyone already try out? Replacing (or overriding via symlink) /bin/gzip with a symlink to /usr/bin/pigz works fine for regular usage since the other commands are just scripts that invoke gzip with the appropriate options (which is unfortunately not the case when replacing bzip2 with pbzip2). Regular packing/unpacking with tar works fine; the showstopper is that emerge breaks when trying to uncompress the /var/tmp/.../distfile/ symlink to the real tarball in /usr/portage/distfiles, because (I think) pigz barfs on the symlink. :-(
(In reply to comment #12) > Regular packing/unpacking with tar works fine; the showstopper is that emerge > breaks when trying to uncompress the /var/tmp/.../distfile/ symlink to the real > tarball in /usr/portage/distfiles, because (I think) pigz barfs on the symlink. It was indeed a bug/difference in behaviour by pigz. I sent a patch upstream to mimic the exact gzip behaviour when uncompressing symlinks to stdout.
(In reply to comment #13) > It was indeed a bug/difference in behaviour by pigz. I sent a patch upstream to > mimic the exact gzip behaviour when uncompressing symlinks to stdout. Fixes for pigz-2.1.5 can be found in bug#294752 - please test with USE=symlink. Me and a friend are using this "live", so far without any bad side effects; it requires /usr/bin to be on the $PATH before /bin, which I believe is the default for Gentoo.
Both pbzip2 and pigz now have a symlink USE flag which makes them replace bzip2/gzip. I guess that's at least a step forward.
Created attachment 228063 [details, diff] custom-bzip-command (In reply to comment #1) > How about some variables that can be defined in make.conf, like > PORTAGE_BZIP_COMMAND and PORTAGE_GZIP_COMMAND? That will allow you to choose > the command and also pass some options to it. > I took this approach. PORTAGE_BZIP_COMMAND
(In reply to comment #16) > Created an attachment (id=228063) [details] > custom-bzip-command An unintentional side-effect to this patch is that unpack() can fail now. It is probably more correct to a) just use pbzip2[symlink] or b) create a virtual/bzip package. Anyway, it was a fun exercise to implement this but in practice..meh, now I know why this bug has been open for so long.
(In reply to comment #17) > An unintentional side-effect to this patch is that unpack() can fail now. It is > probably more correct to a) just use pbzip2[symlink] or b) create a > virtual/bzip package. Anyway, it was a fun exercise to implement this but in > practice..meh, now I know why this bug has been open for so long. Actually, virtual/bzip gets my vote- same for virtual/gzip. I'm working on some code w/ the explicit intention of folding pbzip2/pigz/parallel xz/few others into one binary. It's not done yet, but I definitely was intending on dropping compatible bzip2 links in place...
(In reply to comment #18) > Actually, virtual/bzip gets my vote- same for virtual/gzip. I'm working on > some code w/ the explicit intention of folding pbzip2/pigz/parallel xz/few > others into one binary. It's not done yet, but I definitely was intending on > dropping compatible bzip2 links in place... I would like to notice that I've chosen /usr/bin as the pbzip2 symlink location for an additional reason - it relies on libbz2, which is installed in /usr/lib. Thus, it should be impossible to have only this 'additional' implementation of bzip2 installed without the basic one (or else we would have bzip2 available only after mounting /usr). Moreover, even if we move all these files into / (and separate libbz2.so from bzip2 ebuild), I still wouldn't like to see only pbzip2 or some other implementation installed without the fallback basic one (especially that on my notebook I had issues with pbzip2 on high load). This way, we get to the point that the basic implementation (at least of the most basic compressors) should always be installed, and then the virtual starts being useless -- as one of possible solutions would always be installed.
(In reply to comment #11) > Wouldn't it make sense to introduce virtual packages (one for p?bzip2 and one > for pigz/gzip) and set a symlink via an eselect module? This way, all software > (not only portage, but also the gnome archive manager and others) can depend on > the virtual packages. IMO, this is by far the best and most forward-looking idea, ... and no one has commented on it? Please provide feedback for this idea.
(In reply to comment #20) > IMO, this is by far the best and most forward-looking idea, ... and no one has > commented on it? Please provide feedback for this idea. Problem is that pbzip2 (and maybe pigz) isn't a drop-in replacement library but a drop-in replacement _program_ that _uses_ the bzip2 libs itself. One would have to provide a library interface to pbzip2 for such a thing to be even considered.
This is fixed in portage-2.1.9. See bug 335340. *** This bug has been marked as a duplicate of bug 335340 ***
