Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 248089

Summary: compression support for portage 2.2 binary packages index file ('Packages')
Product: Portage Development Reporter: Wolfram Schlich (RETIRED) <wschlich>
Component: Binary packages supportAssignee: Portage team <dev-portage>
Status: CONFIRMED ---    
Severity: enhancement CC: jsbronder, syu.os
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=877241
https://github.com/pkgcore/pkgcore/issues/60
Whiteboard:
Package list:
Runtime testing required: ---

Description Wolfram Schlich (RETIRED) gentoo-dev 2008-11-21 23:45:51 UTC 
downloading a 500kb (uncompressed) Packages file from
the binhost over a somewhat slow connection (UMTS in my case)
is a PITA, so can we please add compression support to the
code handling that file (creation on the binhost and
handling on the client)?
Comment 1 Zac Medico gentoo-dev 2008-11-22 08:35:32 UTC 
How about if you configure your webserver for gzip compression and we add support to portage for that? I found some sample code that we can use here:

http://diveintopython.org/http_web_services/gzip_compression.html
Comment 2 Wolfram Schlich (RETIRED) gentoo-dev 2008-11-22 13:05:45 UTC 
Well, that would be a start, but remember that portage also supports FTP
for fetching stuff from the binhost... wouldn't it be nice if that
compression would be transfer protocol agnostic?
Comment 3 Zac Medico gentoo-dev 2008-11-22 19:23:42 UTC 
I suppose we can have portage create a Packages.gz and download that one instead of the uncompressed version.
Comment 4 Fabian Groffen gentoo-dev 2008-11-22 19:28:22 UTC 
would make more sense to use bzip2 instead, since the packages are compressed with the same algorithm
Comment 5 Wolfram Schlich (RETIRED) gentoo-dev 2008-11-23 11:39:12 UTC 
(In reply to comment #4)
> would make more sense to use bzip2 instead, since the packages are compressed
> with the same algorithm

Shouldn't the algorithm for both (tarballs and Packages) be
configurable anyway?! :) Something line PORTAGE_COMPRESS...
Could be named PORTAGE_BINHOST_COMPRESS and would need to
be specified on both the binhost and the clients in order
to use a specific compression method.
It could default to gzip though.
What do you think?
Comment 6 Fabian Groffen gentoo-dev 2008-11-23 11:43:47 UTC 
(In reply to comment #5)
> It could default to gzip though.
> What do you think?

lzma, because it is compress once, decompress many times, IMO.  And lzma is a much better compressor too.

A configurable compressor would be pretty nice but causes some problems in distribution (think of hosts not having the right decompressor).  However, defaulting to gzip sounds unreasonable to me, since Gentoo has been using bzip2 for ages, and it is a better compressor than gzip.  And pbzip2 is nice too...:wq
Comment 7 Wolfram Schlich (RETIRED) gentoo-dev 2009-02-13 14:40:40 UTC 
Any news on this one? :) TIA!