/me wonders why a gzip'ed file under /var/log/portage is bigger then its uncompressed and again gzip'ed sibbling : $ ls -l /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz -rw-rw---- 1 portage portage 69424 Nov 11 00:24 /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz $ cp /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz . $ gunzip app-emulation\:libvirt-1.0.0\:20121110-231620.log.gz $ gzip app-emulation\:libvirt-1.0.0\:20121110-231620.log $ ls -l app-emulation\:libvirt-1.0.0\:20121110-231620.log.gz -rw-r----- 1 tfoerste users 25097 Nov 19 20:46 app-emulation:libvirt-1.0.0:20121110-231620.log.gz FWIW: I stumbled over this while diging into an issue, that ark-4.9.3 does only show the first 4 lines of the compressed original log file. Furthermore /me wonders why the "file" command show "max. compression" for portage's compressed file : $ file /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz: gzip compressed data, last modified: Sun Nov 11 00:16:21 2012, max compression $ file app-emulation\:libvirt-1.0.0\:20121110-231620.log.gz app-emulation:libvirt-1.0.0:20121110-231620.log.gz: gzip compressed data, was "app-emulation:libvirt-1.0.0:20121110-231620.log", from Unix, last modified: Mon Nov 19 20:46:38 2012 if it is not really the maximal compression, isn't it ?
Created attachment 329978 [details] portage gziped file
Created attachment 329980 [details] common gziped file
Please post your `emerge --info' output in a comment.
AFAIK sys-apps/portage does not compress build logs at all. I run a cron job for that, instead.
(In reply to comment #3) > Please post your `emerge --info' output in a comment. tfoerste@n22 ~/devel/wireshark $ emerge --info portage Portage 2.1.11.31 (default/linux/x86/10.0/desktop/kde, gcc-4.6.3, glibc-2.15-r3, 3.6.7 i686) ================================================================= System Settings ================================================================= System uname: Linux-3.6.7-i686-Intel-R-_Core-TM-_i5-2540M_CPU_@_2.60GHz-with-gentoo-2.1 Timestamp of tree: Mon, 19 Nov 2012 18:45:01 +0000 ld GNU ld (GNU Binutils) 2.22 ccache version 3.1.7 [disabled] app-shells/bash: 4.2_p37 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.3-r2, 3.2.3 dev-util/ccache: 3.1.7 dev-util/cmake: 2.8.9 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.10.5 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.13, 2.68 sys-devel/automake: 1.9.6-r3, 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.5.4, 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.4 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo toralf ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="*" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=native -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=native -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--autounmask=n --keep-going=y --nospinner --tree --deep --quiet-build" FCFLAGS="-O2 -march=i686 -pipe" FEATURES="assume-digests binpkg-logs compress-build-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict test test-fail-continue unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -march=i686 -pipe" GENTOO_MIRRORS="http://mirror.leaseweb.com/gentoo/ http://gentoo.mirror.dkm.cz/pub/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://gentoo.mneisen.org/ http://www.gtlib.gatech.edu/pub/gentoo" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_GB" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://linux.rz.ruhr-uni-bochum.de/gentoo-portage" USE="X a52 aac acl acpi alsa apache2 berkdb bittorrent branding bzip2 cairo cdda cdr classic cli consolekit corefonts cracklib crypt css cups cxx dbus declarative dri dts dvb dvd dvdr emboss encode exif fam fastbuild ffmpeg firefox flac fontconfig fortran g3dvl gdbm gif git gmp gphoto2 gpm gtk hyphenation iconv ipv6 java jpeg kde kipi kqemu kvm lcms ldap libnotify lirc logrotate mad mbox mmx mng modules mp3 mp4 mpeg mudflap mysql mysqli ncurses nls nptl nsplugin ntfs ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pppd qemu qt3support qt4 rdesktop readline sdl session sna spell spice sse sse2 sse4_1 ssl ssse3 startup-notification svg tcpd thinkpad threads tiff tk transparent-proxy truetype udev udisks uml unicode upower usb uxa v4l vaapi vim-syntax virtualbox vlc vnc vorbis wxwidgets x264 x86 xa xcb xcomposite xinerama xml xpm xscreensaver xv xvid xvmc zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB" LIRC_DEVICES="devinput inputlirc" PHP_TARGETS="php5-4" PYTHON_TARGETS="python2_7 python3_2" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby19" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON ================================================================= Package Settings ================================================================= sys-apps/portage-2.1.11.31 was built with the following: USE="(consolekit) (ipc) (policykit) test -build -doc -epydoc (-pypy1_9) -python2 -python3 (-selinux) -xattr" LINGUAS="-pl"
(In reply to comment #4) > AFAIK sys-apps/portage does not compress build logs at all. I run a cron job > for that, instead. Errm : $ grep FEAT /etc/portage/make.conf FEATURES="parallel-fetch compress-build-logs test test-fail-continue"
/usr/lib/portage/bin/ebuild-helpers/ecompress: if [[ ${PORTAGE_COMPRESS_FLAGS+set} != "set" ]] ; then case ${PORTAGE_COMPRESS} in bzip2|gzip) PORTAGE_COMPRESS_FLAGS="-9";; [...] So it automatically picks the "best" (--best / -9) compression if you don't set the variable.
> URL: https://bugs.kde.org/show_bug.cgi?id=310370 Wouldn't it be app-arch/libarchive that is at fault here?
(In reply to comment #8) > > URL: https://bugs.kde.org/show_bug.cgi?id=310370 > > Wouldn't it be app-arch/libarchive that is at fault here? upstream says no, but has an interesting comemtn on it: https://bugs.kde.org/show_bug.cgi?id=310370#c2
FEATURE=compress-build-logs uses python's gzip.GzipFile class: http://docs.python.org/2/library/gzip.html Portage opens the log file multiple times in append mode, which may lead to sub-optimal compression. However, I think this behavior is acceptable.
Comment on attachment 329978 [details] portage gziped file this seems to double gzip'ed due to the web interface ?
(In reply to comment #10) > FEATURE=compress-build-logs uses python's gzip.GzipFile class: > > http://docs.python.org/2/library/gzip.html > > Portage opens the log file multiple times in append mode, which may lead to > sub-optimal compression. However, I think this behavior is acceptable. The problem is th efollowing : 1. Download the attached file 2. gunzip it 3. now do : "cut -c1-24732 app-emulation:libvirt-1.0.0:20121110-231620.log | gunzip -c" and you see nearly the whole log file, which let me wonders, what the rest of the 68 KB of the gzip'ed file does containtoo
(In reply to comment #11) > Comment on attachment 329978 [details] > portage gziped file > > this seems to double gzip'ed due to the web interface ? If do a local md5sum on the file then we'll be able to compare to the one that we download, to make sure it's not corrupted somehow.
(In reply to comment #13) > (In reply to comment #11) > > Comment on attachment 329978 [details] > > portage gziped file > > > > this seems to double gzip'ed due to the web interface ? > > If do a local md5sum on the file then we'll be able to compare to the one > that we download, to make sure it's not corrupted somehow. tfoerste@n22 ~/tmp $ md5sum /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz f3073679c8e5d0fa4748bac8372fb427 /var/log/portage/app-emulation:libvirt-1.0.0:20121110-231620.log.gz but /me beliefs, that the web server itself compressed it too just b/c bugzilla doesn't seem to realize the file type right (it often seems to not work FWIW)
(In reply to comment #12) > (In reply to comment #10) > > FEATURE=compress-build-logs uses python's gzip.GzipFile class: > > > > http://docs.python.org/2/library/gzip.html > > > > Portage opens the log file multiple times in append mode, which may lead to > > sub-optimal compression. However, I think this behavior is acceptable. > > The problem is th efollowing : > 1. Download the attached file > 2. gunzip it > 3. now do : "cut -c1-24732 app-emulation:libvirt-1.0.0:20121110-231620.log | > gunzip -c" > > and you see nearly the whole log file, which let me wonders, what the rest > of the 68 KB of the gzip'ed file does containtoo I experienced this issue with the file when I downloaded it with chromium (version 24.0.1312.14), and the file had md5sum afe297177bc1ba42252273076c113b85 which does not match the original file. When I downloaded the file again with wget, I got the correct md5sum f3073679c8e5d0fa4748bac8372fb427, and the file gunzips correctly.
4 years old ...
