For a change, 'emerge --info' first, as it most probably will matter here: Portage 2.2_rc62 (default/linux/x86/10.0/desktop, gcc-4.4.3, glibc-2.11-r1, 2.6.32-gentoo i686) ================================================================= System uname: Linux-2.6.32-gentoo-i686-AMD_Athlon-tm-_XP_2500+-with-gentoo-2.0.1 Timestamp of tree: Sun, 14 Feb 2010 21:15:01 +0000 ccache version 2.4 [enabled] app-shells/bash: 4.1_p2 dev-java/java-config: 2.1.10 dev-lang/python: 2.6.4-r1 dev-python/pycrypto: 2.1.0 dev-util/ccache: 2.4-r7 dev-util/cmake: 2.8.0 sys-apps/baselayout: 2.0.1 sys-apps/openrc: 0.6.0-r1 sys-apps/sandbox: 2.2 sys-devel/autoconf: 2.13, 2.65 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2, 1.11 sys-devel/binutils: 2.19.1-r1, 2.20 sys-devel/gcc: 4.1.2, 4.3.4, 4.4.3 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6b virtual/os-headers: 2.6.30-r1 ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=athlon -mtune=athlon -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=athlon -mtune=athlon -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests ccache confcache distlocks fixpackages metadata-transfer news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch userpriv usersandbox" GENTOO_MIRRORS="http://gentoo.osuosl.org/ ftp://distro.ibiblio.org/pub/linux/distributions/gentoo/ http://distro.ibiblio.org/pub/linux/distributions/gentoo/ ftp://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ http://src.gentoo.pl http://gentoo.prz.rzeszow.pl http://gentoo.zie.pg.gda.pl http://gentoo.mirror.pw.edu.pl/ " LANG="pl_PL.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--sort-common -Wl,-z,relro" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage/layman/gnome /usr/local/portage/layman/science /usr/local/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="X Xaw3d a52 aac acl acpi afs alsa apache2 apm audiofile avi bash-completion berkdb bidi bluetooth branding bzip2 bzlib cairo caps cdr cjk cli consolekit cracklib crypt cscope curl cxx dbus directfb doc dri dts dvd dvdr dvdread eds emboss encode evo fam fbcon ffmpeg firefox flac fortran ftp gdbm ggi gif glut gmp gnutls gpm gstreamer gtk gtk2 gtkhtml guile hal iconv idn imagemagick imap ipv6 java javascript joystick jpeg jpeg2k kerberos lcms ldap leim libnotify libwww mad maildir matroska mikmod mmap mmx mng modules motif mp3 mp4 mpeg mudflap mysql ncurses nis nls nptl nptlonly nvidia ogg opengl openmp pam pcre pdf perl png posix ppds pppd python qt qt3support qt4 quicktime readline reflection ruby sasl sdl session sharedext slp sndfile sockets speex spell spl sse ssl startup-notification svg sysfs tcpd tetex theora thunar tiff tokenizer truetype unicode usb v4l vcd vhosts vorbis win32codecs wmf wxwidgets x264 x86 xcb xface xinerama xml xml2 xorg xosd xulrunner xv xvid zlib" ALSA_CARDS="dummy virmidi hda-intel hrtimer intel8x0 mpu401" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="worker" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev linuxinput ps2mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="vesa fbdev radeon" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS Now, what may matter, is that I'm using 9999 of xz-utils. For some strange reason, portage fails at unpacking tar.lzma archives, even though tar.bz2, tar.gz and tar.xz work just fine. The error is: lzma: /var/tmp/portage/sys-devel/libtool-2.2.6b/distdir/libtool-2.2.6b.tar.lzma: Unexpected end of input tar: This doesn't look like a tar archive (libtool is just an example here) However the archives are correct and they unpack manually. What's more, if I change in ebuild.sh '_unpack_tar lzma' to 'tar xoJf "$srcdir$x" || die "$myfail"' (OK, tar 1.22, but not that important), it works just fine. It's really hard to tell when it started to fail, as very few packages use tar.lzma. This was discovered due to jpeg rebuild (netpbm).
Duplicate of bug 293580? Out of memory.
While I do have very little memory, it's very unlikely this is the same problem - message is clearly different. The archives unpack correctly with default settings. It works on the commandline - I mean the version from '_unpack_tar lzma' (lzma -dc "$srcdir$x" | tar xof -), it only fails, while invoked by portage.
(In reply to comment #2) > '_unpack_tar lzma' (lzma -dc "$srcdir$x" | tar xof -), > it only fails, while invoked by portage. You need to check both elements of ${PIPESTATUS[@]} or else you can't be sure. It unpacks successfully for me, so I suspect that you are experiencing bug 293580.
No luck, XZ_OPTS doesn't affect if at all. Now, there's a funny thing: while tar with -J works, lzma -dc -- "${s}" | tar xof - || die fails. ...makes no sense to me. What exactly does that PIPESTATUS var reports (and how) ? Cause on the commandline i.e. lzma -dc -- netpbm-10.48.00.tar.lzma | tar xof - ; echo $PIPESTATUS prints '0' and unpacks correctly. a problem with shell ?
(In reply to comment #4) > What exactly does that PIPESTATUS var reports (and how) ? > > Cause on the commandline i.e. > lzma -dc -- netpbm-10.48.00.tar.lzma | tar xof - ; echo $PIPESTATUS > > prints '0' and unpacks correctly. PIPESTATUS is an internal bash array that contains the status of each process in the previous pipe. You have to echo ${PIPESTATUS[@]} or else you won't see all elements of the array.
Well, changing in the last example $PIPESTATUS to ${PIPESTATUS[@]} changes output to '0 0' That's still a correct unpack.
Well, that's exactly what portage does. So, I guess it's failing when portage does it because then there's more memory load?
While that seems possible, it doesn't work even for XZ_OPT="--memory=64M". While this system does quite a bit of swapping, there should be enough memory to unpack an archive. And there's still that strange error for lzma. "Unexpected end of input" ?
(In reply to comment #8) > And there's still that strange error for lzma. > "Unexpected end of input" ? That suggests that the lzma program is failing to produce all of the input that tar is expecting. I you use --debug then it should give more info. Here's the relevant part when I run unpack with --debug: + lzma -dc /var/tmp/portage/sys-devel/libtool-2.2.6b/distdir/libtool-2.2.6b.tar.lzma + assert 'failure unpacking libtool-2.2.6b.tar.lzma' + local x 'pipestatus=0 0' + for x in '$pipestatus' + [[ 0 -eq 0 ]] + for x in '$pipestatus' + [[ 0 -eq 0 ]]
For me that part looks: + case "${x##*.}" in + _unpack_tar lzma + '[' tar == tar ']' + tar xof - + lzma -dc /var/tmp/portage/sys-devel/libtool-2.2.6b/distdir/libtool-2.2.6b.tar.lzma lzma: /var/tmp/portage/sys-devel/libtool-2.2.6b/distdir/libtool-2.2.6b.tar.lzma: Unexpected end of input tar: To nie wygląda jak archiwum tar tar: Zakończenie w stanie błędu z powodu uprzednich błędów + assert 'failure unpacking libtool-2.2.6b.tar.lzma' + local x 'pipestatus=1 2' (testing on libtool ebuild seemed most simple)
Maybe update/rebuild xz-utils in case that helps.
It's 9999 and was rebuilt yesterday. And if that should be memory management, then why does 'tar xoJf' suceed ?
It works here with xz-utils-4.999.9_beta. Maybe there's a bug in 9999.
It would have to be strange one, as both commandline and texlive 2009 ebuilds (so tar.xz) work.
Though now that you mention it, portage started to feel a bit sluggish lately, as if something with memory management was wrong (as mentioned, I do rely on swap quite a bit, mainly due to firefox).
However, after I close firefox, I've got a bit over 200MB free, so I think more than enough and it still fails.
So OK, with beta it works fine, however, given that it works just fine on the command line, I doubt upstream will be able to fix it without a testcase, especially that AFAIK it consists of a single person.
*** Bug 307929 has been marked as a duplicate of this bug. ***
(In reply to comment #18) > *** Bug 307929 has been marked as a duplicate of this bug. *** > Happy to see, it's not just my machine. However, the other reporter is asked for info about his machine.
> the other reporter is asked for info > about his machine. Hi it's me. Well it's a ~x86. Just updated xz-utils has been broken for at least 2 weeks for me. Latest git still broken. Or I think I read it's portage problem.
It looks like we're going to have to do some debugging in xz-utils. Maybe we'll get a clue if we strace it when it's failing.
(In reply to comment #21) > It looks like we're going to have to do some debugging in xz-utils. Maybe we'll > get a clue if we strace it when it's failing. > We need a testing custom ebuild with customized unpack stage.
Created attachment 222297 [details, diff] patch the libtool-2.2.6b ebuild to create $T/lzma.strace Please try this patch and attach $T/lzma.strace from a failed unpack.
Created attachment 222313 [details] strace with current xz-utils git
It did a single read from libtool-2.2.6b.tar.lzma which returned a full 8192 byte buffer, then without trying to read any more data it showed this error before exiting with status 1: lzma: /var/tmp/portage/sys-devel/libtool-2.2.6b/distdir/libtool-2.2.6b.tar.lzma: Unexpected end of input
After IRCing with upstream a bit, it's fixed in the git now.
(In reply to comment #26) > After IRCing with upstream a bit, it's fixed in the git now. Great, I guess people who have this issue can just rebuild xz-utils-9999 in order to get the fix.
