Bug 145601 – emerge loops on single file download after previous download errors

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Christoph Probst 2006-08-30 07:28:26 UTC]

Today I was having an issue when updateing my system as usual (emerge -uD world). In the beginning the download of one of the source files failed multiple times due to connection problems. Suddenly wget decided to rename the files tar.bz2.1, tar.bz2.2, tar.bz2.3, ... and emerge kept on restarting the download. I stopped it manually at .5 and I guess it wouldn't had stopped by itself.

Have a look at the following output and you'll understand what I mean. This can certainly be fix manually, though it shouldn't happen at all. The output of emerge --info is below aswell.

-----------------------------8<--------------------------------------------
>>> Emerging (4 of 36) kde-base/kdepim-3.5.4 to /
>>> Downloading 'http://gentoo.osuosl.org/distfiles/kdepim-3.5.4.tar.bz2'
--15:02:24--  http://gentoo.osuosl.org/distfiles/kdepim-3.5.4.tar.bz2
           => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2'
Resolving gentoo.osuosl.org... 64.50.236.52, 64.50.238.52
Connecting to gentoo.osuosl.org|64.50.236.52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12,925,075 (12M) [application/x-tar]

66% [========================================================================================>                                              ] 8,556,191     --.--K/s    ETA 03:53

15:10:02 (18.27 KB/s) - Read error at byte 8556191/12925075 (Connection timed out). Retrying.

[...]

--15:14:23--  http://gentoo.osuosl.org/distfiles/kdepim-3.5.4.tar.bz2
  (try: 5) => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2'
Connecting to gentoo.osuosl.org|64.50.236.52|:80... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.

>>> Downloading 'http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/kdepim-3.5.4.tar.bz2'
--15:15:33--  http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/kdepim-3.5.4.tar.bz2
           => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.1'
Resolving www.ibiblio.org... 152.2.210.80
Connecting to www.ibiblio.org|152.2.210.80|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/kdepim-3.5.4.tar.bz2 [following]
--15:15:53--  http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/kdepim-3.5.4.tar.bz2
           => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.1'
Resolving distro.ibiblio.org... 152.46.7.109
Connecting to distro.ibiblio.org|152.46.7.109|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12,925,075 (12M) [application/x-bzip2]

 0% [                                                                                                                                       ] 32,837        --.--K/s  ETA 8:57:11

15:17:16 (399.93 B/s) - Read error at byte 32837/12925075 (Connection timed out). Retrying.

[...]

--15:18:55--  http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/kdepim-3.5.4.tar.bz2
  (try: 3) => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.1'
Connecting to distro.ibiblio.org|152.46.7.109|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 12,925,075 (12M), 12,885,354 (12M) remaining [application/x-bzip2]

100%[======================================================================================================================================>] 12,925,075     9.50K/s    ETA 00:00

15:35:06 (13.00 KB/s) - `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.1' saved [12925075/12925075]

>>> Downloading 'http://ibiblio.org/pub/mirrors/kde/stable/3.5.4/src/kdepim-3.5.4.tar.bz2'
--15:35:06--  http://ibiblio.org/pub/mirrors/kde/stable/3.5.4/src/kdepim-3.5.4.tar.bz2
           => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.2'
Resolving ibiblio.org... 152.2.210.80
Connecting to ibiblio.org|152.2.210.80|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://mirrors.ibiblio.org/pub/mirrors/kde/stable/3.5.4/src/kdepim-3.5.4.tar.bz2 [following]
--15:35:12--  http://mirrors.ibiblio.org/pub/mirrors/kde/stable/3.5.4/src/kdepim-3.5.4.tar.bz2
           => `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.2'
Resolving mirrors.ibiblio.org... 152.2.210.65
Connecting to mirrors.ibiblio.org|152.2.210.65|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12,925,075 (12M) [application/x-bzip2]

100%[======================================================================================================================================>] 12,925,075    14.58K/s    ETA 00:00

15:47:08 (17.64 KB/s) - `/usr/portage/distfiles/kdepim-3.5.4.tar.bz2.2' saved [12925075/12925075]
-----------------------------8<--------------------------------------------


-----------------------------8<--------------------------------------------
# emerge --info
Portage 2.1.1_rc1-r1 (default-linux/x86/2006.0, gcc-3.4.6/vanilla, glibc-2.4-r3, 2.6.14-suspend2-r5 i686)
=================================================================
System uname: 2.6.14-suspend2-r5 i686 AMD-K7(tm) Processor
Gentoo Base System version 1.12.4
Last Sync: Mon, 28 Aug 2006 23:30:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
ccache version 2.4 [disabled]
app-admin/eselect-compiler: 2.0.0_rc2-r1
dev-lang/python:     2.3.5, 2.4.3-r3
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.4-r2
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.18.1
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.17
sys-devel/gcc-config: 1.3.13-r3
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.17
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=athlon -Os -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/eselect/compiler /etc/gconf /etc/java-config/vms/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c"
CXXFLAGS="-O2 -mcpu=i686 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig distlocks metadata-transfer sandbox sfperms strict"
GENTOO_MIRRORS=" http://gentoo.osuosl.org http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LINGUAS="de"
MAKEOPTS="-s -j1"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 3dnow 3dnowext X a52 aac aalib acl acpi alsa apache2 arts avi berkdb bitmap-fonts browserplugin bzlib calendar chipcard cpdflib crypt cups dba dga divx4linux dri dv dvd dvdread elibc_glibc encode exif extras fame foomaticdb ftp gd gd-external gdbm gif gphoto2 gtk gtk2 hbci iconv ijs imagemagick imap imlib input_devices_keyboard input_devices_mouse ipv6 java jpeg kde kdeenablefinal kernel_linux lcms libg++ linguas_de maildir mbox mikmod mmx mp3 mpeg mysql nas ncurses nls nptl nptlonly ogg oggvorbis opengl pam pcre pdflib perl png pnp python qt qtmt quicktime readline real samba sdl session sftplogging simplexml smartcard sockets spell sqlite ssl subtitles tetex tiff truetype truetype-fonts type1-fonts usb userland_GNU vcd video_cards_r128 video_cards_rage128 vorbis win32codecs wmf xml xml2 xmlrpc xmms xorg xv xvid zlib"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, PORTAGE_RSYNC_EXTRA_OPTS
-----------------------------8<--------------------------------------------

[Comment 1 Doug Goldstein (RETIRED) 2006-09-17 17:00:23 UTC]

Same issue here..

$ emerge --info
Portage 2.1.2_pre1 (default-linux/x86/2006.1/desktop, gcc-4.1.1, glibc-2.4-r3, 2.6.17-gentoo-r8 i686)
=================================================================
System uname: 2.6.17-gentoo-r8 i686 Intel(R) Pentium(R) M processor 1.80GHz
Gentoo Base System version 1.12.5
Last Sync: Sun, 17 Sep 2006 21:30:07 +0000
ccache version 2.4 [enabled]
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: 1.3.3-r1, 2.0.28-r1
dev-lang/python:     2.4.3-r3
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.4-r5
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.18.1
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.17
sys-devel/gcc-config: 1.3.13-r3
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.17-r1
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-Os -march=pentium-m -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/revdep-rebuild /etc/splash /etc/terminfo"
CXXFLAGS="-Os -march=pentium-m -fomit-frame-pointer -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig ccache cvs distlocks metadata-transfer sandbox sfperms sign strict"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"
LDFLAGS="-Wl,--as-needed"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/home/cardoe/workspace/gnome /home/cardoe/workspace/gentopia/overlay /home/cardoe/workspace/ferringb"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 X a52 aac acpi alsa avahi bash-completion beagle berkdb bitmap-fonts bzip2 cairo cdr cli crypt dbus dlloader dri dts dvd dvdr elibc_glibc encode exif fam firefox gdbm gif glitz gnome gstreamer gtk hal howl input_devices_keyboard input_devices_mouse ipv6 isdnlog jpeg kernel_linux lcms libg++ libnotify lirc_devices_mceusb2 live mad mjpeg mmx mono mp3 mpeg mythtv ncurses nptl nptlonly oav ogg opengl pam pcre pdf perl png ppds pppd python qt3 quicktime readline reflection samba sdl session spell sse sse2 ssl svg tcpd tiff truetype truetype-fonts type1-fonts udev unicode userland_GNU video_cards_ati video_cards_radeon vorbis win32codecs xml xorg xv xvid zlib"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS

[Comment 2 Zac Medico 2006-09-18 19:40:58 UTC]

Created attachment 97380 [details, diff]
properly resume after wget fails and leaves a partial download

This patch should fix it (in svn r4478).  It will case RESUMECOMMAND to be called when the previous fetch failed and left a partial download.

[Comment 3 Zac Medico 2006-09-22 19:24:24 UTC]

This has been released in 2.1.2_pre1-r1.

[Comment 4 Jakub Moc (RETIRED) 2006-10-08 10:59:29 UTC]

*** Bug 150487 has been marked as a duplicate of this bug. ***

[Comment 5 Zac Medico 2006-10-27 14:04:57 UTC]

*** Bug 153040 has been marked as a duplicate of this bug. ***