Emerging sys-apps/shadow-4.1.4.3 (emerge -uD @system) fails, the needed archive (shadow-4.1.4.3.tar.bz2) is not available for download. Reproducible: Always Steps to Reproduce: emerge -uD @system pulls sys-apps/shadow-4.1.4.3 on my stable amd64 machine. merging fails: >>> Emerging (1 of 2) sys-apps/shadow-4.1.4.3 >>> Downloading 'http://mirror.netcologne.de/gentoo/distfiles/shadow-4.1.4.3.tar.bz2' --2011-02-21 12:13:18-- http://mirror.netcologne.de/gentoo/distfiles/shadow-4.1.4.3.tar.bz2 Resolving mirror.netcologne.de (mirror.netcologne.de)... 2001:4dd0:1234:1::deb, 194.8.197.22 Connecting to mirror.netcologne.de (mirror.netcologne.de)|2001:4dd0:1234:1::deb|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 12:13:19 ERROR 404: Not Found. >>> Downloading 'http://de-mirror.org/distro/gentoo/distfiles/shadow-4.1.4.3.tar.bz2' --2011-02-21 12:13:19-- http://de-mirror.org/distro/gentoo/distfiles/shadow-4.1.4.3.tar.bz2 Resolving de-mirror.org (de-mirror.org)... 87.106.91.70 Connecting to de-mirror.org (de-mirror.org)|87.106.91.70|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 12:13:19 ERROR 404: Not Found. >>> Downloading 'http://distfiles.gentoo.org/distfiles/shadow-4.1.4.3.tar.bz2' --2011-02-21 12:13:19-- http://distfiles.gentoo.org/distfiles/shadow-4.1.4.3.tar.bz2 Resolving distfiles.gentoo.org (distfiles.gentoo.org)... 64.50.236.52, 137.226.34.42, 140.211.166.134, ... Connecting to distfiles.gentoo.org (distfiles.gentoo.org)|64.50.236.52|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 12:13:19 ERROR 404: Not Found. >>> Downloading 'http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/shadow-4.1.4.3.tar.bz2' --2011-02-21 12:13:19-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/shadow-4.1.4.3.tar.bz2 Resolving www.ibiblio.org (www.ibiblio.org)... 152.46.7.80 Connecting to www.ibiblio.org (www.ibiblio.org)|152.46.7.80|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 12:13:20 ERROR 404: Not Found. >>> Downloading 'ftp://pkg-shadow.alioth.debian.org/pub/pkg-shadow/shadow-4.1.4.3.tar.bz2' --2011-02-21 12:13:20-- ftp://pkg-shadow.alioth.debian.org/pub/pkg-shadow/shadow-4.1.4.3.tar.bz2 => “/usr/portage/distfiles/shadow-4.1.4.3.tar.bz2” Resolving pkg-shadow.alioth.debian.org (pkg-shadow.alioth.debian.org)... 217.196.43.134 Connecting to pkg-shadow.alioth.debian.org (pkg-shadow.alioth.debian.org)|217.196.43.134|:21... failed: Connection timed out. Retrying. --2011-02-21 12:13:41-- ftp://pkg-shadow.alioth.debian.org/pub/pkg-shadow/shadow-4.1.4.3.tar.bz2 (try: 2) => “/usr/portage/distfiles/shadow-4.1.4.3.tar.bz2” Connecting to pkg-shadow.alioth.debian.org (pkg-shadow.alioth.debian.org)|217.196.43.134|:21... failed: Connection timed out. Retrying. --2011-02-21 12:14:03-- ftp://pkg-shadow.alioth.debian.org/pub/pkg-shadow/shadow-4.1.4.3.tar.bz2 (try: 3) => “/usr/portage/distfiles/shadow-4.1.4.3.tar.bz2” Connecting to pkg-shadow.alioth.debian.org (pkg-shadow.alioth.debian.org)|217.196.43.134|:21... failed: Connection timed out. Giving up. !!! Couldn't download 'shadow-4.1.4.3.tar.bz2'. Aborting. Since distfiles.gentoo.org is among the locations that were tried, I feel someone may have forgotten to put it up, rather than it's just my mirrors that don't have the file yet.
I don't see the file on masterdistfiles. Any ideas on what might be happening?
I noticed the same thing this morning. My work firewall blocks a LOT of upstream mirrors, here are some other packages whose distfiles don't seem to be on our mirrors as of this morning: net-misc/curl-7.21.4 (48 hrs old) net-misc/iputils-20101006 (47 hrs old) app-portage/pfl-2.1-r1 (57 hrs since RESTRICT="mirror" was removed) Sure seems like this issue is bigger than shadow.
your log already told you the problem: (pkg-shadow.alioth.debian.org)|217.196.43.134|:21... failed: Connection timed out.
it'll work itself out eventually
A few more examples I found (this will be the last list, promise!): sys-apps/ethtool-2.6.37 (24 hrs) app-emulation/virtualbox-modules-4.0.4 (57 hrs) app-emulation/virtualbox-4.0.4 (57 hrs) dev-lang/ruby-1.8.7_p334 (28 hrs) Are ALL these packages' upstream mirrors having issues, all within the 24-48 hr time period?
The tarball is available on the mirror now, and as Ben points out, there are a bunch of other tarballs that have not been synced, so reopening.
Thanks for the report guys, however, there is nothing really to fix from the mirror-admin side. Comment #4 is right, it will work itself out. http://dev.gentoo.org/distfile-mirroring/ (see section 2) % ebuild ethtool-2.6.37.ebuild unpack>>> Downloading 'http://distfiles.gentoo.org/distfiles/ethtool-2.6.37.tar.bz2' --2011-02-21 13:32:41-- http://distfiles.gentoo.org/distfiles/ethtool-2.6.37.tar.bz2 Resolving distfiles.gentoo.org... 156.56.247.195, 149.20.20.135, 140.211.166.134, ... Connecting to distfiles.gentoo.org|156.56.247.195|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 13:32:41 ERROR 404: Not Found. >>> Downloading 'http://www.kernel.org/pub/software/network/ethtool/ethtool-2.6.37.tar.bz2' --2011-02-21 13:32:41-- http://www.kernel.org/pub/software/network/ethtool/ethtool-2.6.37.tar.bz2 Resolving www.kernel.org... 130.239.17.4, 199.6.1.164, 149.20.20.133, ... Connecting to www.kernel.org|130.239.17.4|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 158183 (154K) [application/x-bzip2] Saving to: `/usr/portage/distfiles/ethtool-2.6.37.tar.bz2' 100%[======================================>] 158,183 162K/s in 1.0s 2011-02-21 13:32:43 (162 KB/s) - `/usr/portage/distfiles/ethtool-2.6.37.tar.bz2' saved [158183/158183] * ethtool-2.6.37.tar.bz2 RMD160 SHA1 SHA256 size ;-) ... [ ok ] * checking ebuild checksums ;-) ... [ ok ] * checking auxfile checksums ;-) ... [ ok ] * checking miscfile checksums ;-) ... [ ok ] * checking ethtool-2.6.37.tar.bz2 ;-) ... [ ok ] * Package: sys-apps/ethtool-2.6.37 * Repository: gentoo * Maintainer: base-system@gentoo.org * USE: amd64 elibc_glibc kernel_linux multilib userland_GNU * FEATURES: sandbox >>> Unpacking source... >>> Unpacking ethtool-2.6.37.tar.bz2 to /mnt/tmpfs/portage/sys-apps/ethtool-2.6.37/work >>> Source unpacked in /mnt/tmpfs/portage/sys-apps/ethtool-2.6.37/work ======== % ebuild virtualbox-modules-4.0.4.ebuild unpack >>> Downloading 'http://distfiles.gentoo.org/distfiles/vbox-kernel-module-src-4.0.4.tar.bz2' --2011-02-21 13:33:20-- http://distfiles.gentoo.org/distfiles/vbox-kernel-module-src-4.0.4.tar.bz2 Resolving distfiles.gentoo.org... 204.152.191.39, 156.56.247.195, 149.20.20.135, ... Connecting to distfiles.gentoo.org|204.152.191.39|:80... failed: Connection refused. Connecting to distfiles.gentoo.org|156.56.247.195|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 13:33:20 ERROR 404: Not Found. >>> Downloading 'http://dev.gentoo.org/~polynomial-c/virtualbox/vbox-kernel-module-src-4.0.4.tar.bz2' --2011-02-21 13:33:20-- http://dev.gentoo.org/~polynomial-c/virtualbox/vbox-kernel-module-src-4.0.4.tar.bz2 Resolving dev.gentoo.org... 140.211.166.183 Connecting to dev.gentoo.org|140.211.166.183|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 850981 (831K) [application/x-bzip2] Saving to: `/usr/portage/distfiles/vbox-kernel-module-src-4.0.4.tar.bz2' 100%[======================================>] 850,981 475K/s in 1.7s 2011-02-21 13:33:22 (475 KB/s) - `/usr/portage/distfiles/vbox-kernel-module-src-4.0.4.tar.bz2' saved [850981/850981] ===== % ebuild virtualbox-4.0.4.ebuild unpack >>> Downloading 'http://distfiles.gentoo.org/distfiles/VirtualBox-4.0.4.tar.bz2' --2011-02-21 13:34:12-- http://distfiles.gentoo.org/distfiles/VirtualBox-4.0.4.tar.bz2 Resolving distfiles.gentoo.org... 64.50.236.52, 137.226.34.42, 140.211.166.134, ... Connecting to distfiles.gentoo.org|64.50.236.52|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 13:34:12 ERROR 404: Not Found. >>> Downloading 'http://download.virtualbox.org/virtualbox/4.0.4/VirtualBox-4.0.4.tar.bz2' --2011-02-21 13:34:12-- http://download.virtualbox.org/virtualbox/4.0.4/VirtualBox-4.0.4.tar.bz2 Resolving download.virtualbox.org... 137.254.16.69 Connecting to download.virtualbox.org|137.254.16.69|:80... connected. HTTP request sent, awaiting response... 302 Moved Temporarily Location: http://dlc.sun.com.edgesuite.net/virtualbox/4.0.4/VirtualBox-4.0.4.tar.bz2 [following] --2011-02-21 13:34:13-- http://dlc.sun.com.edgesuite.net/virtualbox/4.0.4/VirtualBox-4.0.4.tar.bz2 Resolving dlc.sun.com.edgesuite.net... 96.17.156.18, 96.17.156.74 Connecting to dlc.sun.com.edgesuite.net|96.17.156.18|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 70190670 (67M) [application/x-tar] Saving to: `/usr/portage/distfiles/VirtualBox-4.0.4.tar.bz2' 1% [ ] 1,259,190 555K/s ==== % ebuild ruby-1.8.7_p334.ebuild unpack >>> Downloading 'http://distfiles.gentoo.org/distfiles/ruby-1.8.7-p334.tar.bz2' --2011-02-21 13:34:46-- http://distfiles.gentoo.org/distfiles/ruby-1.8.7-p334.tar.bz2 Resolving distfiles.gentoo.org... 216.165.129.135, 204.152.191.39, 156.56.247.195, ... Connecting to distfiles.gentoo.org|216.165.129.135|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 13:34:46 ERROR 404: Not Found. >>> Downloading 'ftp://ruby.cdpa.nsysu.edu.tw/ruby/1.8/ruby-1.8.7-p334.tar.bz2' --2011-02-21 13:34:46-- ftp://ruby.cdpa.nsysu.edu.tw/ruby/1.8/ruby-1.8.7-p334.tar.bz2 => `/usr/portage/distfiles/ruby-1.8.7-p334.tar.bz2' Resolving ruby.cdpa.nsysu.edu.tw... 140.110.123.9 Connecting to ruby.cdpa.nsysu.edu.tw|140.110.123.9|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /ruby/1.8 ... No such directory `ruby/1.8'. >>> Downloading 'http://www.ibiblio.org/pub/languages/ruby/1.8/ruby-1.8.7-p334.tar.bz2' --2011-02-21 13:34:49-- http://www.ibiblio.org/pub/languages/ruby/1.8/ruby-1.8.7-p334.tar.bz2 Resolving www.ibiblio.org... 152.46.7.80 Connecting to www.ibiblio.org|152.46.7.80|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://mirrors.ibiblio.org/pub/mirrors/ruby/1.8/ruby-1.8.7-p334.tar.bz2 [following] --2011-02-21 13:34:49-- http://mirrors.ibiblio.org/pub/mirrors/ruby/1.8/ruby-1.8.7-p334.tar.bz2 Resolving mirrors.ibiblio.org... 152.19.134.44 Connecting to mirrors.ibiblio.org|152.19.134.44|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-02-21 13:34:49 ERROR 404: Not Found. >>> Downloading 'ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p334.tar.bz2' --2011-02-21 13:34:49-- ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p334.tar.bz2 => `/usr/portage/distfiles/ruby-1.8.7-p334.tar.bz2' Resolving ftp.ruby-lang.org... 221.186.184.68 Connecting to ftp.ruby-lang.org|221.186.184.68|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pub/ruby/1.8 ... done. ==> SIZE ruby-1.8.7-p334.tar.bz2 ... 4190857 ==> PASV ... done. ==> RETR ruby-1.8.7-p334.tar.bz2 ... done. Length: 4190857 (4.0M) (unauthoritative) 76% [============================> ] 3,196,560 54.0K/s eta 14s ^ So, yes. You have identified a few packages that aren't on the gentoo mirrors right at this moment but nothing has changed with the distfiles mirroring system in the past....long time. Closing this bug, there is nothing to fix.
darkside: something is still weird. The VirtualBox-4.0.4.tar.bz2 should have been fetched onto osprey by now. Specifically by the pdate_distfiles.sh script. Grepping the general.log, there is no package entry for virtualbox-4.0.4 at all, when there should be. Specifically, it has /home/gmirror/rsync as a PORTDIR, which is no longer updated as you migrated snapshot generation to the new box.
Alright, thanks for that tracing Robin. I just watched the masterdistfiles host run the job that fetches distfiles. finished in 1841 seconds failed on 10 files cleansed 1 files death watch for 796 files new bytes added, 1159232133 shutting down So, once the master distfiles mirror syncs (every h:05) you will see the new files trickle out to the gentoo mirrors. Really fixed now. Sorry for doubting anyone, my apologies.
