Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 581924 - Mirroring of new large distfiles broken: stage3-i686-20160531.tar.bz2 is corrupted
Summary: Mirroring of new large distfiles broken: stage3-i686-20160531.tar.bz2 is corr...
Status: IN_PROGRESS
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
: 583082 584922 586150 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-05-03 07:56 UTC by Jocelyn Mayer
Modified: 2016-06-17 11:54 UTC (History)
8 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jocelyn Mayer 2016-05-03 07:56:58 UTC
While trying to merge www-client/chromium-50.0.2661.94, download always fails with a "Fetched file: chromium-50.0.2661.94.tar.xz VERIFY FAILED!" issue.
Either the downloaded file is corrupted, whatever the gentoo mirror I choose, or the Manifest file is invalid.


Reproducible: Always

Steps to Reproduce:
1. emerge --verbose =www-client/chromium-50.0.2661.94::gentoo

Actual Results:  
2016-05-03 07:42:59 (276 KB/s) - '/usr/portage/distfiles/chromium-50.0.2661.94.tar.xz' saved [531491584/531491584]

!!! Fetched file: chromium-50.0.2661.94.tar.xz VERIFY FAILED!
!!! Reason: Failed on SHA256 verification
!!! Got:      85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2
!!! Expected: 66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19


Expected Results:  
Fetched file checksums to match the Manifest file ones.

Checking by hand using sha256sum, sha512sum and whirlpooldeep also show checksums which are not the ones listed in the www-client/chromium/Manifest file:
# wget 'http://gentoo.mirrors.ovh.net/gentoo-distfiles/distfiles/chromium-50.0.2661.94.tar.xz'
Connecting to gentoo.mirrors.ovh.net (gentoo.mirrors.ovh.net)|91.121.189.201|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 531491584 (507M) [application/x-xz]
Saving to: 'chromium-50.0.2661.94.tar.xz'

chromium-50.0.2661. 100%[=====================>] 506.87M   231KB/s   in 32m 28ss

2016-05-03 09:42:22 (266 KB/s) - 'chromium-50.0.2661.94.tar.xz' saved [531491584/531491584]

# sha256sum chromium-50.0.2661.94.tar.xz
85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2  chromium-50.0.2661.94.tar.xz
# sha512sum chromium-50.0.2661.94.tar.xz
42a2ca88b014c48a8bca26f8cc2f2ccb017cd2f89f44361640e15ad90297001fc96f690752709b2d20a28e8c3386e20cbcabcd1cc1a739bd59254dd85c225370  chromium-50.0.2661.94.tar.xz
# whirlpooldeep chromium-50.0.2661.94.tar.xz
1f45ffc25101002bce49d13b9aacf18b8a596320dc0e6b6c7f6c585577f3294c1beb591691dd64707a95582ce0f2d143b9db561b3d407ec8b7b4d08974e2ebe2  /usr/portage/distfiles/chromium-50.0.2661.94.tar.xz

while Manifest file contains:
SHA256 6cc22761c4a774408cb34e77e056691526f469eca0899f578d5372cf0f756786
SHA512 64549732146ab2aa3c7582b4596f2a4fde7769c76facf380a6f2e3504c32be42560d1b230a943edb79dd0f019eca7ad133a32765b17143fc9d3eada1d3744ddd
WHIRLPOOL 837a977c7ec66e3d2f1b58a54875e3b5b1c130117a1add077b6f38c196c432f175d55a0663f5ac576dd44c9ee77199689125b8b57794e4422fed7b5fe9961fa0

# emerge --info
Portage 2.2.26 (python 3.4.3-final-0, default/linux/amd64/13.0/developer, gcc-4.9.3, glibc-2.22-r4, 4.1.15-gentoo-r1 x86_64)
=================================================================
System uname: Linux-4.1.15-gentoo-r1-x86_64-AMD_Athlon-tm-_II_X2_255_Processor-with-gentoo-2.2
KiB Mem:    15925460 total,   4443492 free
KiB Swap:   33554424 total,  30177496 free
Timestamp of repository gentoo: Tue, 03 May 2016 05:45:01 +0000
sh bash 4.3_p42-r1
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash:          4.3_p42-r1::gentoo
dev-lang/perl:            5.20.2::gentoo
dev-lang/python:          2.7.10-r1::gentoo, 3.3.5-r3::gentoo, 3.4.3-r1::gentoo
dev-util/cmake:           3.3.1-r1::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.19.1::gentoo
sys-apps/sandbox:         2.10-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo
sys-devel/automake:       1.4_p6-r2::gentoo, 1.5-r2::gentoo, 1.6.3-r2::gentoo, 1.7.9-r3::gentoo, 1.8.5-r5::gentoo, 1.9.6-r4::gentoo, 1.10.3-r1::gentoo, 1.11.6-r1::gentoo, 1.12.6::gentoo, 1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.20.1-r1::gentoo, 2.21.1-r1::gentoo, 2.22-r1::gentoo, 2.23.2::gentoo, 2.25.1-r1::gentoo
sys-devel/gcc:            3.4.6-r2::gentoo, 4.3.6-r1::gentoo, 4.4.7::gentoo, 4.5.4::gentoo, 4.6.4::gentoo, 4.7.4::gentoo, 4.8.5::gentoo, 4.9.3::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 4.3::gentoo (virtual/os-headers)
sys-libs/glibc:           2.22-r4::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage
    priority: -1000

x-portage
    location: /usr/local/portage
    masters: gentoo
    priority: 0

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -g"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/share/gnupg/qualified.txt /var/yp/Makefile"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.5/ext-active/ /etc/php/apache2-php5.6/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=native -O2 -pipe -fomit-frame-pointer -g"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildsyspkg collision-protect config-protect-if-modified ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sfperms sign splitdebug strict test-fail-continue unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.modulix.net/gentoo/ http://gentoo.mirrors.ovh.net/gentoo-distfiles/ http://mirror.netcologne.de/gentoo/ http://mirror.leaseweb.com/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://mirror.bytemark.co.uk/gentoo/ http://mirror.eu.oneandone.net/linux/distributions/gentoo/gentoo/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://tux.rainside.sk/gentoo/"
LDFLAGS="-Wl,--hash-style=gnu -Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j 1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="X Xaw3d a52 aac aalib acl acpi adns aim alsa amd64 ao apache2 apm audiofile bash-completion bcmath berkdb bidi blas bluetooth branding bzip2 cairo calendar caps cdb cdda cddb cdparanoia cdr cgi cli consolekit cracklib crypt cscope css ctype cups curl curlwrappers cvs cxx dbus dga djvu doc dri dts dv dvb dvd dvdr eds emacs encode enscript evo examples exif expat fam fastcgi fbcon ffmpeg fftw filecaps firefox flac fltk fontconfig foomaticdb fortran ftp gd gdbm geoip ggi gif gimp git glamor glut gmp gnuplot gnutls gphoto2 gpm gps graphviz gsl gsm gstreamer gtk guile gzip hddtemp iconv icq icu idn ieee1394 imagemagick imap imlib inotify introspection iodbc ios ipod ipv6 jack javascript jingle jit jpeg jpeg2k ladspa lame lapack lash latex lcms ldap libass libcaca libffi libnotify libsamplerate libwww lm_sensors lua lzma lzo m17n-lib mad maildir matroska mbox memlimit mhash mikmod mime mmap mms mmx mmxext mng modplug modules motif mozilla mp3 mp4 mpeg mpi mplayer msn mtp multilib musepack musicbrainz nas ncurses neXt netcdf nls nntp nptl nsplugin ocaml ocamlopt odbc offensive ofx ogg openal openexr opengl openmp orc osc oscar oss pam pango pcntl pcre pda pdf perl pie plotutils png policykit portaudio posix postgres postscript ppds pulseaudio python qt3support quicktime raw rdesktop readline recode rss ruby sasl savedconfig scanner sdl seccomp sharedmem shorten simplexml skey slang slp smp sndfile snmp soap sockets sound sox speex spell sqlite sse sse2 ssl startup-notification static-libs subversion svg syslog sysvipc szip taglib tcl tcpd theora threads tidy tiff timidity tk truetype udev unicode upnp upnp-av upower usb v4l vaapi vcd vdpau vhosts videos vnc vorbis wavpack wddx webkit wifi wmf wxwidgets x264 xattr xcb xcomposite xface xft xine xinerama xinetd xml xmlrpc xmp xmpp xosd xpm xscreensaver xv xvid yahoo yaz zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimmizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 1 Alex Xu (Hello71) 2016-05-03 14:08:37 UTC
$ wget https://storage.googleapis.com/chromium-browser-official/chromium-50.0.2661.94.tar.xz
[ ... ]
$ wget http://ftp-osl.osuosl.org/distfiles/chromium-50.0.2661.94.tar.xz
[ ... ]
$ wget http://gentoo.ussg.indiana.edu/distfiles/chromium-50.0.2661.94.tar.xz
[ ... ]
$ rsync ftp.ussg.iu.edu::gentoo-distfiles/distfiles/chromium-50.0.2661.94.tar.xz [ ... ]
[ ... ]
$ sha256sum chromium*
66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19  chromium-50.0.2661.94.tar.xz (correct)
5daabd61e18bd670329cf664e92c64c324e0f93c2cd72f2b11f8b17e3a4b49b7  chromium-50.0.2661.94.tar.xz.1 (invalid from osuosl)
85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2  chromium-50.0.2661.94.tar.xz.3 (invalid from iu.edu via HTTP)
85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2  chromium-50.0.2661.94.tar.xz.4 (invalid from iu.edu via rsync)
Comment 2 Alex Xu (Hello71) 2016-05-03 14:10:43 UTC
$ cmp -b chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.1
chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.1 differ: byte 317835433, line 1239861 is 241 M-!   0 ^@
$ cmp -b chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.3
chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.3 differ: byte 317653889, line 1239176 is  20 ^P   0 ^@
I suspect there is some hardware issue in the servers responsible for mirroring.
Comment 3 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2016-05-04 22:17:55 UTC
The distribution path for distfiles goes dipper.gentoo.org -> FTP-OSL -> $other_mirrors.

The copy on dipper is CORRECT, so that means FTP-OSL is the source of corruption.

dipper:
# stat ./distfiles/chromium-50.0.2661.94.tar.xz
  File: ‘./distfiles/chromium-50.0.2661.94.tar.xz’
  Size: 531491584 	Blocks: 1038072    IO Block: 4096   regular file
Device: fc04h/64516d	Inode: 805317849   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-05-01 01:56:05.000000000 +0000
Modify: 2016-04-28 19:33:49.000000000 +0000
Change: 2016-05-01 01:56:08.285454932 +0000
 Birth: -
# sha256sum ./distfiles/chromium-50.0.2661.94.tar.xz
66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19 ./distfiles/chromium-50.0.2661.94.tar.xz

I'm going to touch the file and see if OSL re-fetches it.

lance/greg:
If it doesn't get re-fetched, can you please drop that file on the mirror so it re-fetches.
Comment 4 Alex Xu (Hello71) 2016-05-05 00:16:36 UTC
(In reply to Robin Johnson from comment #3)
> The distribution path for distfiles goes dipper.gentoo.org -> FTP-OSL ->
> $other_mirrors.
> 
> The copy on dipper is CORRECT, so that means FTP-OSL is the source of
> corruption.

assuming that the network in between them is OK.

also that doesn't explain why the corruption found at iu.edu is slightly different.

semi-happily though, other mirrors (tested mirror.us.leaseweb.net, gentoo-distfiles.mirrors.tds.net) have the same corruption as iu.edu.

ftp-osl.osuosl.org:
$ LANG=C cmp -bl chromium-50.0.2661.94.tar.xz{,.2}
317835433 241 M-!    0 ^@
317835434 132 Z      0 ^@
317835435  26 ^V     0 ^@
317835436 241 M-!    0 ^@
317835437 363 M-s    0 ^@
317835438 264 M-4    0 ^@
317835439 106 F      0 ^@
317835440 214 M-^L   0 ^@

gentoo-distfiles.mirrors.tds.net:
$ LANG=C cmp -bl chromium-50.0.2661.94.tar.xz{,.1}
317653889  20 ^P     0 ^@
317653890 275 M-=    0 ^@
317653891  35 ^]     0 ^@
317653892 256 M-.    0 ^@
317653893 267 M-7    0 ^@
317653894 226 M-^V   0 ^@
317653895 202 M-^B   0 ^@
317653896 323 M-S    0 ^@

317689713  40        0 ^@
317689714 143 c      0 ^@
317689715 135 ]      0 ^@
317689716 255 M--    0 ^@
317689717 103 C      0 ^@
317689718 153 k      0 ^@
317689719 363 M-s    0 ^@
317689720 163 s      0 ^@

317835433 241 M-!    0 ^@
317835434 132 Z      0 ^@
317835435  26 ^V     0 ^@
317835436 241 M-!    0 ^@
317835437 363 M-s    0 ^@
317835438 264 M-4    0 ^@
317835439 106 F      0 ^@
317835440 214 M-^L   0 ^@

(new lines added for ease of reading)

also, I got those files at least one and a half hours after your comment, and I think distfiles mirrors are supposed to sync twice an hour?
Comment 5 Anton Kuleshov 2016-05-05 14:36:59 UTC
Same on https://mirror.yandex.ru/gentoo-distfiles/distfiles/

Sum is 85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2
Comment 6 Lance Albertson 2016-05-05 15:51:56 UTC
(In reply to Robin Johnson from comment #3)

> lance/greg:
> If it doesn't get re-fetched, can you please drop that file on the mirror so
> it re-fetches.

This is odd, here's what I see on my end:

[root@ftp-osl gentoo]# stat distfiles/chromium-50.0.2661.94.tar.xz
  File: distfiles/chromium-50.0.2661.94.tar.xz
  Size: 531491584       Blocks: 1038072    IO Block: 65536  regular file
Device: fd06h/64774d    Inode: 4298642613  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:default_t:s0
Access: 2016-05-05 09:35:35.847228770 +0000
Modify: 2016-04-28 19:33:49.000000000 +0000
Change: 2016-05-01 02:05:57.752528151 +0000
 Birth: -
[root@ftp-osl gentoo]# sha256sum ./distfiles/chromium-50.0.2661.94.tar.xz
5daabd61e18bd670329cf664e92c64c324e0f93c2cd72f2b11f8b17e3a4b49b7  ./distfiles/chromium-50.0.2661.94.tar.xz

So I re-ran the rsync and it didn't update the file so I went ahead and rm'd it manually and did a resync again. Now I see this:

[root@ftp-osl gentoo]# stat distfiles/chromium-50.0.2661.94.tar.xz
  File: distfiles/chromium-50.0.2661.94.tar.xz
  Size: 531491584       Blocks: 1071104    IO Block: 65536  regular file
Device: fd06h/64774d    Inode: 4298642613  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:default_t:s0
Access: 2016-05-05 15:48:37.855179868 +0000
Modify: 2016-04-28 19:33:49.000000000 +0000
Change: 2016-05-05 15:48:37.855179868 +0000
 Birth: -
[root@ftp-osl gentoo]# sha256sum ./distfiles/chromium-50.0.2661.94.tar.xz
66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19  ./distfiles/chromium-50.0.2661.94.tar.xz

Not sure what's causing this, the rsync opts we're using is "-avH --delete-after". Is this the only file that had the problem?

We did have some xfs corruption the other day and ran fsck on ftp-osl. It might be related to that perhaps.

Let me know if this fixes it.
Comment 7 Alex Xu (Hello71) 2016-05-06 16:30:31 UTC
(In reply to Lance Albertson from comment #6)
> Let me know if this fixes it.

the file appears OK when fetched from ftp-osl now, but still broken on iu.edu. this particular file will probably be dropped anyways, but I think it would be a good idea to run rsync --checksum this one time against the master if it's possible to check if any other files have issues.
Comment 8 Lance Albertson 2016-05-06 16:37:48 UTC
(In reply to Alex Xu (Hello71) from comment #7)
> (In reply to Lance Albertson from comment #6)
> > Let me know if this fixes it.
> 
> the file appears OK when fetched from ftp-osl now, but still broken on
> iu.edu. this particular file will probably be dropped anyways, but I think
> it would be a good idea to run rsync --checksum this one time against the
> master if it's possible to check if any other files have issues.

I verified the checksum on ftp-nyc and ftp-chi to be the same as on the master so what I have is correct now.
Comment 9 Alex Xu (Hello71) 2016-06-06 15:23:48 UTC
*** Bug 584922 has been marked as a duplicate of this bug. ***
Comment 10 Alex Xu (Hello71) 2016-06-06 15:26:59 UTC
see duplicate bug for more information on this file.
Comment 11 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2016-06-09 23:54:09 UTC
For everybody else following this, I wrote & posted a tool, and did a write-up for the mirrors list about the issue:
https://archives.gentoo.org/gentoo-mirrors/message/a9fbc7213f832e9918784bb8d334628b
Comment 12 Lance Albertson 2016-06-10 15:48:38 UTC
I've run this tool on the releases directory on ftp-osl (the gentoo master public mirror) and got the following failures:

stage3-alpha-20160210.tar.bz2: FAILED open or read
stage3-alpha-20160210.tar.bz2.CONTENTS: OK
stage3-armv4tl-20160327.tar.bz2: FAILED open or read
stage3-armv4tl-20160327.tar.bz2.CONTENTS: OK
Comment 13 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2016-06-12 18:30:39 UTC
(In reply to Lance Albertson from comment #12)
> I've run this tool on the releases directory on ftp-osl (the gentoo master
> public mirror) and got the following failures:
> 
> stage3-alpha-20160210.tar.bz2: FAILED open or read
> stage3-alpha-20160210.tar.bz2.CONTENTS: OK
> stage3-armv4tl-20160327.tar.bz2: FAILED open or read
> stage3-armv4tl-20160327.tar.bz2.CONTENTS: OK
I  confirm those files were never uploaded from the relevant build hosts. I have removed the contents+digests files for them.
Comment 14 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2016-06-12 18:38:54 UTC
@lance:
I have more reports of broken files from the mirrors; and they say it is broken on one of the masterdistfiles.g.o/rsync.osuosl.org rotation (but didn't give an IP sadly).

experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2
releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2
Comment 15 Lance Albertson 2016-06-12 19:35:38 UTC
(In reply to Robin Johnson from comment #14)
> @lance:
> I have more reports of broken files from the mirrors; and they say it is
> broken on one of the masterdistfiles.g.o/rsync.osuosl.org rotation (but
> didn't give an IP sadly).
> 
> experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.
> tar.bz2
> releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-
> 20160602.tar.bz2

I found failures on ftp-chi/ftp-nyc with the following files and fixed them:

./amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2.DIGESTS.asc
./amd64/autobuilds/20160602/install-amd64-minimal-20160602.iso.DIGESTS.asc
./x86/autobuilds/20160531/stage3-i686-20160531.tar.bz2.DIGESTS.asc

I did not see any errors with mips32r2 on any mirror.
Comment 16 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2016-06-14 07:03:06 UTC
(In reply to Lance Albertson from comment #15)
> I did not see any errors with mips32r2 on any mirror.
Still getting reports for it:

> Right, so masterdistfiles.gentoo.org resolves to the two IP addresses
> of 64.50.236.52, 64.50.233.100 here. Consequently, I just now fetched
> the file
> /experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2
> (to test one of the files I spotted yesterday) once from both IPs, and
> in both cases, they yield the sha512sum of
> ab01d59eebc811789d1ca592da4330e34233832f07ebaa9808bb12a24b56a1a694a657ad817b7e8de30cd031da6f630d2cf41ad5066e09692b791efefe8960dd
> Correct, however, according to the digest files, is
> 
> 758d304df8b24a842466713ee5297b62bc66674041ed462a1f8374a6281455b283411395cccd318b31edc2a3ce289f475fb090eb418bec6a2e1bdd3b4aee4610
Comment 17 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-06-17 11:54:24 UTC
*** Bug 583082 has been marked as a duplicate of this bug. ***
Comment 18 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-06-17 11:54:29 UTC
*** Bug 586150 has been marked as a duplicate of this bug. ***