Summary: | Mirroring of new large distfiles broken: stage3-i686-20160531.tar.bz2 is corrupted | ||
---|---|---|---|
Product: | Gentoo Infrastructure | Reporter: | Jocelyn Mayer <l_indien> |
Component: | Other | Assignee: | Gentoo Infrastructure <infra-bugs> |
Status: | IN_PROGRESS --- | ||
Severity: | normal | CC: | alex_y_xu, andruhamurz, chromium, gchaix, lance, phmagic, sergeev917, siebz0r |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- |
Description
Jocelyn Mayer
2016-05-03 07:56:58 UTC
$ wget https://storage.googleapis.com/chromium-browser-official/chromium-50.0.2661.94.tar.xz [ ... ] $ wget http://ftp-osl.osuosl.org/distfiles/chromium-50.0.2661.94.tar.xz [ ... ] $ wget http://gentoo.ussg.indiana.edu/distfiles/chromium-50.0.2661.94.tar.xz [ ... ] $ rsync ftp.ussg.iu.edu::gentoo-distfiles/distfiles/chromium-50.0.2661.94.tar.xz [ ... ] [ ... ] $ sha256sum chromium* 66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19 chromium-50.0.2661.94.tar.xz (correct) 5daabd61e18bd670329cf664e92c64c324e0f93c2cd72f2b11f8b17e3a4b49b7 chromium-50.0.2661.94.tar.xz.1 (invalid from osuosl) 85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2 chromium-50.0.2661.94.tar.xz.3 (invalid from iu.edu via HTTP) 85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2 chromium-50.0.2661.94.tar.xz.4 (invalid from iu.edu via rsync) $ cmp -b chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.1 chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.1 differ: byte 317835433, line 1239861 is 241 M-! 0 ^@ $ cmp -b chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.3 chromium-50.0.2661.94.tar.xz chromium-50.0.2661.94.tar.xz.3 differ: byte 317653889, line 1239176 is 20 ^P 0 ^@ I suspect there is some hardware issue in the servers responsible for mirroring. The distribution path for distfiles goes dipper.gentoo.org -> FTP-OSL -> $other_mirrors. The copy on dipper is CORRECT, so that means FTP-OSL is the source of corruption. dipper: # stat ./distfiles/chromium-50.0.2661.94.tar.xz File: ‘./distfiles/chromium-50.0.2661.94.tar.xz’ Size: 531491584 Blocks: 1038072 IO Block: 4096 regular file Device: fc04h/64516d Inode: 805317849 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2016-05-01 01:56:05.000000000 +0000 Modify: 2016-04-28 19:33:49.000000000 +0000 Change: 2016-05-01 01:56:08.285454932 +0000 Birth: - # sha256sum ./distfiles/chromium-50.0.2661.94.tar.xz 66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19 ./distfiles/chromium-50.0.2661.94.tar.xz I'm going to touch the file and see if OSL re-fetches it. lance/greg: If it doesn't get re-fetched, can you please drop that file on the mirror so it re-fetches. (In reply to Robin Johnson from comment #3) > The distribution path for distfiles goes dipper.gentoo.org -> FTP-OSL -> > $other_mirrors. > > The copy on dipper is CORRECT, so that means FTP-OSL is the source of > corruption. assuming that the network in between them is OK. also that doesn't explain why the corruption found at iu.edu is slightly different. semi-happily though, other mirrors (tested mirror.us.leaseweb.net, gentoo-distfiles.mirrors.tds.net) have the same corruption as iu.edu. ftp-osl.osuosl.org: $ LANG=C cmp -bl chromium-50.0.2661.94.tar.xz{,.2} 317835433 241 M-! 0 ^@ 317835434 132 Z 0 ^@ 317835435 26 ^V 0 ^@ 317835436 241 M-! 0 ^@ 317835437 363 M-s 0 ^@ 317835438 264 M-4 0 ^@ 317835439 106 F 0 ^@ 317835440 214 M-^L 0 ^@ gentoo-distfiles.mirrors.tds.net: $ LANG=C cmp -bl chromium-50.0.2661.94.tar.xz{,.1} 317653889 20 ^P 0 ^@ 317653890 275 M-= 0 ^@ 317653891 35 ^] 0 ^@ 317653892 256 M-. 0 ^@ 317653893 267 M-7 0 ^@ 317653894 226 M-^V 0 ^@ 317653895 202 M-^B 0 ^@ 317653896 323 M-S 0 ^@ 317689713 40 0 ^@ 317689714 143 c 0 ^@ 317689715 135 ] 0 ^@ 317689716 255 M-- 0 ^@ 317689717 103 C 0 ^@ 317689718 153 k 0 ^@ 317689719 363 M-s 0 ^@ 317689720 163 s 0 ^@ 317835433 241 M-! 0 ^@ 317835434 132 Z 0 ^@ 317835435 26 ^V 0 ^@ 317835436 241 M-! 0 ^@ 317835437 363 M-s 0 ^@ 317835438 264 M-4 0 ^@ 317835439 106 F 0 ^@ 317835440 214 M-^L 0 ^@ (new lines added for ease of reading) also, I got those files at least one and a half hours after your comment, and I think distfiles mirrors are supposed to sync twice an hour? Same on https://mirror.yandex.ru/gentoo-distfiles/distfiles/ Sum is 85549f4f044bcb3f67f30c7726cfce4485dfb263651a577791549319ea0a0af2 (In reply to Robin Johnson from comment #3) > lance/greg: > If it doesn't get re-fetched, can you please drop that file on the mirror so > it re-fetches. This is odd, here's what I see on my end: [root@ftp-osl gentoo]# stat distfiles/chromium-50.0.2661.94.tar.xz File: distfiles/chromium-50.0.2661.94.tar.xz Size: 531491584 Blocks: 1038072 IO Block: 65536 regular file Device: fd06h/64774d Inode: 4298642613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:default_t:s0 Access: 2016-05-05 09:35:35.847228770 +0000 Modify: 2016-04-28 19:33:49.000000000 +0000 Change: 2016-05-01 02:05:57.752528151 +0000 Birth: - [root@ftp-osl gentoo]# sha256sum ./distfiles/chromium-50.0.2661.94.tar.xz 5daabd61e18bd670329cf664e92c64c324e0f93c2cd72f2b11f8b17e3a4b49b7 ./distfiles/chromium-50.0.2661.94.tar.xz So I re-ran the rsync and it didn't update the file so I went ahead and rm'd it manually and did a resync again. Now I see this: [root@ftp-osl gentoo]# stat distfiles/chromium-50.0.2661.94.tar.xz File: distfiles/chromium-50.0.2661.94.tar.xz Size: 531491584 Blocks: 1071104 IO Block: 65536 regular file Device: fd06h/64774d Inode: 4298642613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:default_t:s0 Access: 2016-05-05 15:48:37.855179868 +0000 Modify: 2016-04-28 19:33:49.000000000 +0000 Change: 2016-05-05 15:48:37.855179868 +0000 Birth: - [root@ftp-osl gentoo]# sha256sum ./distfiles/chromium-50.0.2661.94.tar.xz 66f0516b076d42b3f00a5fa4ebf31304cb98973179b1cb2fecda8e0b186dba19 ./distfiles/chromium-50.0.2661.94.tar.xz Not sure what's causing this, the rsync opts we're using is "-avH --delete-after". Is this the only file that had the problem? We did have some xfs corruption the other day and ran fsck on ftp-osl. It might be related to that perhaps. Let me know if this fixes it. (In reply to Lance Albertson from comment #6) > Let me know if this fixes it. the file appears OK when fetched from ftp-osl now, but still broken on iu.edu. this particular file will probably be dropped anyways, but I think it would be a good idea to run rsync --checksum this one time against the master if it's possible to check if any other files have issues. (In reply to Alex Xu (Hello71) from comment #7) > (In reply to Lance Albertson from comment #6) > > Let me know if this fixes it. > > the file appears OK when fetched from ftp-osl now, but still broken on > iu.edu. this particular file will probably be dropped anyways, but I think > it would be a good idea to run rsync --checksum this one time against the > master if it's possible to check if any other files have issues. I verified the checksum on ftp-nyc and ftp-chi to be the same as on the master so what I have is correct now. *** Bug 584922 has been marked as a duplicate of this bug. *** see duplicate bug for more information on this file. For everybody else following this, I wrote & posted a tool, and did a write-up for the mirrors list about the issue: https://archives.gentoo.org/gentoo-mirrors/message/a9fbc7213f832e9918784bb8d334628b I've run this tool on the releases directory on ftp-osl (the gentoo master public mirror) and got the following failures: stage3-alpha-20160210.tar.bz2: FAILED open or read stage3-alpha-20160210.tar.bz2.CONTENTS: OK stage3-armv4tl-20160327.tar.bz2: FAILED open or read stage3-armv4tl-20160327.tar.bz2.CONTENTS: OK (In reply to Lance Albertson from comment #12) > I've run this tool on the releases directory on ftp-osl (the gentoo master > public mirror) and got the following failures: > > stage3-alpha-20160210.tar.bz2: FAILED open or read > stage3-alpha-20160210.tar.bz2.CONTENTS: OK > stage3-armv4tl-20160327.tar.bz2: FAILED open or read > stage3-armv4tl-20160327.tar.bz2.CONTENTS: OK I confirm those files were never uploaded from the relevant build hosts. I have removed the contents+digests files for them. @lance: I have more reports of broken files from the mirrors; and they say it is broken on one of the masterdistfiles.g.o/rsync.osuosl.org rotation (but didn't give an IP sadly). experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2 releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2 (In reply to Robin Johnson from comment #14) > @lance: > I have more reports of broken files from the mirrors; and they say it is > broken on one of the masterdistfiles.g.o/rsync.osuosl.org rotation (but > didn't give an IP sadly). > > experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410. > tar.bz2 > releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud- > 20160602.tar.bz2 I found failures on ftp-chi/ftp-nyc with the following files and fixed them: ./amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2.DIGESTS.asc ./amd64/autobuilds/20160602/install-amd64-minimal-20160602.iso.DIGESTS.asc ./x86/autobuilds/20160531/stage3-i686-20160531.tar.bz2.DIGESTS.asc I did not see any errors with mips32r2 on any mirror. (In reply to Lance Albertson from comment #15) > I did not see any errors with mips32r2 on any mirror. Still getting reports for it: > Right, so masterdistfiles.gentoo.org resolves to the two IP addresses > of 64.50.236.52, 64.50.233.100 here. Consequently, I just now fetched > the file > /experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2 > (to test one of the files I spotted yesterday) once from both IPs, and > in both cases, they yield the sha512sum of > ab01d59eebc811789d1ca592da4330e34233832f07ebaa9808bb12a24b56a1a694a657ad817b7e8de30cd031da6f630d2cf41ad5066e09692b791efefe8960dd > Correct, however, according to the digest files, is > > 758d304df8b24a842466713ee5297b62bc66674041ed462a1f8374a6281455b283411395cccd318b31edc2a3ce289f475fb090eb418bec6a2e1bdd3b4aee4610 *** Bug 583082 has been marked as a duplicate of this bug. *** *** Bug 586150 has been marked as a duplicate of this bug. *** |