131293 2006-04-25 23:31 0000 Portage generates duff SHA256 checksums 2007-01-28 22:55:47 0000 1 1 1 Unclassified Portage Development Core unspecified All Linux RESOLVED FIXED P2 normal --- 1 ciaran.mccreesh@googlemail.com python@gentoo.org albertito@gmail.com bugs@iplod.net dev-portage@gentoo.org esenkweb@gmail.com lisa@gentoo.org qa@gentoo.org rockoo@gmail.com sgtphou@fire-eyes.org tom.gl@free.fr zlin@gentoo.org ciaran.mccreesh@googlemail.com 2006-04-25 23:31:07 0000 Portage generates duff SHA256 checksums for some files. One example is xproto-7.0.4.tar.bz2 from x11-proto/xproto-7.0.4 . From the looks of it, the Crypto.Hash SHA256 module is to blame. What the digest says: ciaranm@snowdrop ~ 0 1.96 $ grep SHA256 /usr/portage/x11-proto/xproto/files/digest-xproto-7.0.4 SHA256 37abbc5cf4b254985de55e50e7c7a3c1588c0e07659fc6664d91719ae0441c03 xproto-7.0.4.tar.bz2 127351 Looks like Python agrees: ciaranm@snowdrop ~ 0 0.42 $ python -c'from Crypto.Hash import SHA256 ; m = SHA256.new() ; f = open("/usr/portage/distfiles/xproto-7.0.4.tar.bz2", "r") ; m.update(f.read()) ; print m.hexdigest()' 37abbc5cf4b254985de55e50e7c7a3c1588c0e07659fc6664d91719ae0441c03 A different implementation, sha256deep, from app-crypt/md5deep, disagrees: ciaranm@snowdrop ~ 0 0.65 $ sha256deep /usr/portage/distfiles/xproto-7.0.4.tar.bz2 f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb /usr/portage/distfiles/xproto-7.0.4.tar.bz2 As does gnupg's independent implementation: ciaranm@snowdrop ~ 1 0.43 $ gpg --print-md sha256 /usr/portage/distfiles/xproto-7.0.4.tar.bz2 /usr/portage/distfiles/xproto-7.0.4.tar.bz2: F2775446 6F13272E E5575F1C B5AC17B9 B46D00D0 347410B1 633E9AF9 15837CBB As does paludis' different independent implementation: ciaranm@snowdrop ~ 0 0.36 $ /usr/libexec/paludis/digests/dosha256 /usr/portage/distfiles/xproto-7.0.4.tar.bz2 f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb Aaaannnnd app-crypt/hashsum: ciaranm@snowdrop ~ 0 2.41 $ hashsum -a sha-256 /usr/portage/distfiles/xproto-7.0.4.tar.bz2 f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb /usr/portage/distfiles/xproto-7.0.4.tar.bz2 I checked my methods used on various other files and got agreement between Python, Portage and all other implementations, so I don't think my method is invalid. [ebuild R ] dev-python/pycrypto-2.0.1-r4 USE="-bindist -gmp -test" 0 kB ciaran.mccreesh@googlemail.com 2006-04-26 07:21:53 0000 Some more: * dev-libs/liboil-0.3.8: liboil-0.3.8.tar.gz * app-text/gnome-spell-1.0.7-r1: gnome-spell-1.0.7.tar.bz2 Looks like this is affecting sufficiently many packages that it'll end up being a nuisance... tom.gl@free.fr 2006-04-26 10:28:55 0000 I can reproduce your bug on the xproto-7.0.4.tar.bz2 test case. This changeset from upstream's CVS fixes it: http://cvs.sourceforge.net/viewcvs.py/pycrypto/crypto/src/SHA256.c?r1=1.3&r2=1.4&diff_format=u I think it is worth a dev-python/pycrypto revbump (and portage ebuild should depend on this revision too, sure). Then someone with a gentoo mirror at hand could run a small script to detect and regenerate all duff digests. Or, if they are not too numerous yet, then maybe it's enough to fix them as they are found, i don't know. zmedico@gentoo.org 2006-04-26 15:27:05 0000 My suggested plan of attack is to disable SHA256 generation on the portage side for a period of time and notify devs to make sure they don't put any new SHA256 digests in the tree. Then wait a couple months before doing a pycrypto revbump. After a couple more months, reenable SHA256 generation on the portage side. zmedico@gentoo.org 2006-04-26 15:30:39 0000 Or if the problem is sufficiently rare, we can simply revbump pycrypto and be done with it. vapier@gentoo.org 2006-04-26 16:39:30 0000 2.0.1-r5 in portage with fix tom.gl@free.fr 2006-04-26 17:12:57 0000 Imho, you should wait the portage tree is fixed before closing this bug. I've just written a small script to check how many wrong digest/Manifest are in the tree, and the number seems to be actually rather high. There are 65 broken Manifest in x11-* categories for instance (as for digests, i cant tell, i don't have direct access to a full repository of distfiles). So I think that now pycrypto is bumped, one should run some automated re-digest script, or it will probably end in a lot of bug reports (well, once the tree is fixed, it will happen too, from people who have not yet updated pycrypto, but at least it will be simple to point them to the update they need). I will attach my script in case it can help (yes, it's ugly and slow... oh well...). tom.gl@free.fr 2006-04-26 17:14:05 0000 Created an attachment (id=85578) wrong-sha256-sums.sh tom.gl@free.fr 2006-04-27 05:03:54 0000 As Ciaran explained on gentoo-dev@, it is easy to identify the broken checksums by the size of the files (size%64 == 55). So, why not workarounding the bug in portage for some time, until the tree is reasonably fixed? It would really reduce annoyance for users imho. I will attach a simple patch which does that, in case you agree. tom.gl@free.fr 2006-04-27 05:04:25 0000 Created an attachment (id=85601) portage-2.1_pre9--sha256-workaround.patch bugs@iplod.net 2006-04-27 07:49:43 0000 I don't see how this bug is marked as RESOLVED when it is definately still a big issue. For example, I had to put installing a new Gentoo box on hold today whilst waiting for resolution of these SHA256 discrepancies. Could a temporary workaround be posted, at least? antarus@gentoo.org 2006-04-27 13:54:46 0000 (In reply to comment #10) > I don't see how this bug is marked as RESOLVED when it is definately still a > big issue. > > For example, I had to put installing a new Gentoo box on hold today whilst > waiting for resolution of these SHA256 discrepancies. > > Could a temporary workaround be posted, at least? > FEATURES="-strict" should turn off the digest checking. You are always welcome to downgrade to the stable version of portage which shouldn't have issues with SHA256 checksums. Re-opened until all digests in the tree are fixed, Portage team is handling it, but I will CC QA on this so that they stay informed of the status of digests. bill_krueger@verizon.net 2006-04-27 14:54:27 0000 (In reply to comment #7) > Created an attachment (id=85578) [edit] > wrong-sha256-sums.sh > Just in case others run into the problem I did (which basicly amounts to me skimming this bug and not reading it the first time), to run this script you will need to have emerged md5deep. albertito@gmail.com 2006-05-02 14:48:12 0000 I've just hit this problem, using portage 2.1_pre7-r5, while installing tiff 3.7.4. After reading this bug, I noticed that upgrading pycrypto from 2.0.1-r4 to 2.0.1-r5 should fix it (if the file has been fixed in the tree), and so it did. But I noticed that that pycrypto 2.0.1-r5 is marked ~amd64. Shouldn't it be made stable, not only because of this particular issue, but because maybe other apps depend on it generating right hashes? The _only_ difference among those versions is the sha256 patch, which is trivial. Thanks, Alberto antarus@gentoo.org 2006-05-02 15:31:56 0000 > > But I noticed that that pycrypto 2.0.1-r5 is marked ~amd64. Shouldn't it be > made stable, not only because of this particular issue, but because maybe > other apps depend on it generating right hashes? > No, for two reasons. #1) Portage-2.1_preX is not stable on any arch, so none of it's dependencies are required to be stable. #2) A bug in a program ( even a bug such as this ) does not require immediate stablization. This is not a security bug in pycrypto. It's a normal bug, normal treatment. antarus@gentoo.org 2006-05-26 08:13:50 0000 I think we've taken care of this. zlin@gentoo.org 2006-09-30 19:33:01 0000 Shouldn't the versions of dev-util/pycrypto that are affected by this bug be removed from the tree (or at least p.masked)? liquidx@gentoo.org 2006-10-03 08:21:47 0000 thanks for letting us know. i've cleaned up the older ebuilds zmedico@gentoo.org 2007-01-28 22:39:25 0000 *** Bug 164320 has been marked as a duplicate of this bug. *** 85578 2006-04-26 17:14 0000 wrong-sha256-sums.sh wrong-sha256-sums.sh text/plain IyEvYmluL2Jhc2gKClBPUlRESVI9JChwb3J0YWdlcSBwb3J0ZGlyKQpESVNURElSPSQocG9ydGFn ZXEgZW52dmFyIERJU1RESVIpCgpjZCAiJHtQT1JURElSfSIKW1sgJCMgLWxlIDAgXV0gXAoJJiYg U0VBUkNIX1BBVEhTPSQobHMgLWQgKi0qKSBcCgl8fCBTRUFSQ0hfUEFUSFM9IiR7QH0iCgojIEJy b2tlbiBkaWdlc3QgKHdyb25nIFNIQTI1NiBvbiBzb21lICRESVNURklMRVMgZmlsZXMpCmZpbmQg JHtTRUFSQ0hfUEFUSFN9IC1uYW1lICJkaWdlc3QtKiIgfCBcCndoaWxlIHJlYWQgZGlnZXN0IDsg ZG8KCXBrZ2Rpcj0iJHtkaWdlc3QlJS9maWxlcy8qfSIKCWVidWlsZD0iJHtwa2dkaXJ9LyR7ZGln ZXN0IyMqL2ZpbGVzL2RpZ2VzdC19LmVidWlsZCIKCXNlZCAtbiAnczpeU0hBMjU2IFwoWzAtOWEt Zl1cK1wpIFwoW14gXVwrXCkgLio6XDIgXDE6cCcgIiR7ZGlnZXN0fSIgfCBcCgl3aGlsZSByZWFk IGRpZ2VzdF9maWxlIGRpZ2VzdF9zaGEgOyBkbwoJCVtbIC1mICR7RElTVERJUn0vJHtkaWdlc3Rf ZmlsZX0gXV0gfHwgY29udGludWUKCQlnb29kX3NoYT0kKHNoYTI1NmRlZXAgIiR7RElTVERJUn0v JHtkaWdlc3RfZmlsZX0iKQoJCWdvb2Rfc2hhPSIke2dvb2Rfc2hhJSUgKn0iCgkJW1sgJHtnb29k X3NoYX0gPT0gJHtkaWdlc3Rfc2hhfSBdXSAmJiBjb250aW51ZQoJCWVjaG8gIldyb25nIGRpZ2Vz dCBmb3IgJHtlYnVpbGR9IG9uICR7ZGlnZXN0X2ZpbGV9IgoJCSNmaXg6IGVidWlsZCAiJHtlYnVp bGR9IiBkaWdlc3QKCWRvbmUKZG9uZQoKIyBCcm9rZW4gTWFuaWZlc3QgKHdyb25nIFNIQTI1NiBv biBzb21lICRQT1JURElSIGZpbGVzKQpmaW5kICR7U0VBUkNIX1BBVEhTfSAtbmFtZSAiTWFuaWZl c3QiIHwgXAp3aGlsZSByZWFkIG1hbmlmZXN0IDsgZG8KCXBrZ2Rpcj0iJHttYW5pZmVzdCUlL01h bmlmZXN0fSIKCWVidWlsZD0kKGxzIC0xICR7cGtnZGlyfS8qLmVidWlsZCAyPi9kZXYvbnVsbCB8 IGhlYWQgLW4gMSkgfHwgY29udGludWUKCXNlZCAtbiAnczpeU0hBMjU2IFwoWzAtOWEtZl1cK1wp IFwoW14gXVwrXCkgLio6XDIgXDE6cCcgIiR7bWFuaWZlc3R9IiB8IFwKCXdoaWxlIHJlYWQgbWFu aWZlc3RfZmlsZSBtYW5pZmVzdF9zaGEgOyBkbwoJCVtbIC1mICR7cGtnZGlyfS8ke21hbmlmZXN0 X2ZpbGV9IF1dIHx8IGNvbnRpbnVlCgkJZ29vZF9zaGE9JChzaGEyNTZkZWVwICIke3BrZ2Rpcn0v JHttYW5pZmVzdF9maWxlfSIpCgkJZ29vZF9zaGE9IiR7Z29vZF9zaGElJSAqfSIKCQlbWyAke2dv b2Rfc2hhfSA9PSAke21hbmlmZXN0X3NoYX0gXV0gJiYgY29udGludWUKCQllY2hvICJXcm9uZyBN YW5pZmVzdCBpbiAke3BrZ2Rpcn0gb24gJHttYW5pZmVzdF9maWxlfSIKCQkjZml4OiBlYnVpbGQg IiR7ZWJ1aWxkfSIgZGlnZXN0Cglkb25lCmRvbmUK 85601 2006-04-27 05:04 0000 portage-2.1_pre9--sha256-workaround.patch portage-2.1_pre9--sha256-workaround.patch text/plain LS0tIHB5bS9wb3J0YWdlX2NoZWNrc3VtLnB5Lm9yaWcJMjAwNi0wNC0yNyAxMzo0MzowNC4wMDAw MDAwMDAgKzAyMDAKKysrIHB5bS9wb3J0YWdlX2NoZWNrc3VtLnB5CTIwMDYtMDQtMjcgMTM6NTg6 MjQuMDAwMDAwMDAwICswMjAwCkBAIC0xMDIsNiArMTAyLDkgQEAKIAkJCWNvbnRpbnVlCiAJCWVs aWYgeCBpbiBoYXNoZnVuY19tYXAua2V5cygpOgogCQkJbXloYXNoID0gcGVyZm9ybV9jaGVja3N1 bShmaWxlbmFtZSwgeCwgY2FsY19wcmVsaW5rPWNhbGNfcHJlbGluaylbMF0KKwkJCSNYWFg6IHRl bXBvcmFyeSB3b3JrYXJvdW5kIGZvciBidWcgIzEzMTI5MzoKKwkJCWlmIHggPT0gIlNIQTI1NiIg YW5kIG15ZGljdFsic2l6ZSJdJTY0ID09IDU1OgorCQkJCWNvbnRpbnVlCiAJCQlpZiBteWRpY3Rb eF0gIT0gbXloYXNoOgogCQkJCWlmIHN0cmljdDoKIAkJCQkJcmFpc2UgcG9ydGFnZV9leGNlcHRp b24uRGlnZXN0RXhjZXB0aW9uLCAiRmFpbGVkIHRvIHZlcmlmeSAnJChmaWxlKXMnIG9uIGNoZWNr c3VtIHR5cGUgJyUodHlwZSlzJyIgJSB7ImZpbGUiOmZpbGVuYW1lLCAidHlwZSI6eH0K