Portage generates duff SHA256 checksums for some files. One example is xproto-7.0.4.tar.bz2 from x11-proto/xproto-7.0.4 . From the looks of it, the Crypto.Hash SHA256 module is to blame. What the digest says: ciaranm@snowdrop ~ 0 1.96 $ grep SHA256 /usr/portage/x11-proto/xproto/files/digest-xproto-7.0.4 SHA256 37abbc5cf4b254985de55e50e7c7a3c1588c0e07659fc6664d91719ae0441c03 xproto-7.0.4.tar.bz2 127351 Looks like Python agrees: ciaranm@snowdrop ~ 0 0.42 $ python -c'from Crypto.Hash import SHA256 ; m = SHA256.new() ; f = open("/usr/portage/distfiles/xproto-7.0.4.tar.bz2", "r") ; m.update(f.read()) ; print m.hexdigest()' 37abbc5cf4b254985de55e50e7c7a3c1588c0e07659fc6664d91719ae0441c03 A different implementation, sha256deep, from app-crypt/md5deep, disagrees: ciaranm@snowdrop ~ 0 0.65 $ sha256deep /usr/portage/distfiles/xproto-7.0.4.tar.bz2 f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb /usr/portage/distfiles/xproto-7.0.4.tar.bz2 As does gnupg's independent implementation: ciaranm@snowdrop ~ 1 0.43 $ gpg --print-md sha256 /usr/portage/distfiles/xproto-7.0.4.tar.bz2 /usr/portage/distfiles/xproto-7.0.4.tar.bz2: F2775446 6F13272E E5575F1C B5AC17B9 B46D00D0 347410B1 633E9AF9 15837CBB As does paludis' different independent implementation: ciaranm@snowdrop ~ 0 0.36 $ /usr/libexec/paludis/digests/dosha256 /usr/portage/distfiles/xproto-7.0.4.tar.bz2 f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb Aaaannnnd app-crypt/hashsum: ciaranm@snowdrop ~ 0 2.41 $ hashsum -a sha-256 /usr/portage/distfiles/xproto-7.0.4.tar.bz2 f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb /usr/portage/distfiles/xproto-7.0.4.tar.bz2 I checked my methods used on various other files and got agreement between Python, Portage and all other implementations, so I don't think my method is invalid. [ebuild R ] dev-python/pycrypto-2.0.1-r4 USE="-bindist -gmp -test" 0 kB
Some more: * dev-libs/liboil-0.3.8: liboil-0.3.8.tar.gz * app-text/gnome-spell-1.0.7-r1: gnome-spell-1.0.7.tar.bz2 Looks like this is affecting sufficiently many packages that it'll end up being a nuisance...
I can reproduce your bug on the xproto-7.0.4.tar.bz2 test case. This changeset from upstream's CVS fixes it: http://cvs.sourceforge.net/viewcvs.py/pycrypto/crypto/src/SHA256.c?r1=1.3&r2=1.4&diff_format=u I think it is worth a dev-python/pycrypto revbump (and portage ebuild should depend on this revision too, sure). Then someone with a gentoo mirror at hand could run a small script to detect and regenerate all duff digests. Or, if they are not too numerous yet, then maybe it's enough to fix them as they are found, i don't know.
My suggested plan of attack is to disable SHA256 generation on the portage side for a period of time and notify devs to make sure they don't put any new SHA256 digests in the tree. Then wait a couple months before doing a pycrypto revbump. After a couple more months, reenable SHA256 generation on the portage side.
Or if the problem is sufficiently rare, we can simply revbump pycrypto and be done with it.
2.0.1-r5 in portage with fix
Imho, you should wait the portage tree is fixed before closing this bug. I've just written a small script to check how many wrong digest/Manifest are in the tree, and the number seems to be actually rather high. There are 65 broken Manifest in x11-* categories for instance (as for digests, i cant tell, i don't have direct access to a full repository of distfiles). So I think that now pycrypto is bumped, one should run some automated re-digest script, or it will probably end in a lot of bug reports (well, once the tree is fixed, it will happen too, from people who have not yet updated pycrypto, but at least it will be simple to point them to the update they need). I will attach my script in case it can help (yes, it's ugly and slow... oh well...).
Created attachment 85578 [details] wrong-sha256-sums.sh
As Ciaran explained on gentoo-dev@, it is easy to identify the broken checksums by the size of the files (size%64 == 55). So, why not workarounding the bug in portage for some time, until the tree is reasonably fixed? It would really reduce annoyance for users imho. I will attach a simple patch which does that, in case you agree.
Created attachment 85601 [details, diff] portage-2.1_pre9--sha256-workaround.patch
I don't see how this bug is marked as RESOLVED when it is definately still a big issue. For example, I had to put installing a new Gentoo box on hold today whilst waiting for resolution of these SHA256 discrepancies. Could a temporary workaround be posted, at least?
(In reply to comment #10) > I don't see how this bug is marked as RESOLVED when it is definately still a > big issue. > > For example, I had to put installing a new Gentoo box on hold today whilst > waiting for resolution of these SHA256 discrepancies. > > Could a temporary workaround be posted, at least? > FEATURES="-strict" should turn off the digest checking. You are always welcome to downgrade to the stable version of portage which shouldn't have issues with SHA256 checksums. Re-opened until all digests in the tree are fixed, Portage team is handling it, but I will CC QA on this so that they stay informed of the status of digests.
(In reply to comment #7) > Created an attachment (id=85578) [edit] > wrong-sha256-sums.sh > Just in case others run into the problem I did (which basicly amounts to me skimming this bug and not reading it the first time), to run this script you will need to have emerged md5deep.
I've just hit this problem, using portage 2.1_pre7-r5, while installing tiff 3.7.4. After reading this bug, I noticed that upgrading pycrypto from 2.0.1-r4 to 2.0.1-r5 should fix it (if the file has been fixed in the tree), and so it did. But I noticed that that pycrypto 2.0.1-r5 is marked ~amd64. Shouldn't it be made stable, not only because of this particular issue, but because maybe other apps depend on it generating right hashes? The _only_ difference among those versions is the sha256 patch, which is trivial. Thanks, Alberto
> > But I noticed that that pycrypto 2.0.1-r5 is marked ~amd64. Shouldn't it be > made stable, not only because of this particular issue, but because maybe > other apps depend on it generating right hashes? > No, for two reasons. #1) Portage-2.1_preX is not stable on any arch, so none of it's dependencies are required to be stable. #2) A bug in a program ( even a bug such as this ) does not require immediate stablization. This is not a security bug in pycrypto. It's a normal bug, normal treatment.
I think we've taken care of this.
Shouldn't the versions of dev-util/pycrypto that are affected by this bug be removed from the tree (or at least p.masked)?
thanks for letting us know. i've cleaned up the older ebuilds
*** Bug 164320 has been marked as a duplicate of this bug. ***
