Bugzilla Bug 131293

Portage generates duff SHA256 checksums for some files. One example is
xproto-7.0.4.tar.bz2 from x11-proto/xproto-7.0.4 . From the looks of it, the
Crypto.Hash SHA256 module is to blame.

What the digest says:

ciaranm@snowdrop ~ 0 1.96 $ grep SHA256
/usr/portage/x11-proto/xproto/files/digest-xproto-7.0.4 
SHA256 37abbc5cf4b254985de55e50e7c7a3c1588c0e07659fc6664d91719ae0441c03
xproto-7.0.4.tar.bz2 127351

Looks like Python agrees:

ciaranm@snowdrop ~ 0 0.42 $ python -c'from Crypto.Hash import SHA256 ; m =
SHA256.new() ; f = open("/usr/portage/distfiles/xproto-7.0.4.tar.bz2", "r") ;
m.update(f.read()) ; print m.hexdigest()'
37abbc5cf4b254985de55e50e7c7a3c1588c0e07659fc6664d91719ae0441c03

A different implementation, sha256deep, from app-crypt/md5deep, disagrees:

ciaranm@snowdrop ~ 0 0.65 $ sha256deep
/usr/portage/distfiles/xproto-7.0.4.tar.bz2 
f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb 
/usr/portage/distfiles/xproto-7.0.4.tar.bz2

As does gnupg's independent implementation:

ciaranm@snowdrop ~ 1 0.43 $ gpg --print-md sha256
/usr/portage/distfiles/xproto-7.0.4.tar.bz2 
/usr/portage/distfiles/xproto-7.0.4.tar.bz2: 
F2775446 6F13272E E5575F1C B5AC17B9 B46D00D0 347410B1 633E9AF9 15837CBB

As does paludis' different independent implementation:

ciaranm@snowdrop ~ 0 0.36 $ /usr/libexec/paludis/digests/dosha256
/usr/portage/distfiles/xproto-7.0.4.tar.bz2 
f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb

Aaaannnnd app-crypt/hashsum:

ciaranm@snowdrop ~ 0 2.41 $ hashsum -a sha-256
/usr/portage/distfiles/xproto-7.0.4.tar.bz2 
f27754466f13272ee5575f1cb5ac17b9b46d00d0347410b1633e9af915837cbb 
/usr/portage/distfiles/xproto-7.0.4.tar.bz2

I checked my methods used on various other files and got agreement between
Python, Portage and all other implementations, so I don't think my method is
invalid.

[ebuild   R   ] dev-python/pycrypto-2.0.1-r4  USE="-bindist -gmp -test" 0 kB

Some more:

* dev-libs/liboil-0.3.8: liboil-0.3.8.tar.gz
* app-text/gnome-spell-1.0.7-r1: gnome-spell-1.0.7.tar.bz2

Looks like this is affecting sufficiently many packages that it'll end up being
a nuisance...

I can reproduce your bug on the xproto-7.0.4.tar.bz2 test case.  This changeset
from upstream's CVS fixes it:
http://cvs.sourceforge.net/viewcvs.py/pycrypto/crypto/src/SHA256.c?r1=1.3&r2=1.4&diff_format=u
I think it is worth a dev-python/pycrypto revbump (and portage ebuild should
depend on this revision too, sure).

Then someone with a gentoo mirror at hand could run a small script to detect
and regenerate all duff digests. Or, if they are not too numerous yet, then
maybe it's enough to fix them as they are found, i don't know.

My suggested plan of attack is to disable SHA256 generation on the portage side
for a period of time and notify devs to make sure they don't put any new SHA256
digests in the tree.  Then wait a couple months before doing a pycrypto
revbump.  After a couple more months, reenable SHA256 generation on the portage
side.

Or if the problem is sufficiently rare, we can simply revbump pycrypto and be
done with it.

2.0.1-r5 in portage with fix

Imho, you should wait the portage tree is fixed before closing this bug.  I've
just written a small script to check how many wrong digest/Manifest are in the
tree, and the number seems to be actually rather high. There are 65 broken
Manifest in x11-* categories for instance (as for digests, i cant tell, i don't
have direct access to a full repository of distfiles).

So I think that now pycrypto is bumped, one should run some automated re-digest
script, or it will probably end in a lot of bug reports (well, once the tree is
fixed, it will happen too, from people who have not yet updated pycrypto, but
at least it will be simple to point them to the update they need). I will
attach my script in case it can help (yes, it's ugly and slow... oh well...).

Created an attachment (id=85578) [details]
wrong-sha256-sums.sh

As Ciaran explained on gentoo-dev@, it is easy to identify the broken checksums
by the size of the files (size%64 == 55). 
So, why not workarounding the bug in portage for some time, until the tree is
reasonably fixed? It would really reduce annoyance for users imho. I will
attach a simple patch which does that, in case you agree.

Created an attachment (id=85601) [details]
portage-2.1_pre9--sha256-workaround.patch

I don't see how this bug is marked as RESOLVED when it is definately still a
big issue.

For example, I had to put installing a new Gentoo box on hold today whilst
waiting for resolution of these SHA256 discrepancies.

Could a temporary workaround be posted, at least?

(In reply to comment #10)
> I don't see how this bug is marked as RESOLVED when it is definately still a
> big issue.
> 
> For example, I had to put installing a new Gentoo box on hold today whilst
> waiting for resolution of these SHA256 discrepancies.
> 
> Could a temporary workaround be posted, at least?
> 

FEATURES="-strict" should turn off the digest checking.

You are always welcome to downgrade to the stable version of portage which
shouldn't have issues with SHA256 checksums.

Re-opened until all digests in the tree are fixed, Portage team is handling it,
but I will CC QA on this so that they stay informed of the status of digests.

(In reply to comment #7)
> Created an attachment (id=85578) [edit] [details]
> wrong-sha256-sums.sh
> 

Just in case others run into the problem I did (which basicly amounts to me
skimming this bug and not reading it the first time), to run this script you
will need to have emerged md5deep. 

I've just hit this problem, using portage 2.1_pre7-r5, while installing tiff
3.7.4.

After reading this bug, I noticed that upgrading pycrypto from 2.0.1-r4 to
2.0.1-r5 should fix it (if the file has been fixed in the tree), and so it
did.

But I noticed that that pycrypto 2.0.1-r5 is marked ~amd64. Shouldn't it be
made stable, not only because of this particular issue, but because maybe
other apps depend on it generating right hashes?

The _only_ difference among those versions is the sha256 patch, which is
trivial.

Thanks,
    Alberto

> 
> But I noticed that that pycrypto 2.0.1-r5 is marked ~amd64. Shouldn't it be
> made stable, not only because of this particular issue, but because maybe
> other apps depend on it generating right hashes?
> 

No, for two reasons.

#1) Portage-2.1_preX is not stable on any arch, so none of it's dependencies
are required to be stable.

#2) A bug in a program ( even a bug such as this ) does not require immediate
stablization.  This is not a security bug in pycrypto.  It's a normal bug,
normal treatment.

I think we've taken care of this.

Shouldn't the versions of dev-util/pycrypto that are affected by this bug be
removed from the tree (or at least p.masked)?

thanks for letting us know. i've cleaned up the older ebuilds

*** Bug 164320 has been marked as a duplicate of this bug. ***