Bugzilla Bug 164462

Distfile nwresources129.tar.gz has incorrect sha256 digest entry in Manifest,
though it has correct md5 and rmd160 digests.

$ sha256sum /var/gentoo/distfiles/nwresources129.tar.gz
4da35c38c63d7cbe40047833103fe0d56f417d1895fb572d52c23a873619f537 
/var/gentoo/distfiles/nwresources129.tar.gz

$ grep -r nwresources * | grep SHA256
files/digest-nwn-data-1.29-r1:SHA256
40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2
nwresources129.tar.gz 1212356127
files/digest-nwn-data-1.29:SHA256
40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2
nwresources129.tar.gz 1212356127
Manifest:DIST nwresources129.tar.gz 1212356127 RMD160
1a986d12bc3153ff7646053e60d610155944b5ac SHA1
6f399dbf876f357a57096996447d4b9fa68bcb58 SHA256
40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2

 * Checking MD5 for nwresources129.tar.gz ...                            [ ok ]
 * Checking RMD160 for nwresources129.tar.gz ...                         [ ok ]
 * Checking SHA256 for nwresources129.tar.gz ...                         [ !! ]

This is strange.  I am getting the same results.

wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ rm
files/digest-nwn-data-1.29*
wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ rm Manifest 
wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ ebuild
nwn-data-1.29.ebuild digest
>>> Creating Manifest for /usr/local/portage/games-rpg/nwn-data
wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ ll
total 44
-rw-r--r-- 1 wolf31o2 users  4393 2007-01-11 12:06 ChangeLog
drwxr-xr-x 2 wolf31o2 users  4096 2007-01-31 17:07 files
-rw-r--r-- 1 wolf31o2 users  3567 2007-01-31 17:07 Manifest
-rw-r--r-- 1 wolf31o2 users   254 2007-01-11 12:06 metadata.xml
-rw-r--r-- 1 wolf31o2 users  6787 2007-01-11 12:06 nwn-data-1.29.ebuild
-rw-r--r-- 1 wolf31o2 users 15925 2007-01-11 14:17 nwn-data-1.29-r1.ebuild
wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ cat Manifest | grep
nwresources
DIST nwresources129.tar.gz 1212356127 RMD160
1a986d12bc3153ff7646053e60d610155944b5ac SHA1
6f399dbf876f357a57096996447d4b9fa68bcb58 SHA256
40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2
wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ grep nwresources
files/digest-nwn-data-1.29-r1 | grep SHA256
SHA256 40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2
nwresources129.tar.gz 1212356127

wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ sha256sum
/usr/portage/distfiles/nwresources129.tar.gz 
4da35c38c63d7cbe40047833103fe0d56f417d1895fb572d52c23a873619f537 
/usr/portage/distfiles/nwresources129.tar.gz
wolf31o2@inertia /usr/local/portage/games-rpg/nwn-data $ shash -a SHA256
/usr/portage/distfiles/nwresources129.tar.gz 
# SHA256 HASH
4da35c38c63d7cbe40047833103fe0d56f417d1895fb572d52c23a873619f537 
/usr/portage/distfiles/nwresources129.tar.gz

Portage: How are the SHA256 digests calculated?

(In reply to comment #1)
> Portage: How are the SHA256 digests calculated?

It uses pycrypto.  We've had problems with pycrypto and sha256 before (bug
131293).

OK.  I am definitely able to reproduce this with games-rpg/nwn-data and:

[ebuild   R   ] sys-apps/portage-2.1.2-r7  
[ebuild   R   ] dev-python/pycrypto-2.0.1-r5  

@wolflo, kindly state the arch you're running, gcc, etc.

If you know the arch/gcc of the system that generated the initial sha256, would
be useful; it's paranoid, but if pycrypto is again screwing up, that info will
help nail down the issue

$ emerge --info
Portage 2.1.2-r9 (default-linux/x86/2006.1, gcc-4.1.2, glibc-2.5-r0,
2.6.18-gentoo-r3 i686)
=================================================================
System uname: 2.6.18-gentoo-r3 i686 Intel(R) Core(TM)2 CPU         T7200  @
2.00GHz
Gentoo Base System release 1.12.9
Timestamp of tree: Mon, 19 Feb 2007 15:30:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632)
[enabled]
dev-java/java-config: 1.3.7, 2.0.31-r3
dev-lang/python:     2.4.4
dev-python/pycrypto: 2.0.1-r5
sys-apps/sandbox:    1.2.18.1
sys-devel/autoconf:  2.13, 2.61
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.17
sys-devel/gcc-config: 1.3.14
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.20
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=prescott -msse3 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf
/etc/java-config/vms/ /etc/revdep-rebuild /etc/splash /etc/terminfo"
CXXFLAGS="-O2 -march=prescott -msse3 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="ccache collision-protect cvs distcc distlocks metadata-transfer
sandbox sfperms sign"
GENTOO_MIRRORS="ftp://ftp.ussg.iu.edu/pub/linux/gentoo
ftp://gentoo.mirrors.tds.net/gentoo
http://ftp.snt.utwente.nl/pub/os/linux/gentoo http://ftp.heanet.ie/pub/gentoo/
http://85.25.128.62"
LC_ALL="en_US.UTF-8"
LINGUAS="en"
MAKEOPTS="-j8"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --delete-after --stats --timeout=180
--exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /var/svnroot/wolf31o2
/usr/portage/local/layman/vmware /usr/portage/local/layman/efika
/usr/portage/local/layman/nouveau"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X acpi alsa branding cairo cdr crypt cups dbus dlloader dvd dvdr eds
encode esd ethereal evo fam firefox gif gnome gphoto2 gstreamer gtk gtk2 hal
java jpeg ldap mmx mozbranding mp3 mpeg ncurses nls nocd nptl nptlonly ogg
openal opengl oss pam pcmcia pdf perl pic png pnp ppds quicktime readline samba
snmp spell sse sse2 ssl svg tcpd tiff truetype trusted udev unicode usb videos
vorbis win32codecs wireshark x86 xml xorg xv xvid zlib" ALSA_CARDS="hda-intel"
ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file
hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route
share shm softvol" CAMERAS="kodak" ELIBC="glibc" INPUT_DEVICES="evdev keyboard
mouse synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk
hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" USERLAND="GNU"
VIDEO_CARDS="nvidia nv vesa"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LDFLAGS,
PORTAGE_RSYNC_EXTRA_OPTS

I recently switched the CPU from a Core Duo to a Core 2 Duo, so everything is
still setup for Core.  I also recently upgraded my world, so I've rebuilt
pycrypto and am re-running the digest.  I made sure to clear out the pyc/pyo
files under /usr/lib/python2.4/site-packages/Crypto, too, just in case.

wolf31o2@inertia ~ $ shash -a SHA256
/usr/portage/distfiles/nwresources129.tar.gz 
# SHA256 HASH
4da35c38c63d7cbe40047833103fe0d56f417d1895fb572d52c23a873619f537 
/usr/portage/distfiles/nwresources129.tar.gz
wolf31o2@inertia ~ $ sha256sum /usr/portage/distfiles/nwresources129.tar.gz
4da35c38c63d7cbe40047833103fe0d56f417d1895fb572d52c23a873619f537 
/usr/portage/distfiles/nwresources129.tar.gz
wolf31o2@inertia /var/cvsroot/gentoo-x86/games-rpg/nwn-data $ ebuild
nwn-data-1.29-r1.ebuild digest
Appending /var/cvsroot/gentoo-x86 to PORTDIR_OVERLAY...
>>> Creating Manifest for /var/cvsroot/gentoo-x86/games-rpg/nwn-data
wolf31o2@inertia /var/cvsroot/gentoo-x86/games-rpg/nwn-data $ grep SHA256
Manifest | grep nwresources
DIST nwresources129.tar.gz 1212356127 RMD160
1a986d12bc3153ff7646053e60d610155944b5ac SHA1
6f399dbf876f357a57096996447d4b9fa68bcb58 SHA256
40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2

What's the output of

python -c 'from Crypto.Hash import SHA256; print
SHA256.new(open("nwresources129.tar.gz", "r").read()).hexdigest()'

wolf31o2@inertia ~ $ python -c 'from Crypto.Hash import SHA256; print
SHA256.new(open("/usr/portage/distfiles/nwresources129.tar.gz",
"r").read()).hexdigest()'
40e7bb8aebae931c2ce05f199ed15714845b403c4a6d9fab88092e784214bee2

*** Bug 175585 has been marked as a duplicate of this bug. ***

Also seeing this for the 700Mb ophcrack tables (available on
distfiles.gentoo.org as SSTIC04-5k.zip).  SHA256 from pycrypto is wrong and
disagrees with sha256sum from coreutils and also the hashlib sha256
implementation.  The original hash (the one in Manifest) was made using
pycrypto awhile ago and was also wrong.  The error now occurs presumably
because I've got python-2.5 and so will have started using the hashlib library
instead as of 2.1.3_rc6...

I can attach my emerge --info if it'll help?

Created an attachment (id=125518) [details]
fix from upstream

This patch from upstream seems to correct the problem:

http://pycrypto.cvs.sourceforge.net/pycrypto/crypto/src/SHA256.c?r1=1.4&r2=1.5

Without the patch, the sha256 digest for SSTIC04-5k.zip is:
910cfab447d273564a374d62620d50812d0623657905835da152d53c3542552f

With the patch, the sha256 digest for SSTIC04-5k.zip is:
9ba2c9e6faeb658a77a342a2d45c47e593daae9e087be3254eb1b050b324cf10

The patch brings the sha256 digest into agreement with that calculated by
sha256sum from coreutils.

@python herd: can we do a pycrypto revbump with this patch?

pycrypto-2.0.1-r6 is in cvs with this patch included.

I guess we can remove ourselves from this.  I've manually updated the
nwresources129 digest/Manifest a long time ago.

Similarly, I've also re-digested SSTIC04-5k.zip.  Perhaps this bug can be
marked as resolved?

The new version is stable on all the really relevant archs, so I think we can
consider this fixed.

*** Bug 192603 has been marked as a duplicate of this bug. ***