Bug 11530 – Change portage to md5 sum all files, not just DISTDIR ones

• Product: Portage Development
• Status: RESOLVED FIXED

[Description J Robert Ray 2002-12-03 07:11:31 UTC]

This patch is a step towards pgp-signed ebuilds.  The first goal is to have a
single file that can be pgp-signed.  This single file should have MD5 sums of
not just the downloaded files but of the ebuild files themselves, and all the
files in FILESDIR.  PGP-signing this file will protect any file in the portage
tree from tampering (exception: eclass files are not considered at this point).

Instead of having a digest file per ebuild version, there now is only one file
called 'digests' the resides in FILESDIR.

When a new 'digests' file is created, the existing 'digests' file is loaded,
entries in this file are updated to their current MD5 sum and file size, and new
files are added.  When the 'digests' file is written back out, existing entries
for files that the current ebuild has no knowledge of are left in place.

The reason for this is one ebuild will have different dist files than a newer
version, when calculating the md5sums for one ebuild, it must not remove the
sums of the dist files used in any other ebuild.

Also note that to create a complete 'digests' file for an existing package, one
must run 'ebuild ... digest' on EACH ebuild file in order to download and MD5
sum all of the dist files used in all of the ebuild versions.

My patch makes the transition to this new technique somewhat easier by falling
back to pre-existing digest-... files if the 'digests' files does not exist. 
This can be disabled, however, by adding 'gnupg' to FEATURES.

An example 'digests' file looks like so:

MD5 7ca6ca87a8fca531a0a4b505f51296d4 DISTDIR/Judy-initial_LGPL.src.tar.gz 296274
MD5 20ab5d81cf47fda8c319d404e2d4046d O/ChangeLog 256
MD5 65d04fc92f360c6852990d877e2a8619 O/judy-20020627.ebuild 1069

The faux DISTDIR/ or O/ prefix on each file tells portage where this file is to
be found.  Otherwise the file format is the same as the existing digest-... file
format.  "O" comes from the use of 'settings["O"]' inside portage.py, it is the
path to the ebuild.

When the 'digests' file is created, the directory tree where the ebuild lives is
walked and digests are created for each file found.  Then, when the digests are
verified, the same directory tree is walked again: every file found must have a
correct entry in the 'digests' file.  The dist files are also checked, of course.

The MD5 sums for files in the ebuild directory itself are calculated in such a
way as would be equal to this command:  'grep -v ^# | md5sum'

When the ebuild files are committed to CVS, the RCS tags ($Header: $) are
modified.  Since committing happens after the digest is calculated, the MD5 hash
would be invalidated.  It is necessary to compute the MD5 hash of the file with
the RCS tags removed to prevent this.  This patch simply ignores comment lines.

Creating/updating the 'digests' file should be the last step before committing
changes to CVS.  Because now it is so easy to invalidate the MD5 sums in the
'digests' file, repoman should probably be modified to validate the 'digests'
file before allowing a commit.

When old ebuilds or files from FILESDIR are deleted, their entries should be
removed from the 'digests' file.

[Comment 1 J Robert Ray 2002-12-03 07:12:38 UTC]

Created attachment 6170 [details, diff]
digests.diff

[Comment 2 Martin Holzer (RETIRED) 2003-05-22 13:21:07 UTC]

i think is already done

[Comment 3 Martin Holzer (RETIRED) 2003-05-22 14:02:41 UTC]

forget to close it

Manifest does this