Bug 638908 – Implement GLEP 74 full-tree verification support

• Product: Portage Development
• Status: CONFIRMED ---

[Description Michał Górny 2017-11-26 18:33:27 UTC]

This is specified by GLEP 74 [1] and has been already deployed by Infra, so I think it's time to work on matching Portage support. To avoid reinventing too much of a wheel, the API provided by gemato [2] package can be used. It has all the nice tricks to do on-demand verification and is subject to future performance improvements.

Some implementation notes:

1. We *must not* control full-tree signing support via layout.conf or any guessing logic as that defeats its purpose. Instead, we should have a pure repos.conf key to control that, and default it to on for 'gentoo' in /usr/share/portage/config/repos.conf.

1a. This means that people who use git syncing will have to disable it explicitly. We could make some automagic default like 'on IF gentoo AND NOT git ELSE off' but it would be hacky.

2. Portage *must not* read any file from the repository (not even repo_name!) before verifying it. So we must enable the support early on and start using it while initializing the repository configuration.

3. We need to verify every location that might affect the PM operation, both for existing and *missing* files. If a file at specific path does not exist but would affect the PM operation if it existed, it needs to be checked against Manifest. If any file in a specific directory would affect the PM operation, the whole directory should be checked.

4. We may want to perform a complete verification run after syncing. However, I should point out that it's currently a bit slow (45-60 seconds usually), so we might to default to doing only partial on-demand checks by default.

More tips coming.


[1]:https://www.gentoo.org/glep/glep-0074.html
[2]:https://github.com/mgorny/gemato

[Comment 1 Michał Górny 2017-11-26 18:48:36 UTC]

The central part of gemato API is gemato.recursiveloader module that provides ManifestRecursiveLoader. It automatically verifies and loads sub-Manifests as necessary to proceed.

For pure verification behavior, it would be initialized like:

  import os.path
  from gemato.recursiveloader import ManifestRecursiveLoader
  from gemato.openpgp import OpenPGPEnvironment

  openpgp_env = OpenPGPEnvironment()
  openpgp_env.import_key('/var/lib/gentoo/gkeys/keyrings/gentoo/release/pubring.gpg')
  ManifestRecursiveLoader(os.path.join(repository_path, 'Manifest'),
    verify_openpgp=True,
    openpgp_env=openpgp_env)

Then to clean up temporary files:

  openpgp_env.close()

(OpenPGPEnvironment also supports context manager API)


The runtime API bits of interest are:

1. .find_timestamp() -- to get the timestamp from Manifest for checking. GLEP 74 suggests that Portage should refuse to operate if timestamp is much older than time of sync (i.e. mirrors are pushing very old data).

2. .assert_path_verifies(relpath) -- checks a single path in repo against the Manifest. @relpath is the path relative to top repo directory. It can be used both for files that do and do not exist, and it appropriately fails if __exists__ state mismatches.

3. .assert_directory_verifies(relpath) -- checks the whole directory in repo. @relpath is the path relative to top repo directory. It fails for any mismatched file as well as for stray files.


Functions 2/3 throw exceptions on Manifest mismatches. Additionally, any function that loads Manifests (starting with the constructor) can throw exception if sub-Manifests fail verification. For full list of exceptions, see `pydoc gemato.exceptions`.

I think those functions are enough for the initial implementation. If you have any more ideas, let me know and I can add additional API bits.


By the way, there's also .find_dist_entry(distname, package_path) if you want to avoid separate Manifest parsing code for distfiles.

And although I haven't tested it, I think RecursiveManifestLoader() will work just fine for processing thick and thin package Manifests for repos not using the full structure. However, I suppose the 'lower level' API of gemato.manifest could be preferred for that. Of course, that assuming you don't mind making Portage depend on gemato unconditionally.

[Comment 2 Zac Medico 2018-04-07 17:22:36 UTC]

*** Bug 480190 has been marked as a duplicate of this bug. ***

[Comment 3 Sam James 2023-07-14 10:31:09 UTC]

(I suspect either a lot of this or all of this is already done, but not closing until I or someone actually checks.)