Currently, portage assumes all repositories are mutable, which hurts performance for dependency calculations. If we allow the user to declare a repository as immutable, then this information can be used to optimize dependency calculations to avoid redundant cache validation. For immutable repositories, cache will only have to be validated after each sync.
For the search indexing proposed in bug 525718, we could have it ignore the search index for mutable repositories, and have a repos.conf override that says "trust the search index for this repo even though it happens to be mutable".
Support for MetaManifest will be very handy for this, since a hash of the top-level MetaManifest can be used to identify an immutable repository state.
(In reply to Zac Medico from comment #2) > Support for MetaManifest will be very handy for this, since a hash of the > top-level MetaManifest can be used to identify an immutable repository state. We should be able to assume valid cache for repos that have a "sync-rsync-verify-metamanifest = yes" setting.
I have a patch for this locally but not sure if it's worth it yet. When doing some digging a while ago, I did come across https://github.com/gentoo/portage/commit/a72a01746638debe472496bd8fc661992a6ba08b but it got reverted later on.
(In reply to Sam James from comment #4) > I have a patch for this locally but not sure if it's worth it yet. > > When doing some digging a while ago, I did come across > https://github.com/gentoo/portage/commit/ > a72a01746638debe472496bd8fc661992a6ba08b but it got reverted later on. Ah: https://github.com/gentoo/portage/commit/5c7bf4eb09f644813a6f017ffd91665664142560
Created attachment 794669 [details, diff] 0001-porttree-skip-metadata-verification-if-repository-is.patch Attaching patch. It's not for review or anything, just so if I forget about it and don't end up benchmarking, maybe someone else can use it.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=f05140beb2dd9a3577ff2042941cb3e4fbb1df31 commit f05140beb2dd9a3577ff2042941cb3e4fbb1df31 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-12-17 05:15:27 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-12-21 01:28:02 +0000 porttree: skip metadata verification if repository is immutable (not volatile) With Python 3.11, I get: - ~232695 total syscalls before (38268 read calls) - ~203656 total syscalls after (27381 read calls) It's a cheap improvement for simply promising to not touch ebuilds in a repository. Bug: https://bugs.gentoo.org/528394 See: a72a01746638debe472496bd8fc661992a6ba08b See: 5c7bf4eb09f644813a6f017ffd91665664142560 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/dbapi/porttree.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=ef123a214708c85f9802f2a649b93137fd2ee3be commit ef123a214708c85f9802f2a649b93137fd2ee3be Author: Sam James <sam@gentoo.org> AuthorDate: 2022-12-17 05:12:46 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-12-21 01:28:02 +0000 config: Add 'volatile' configuration option This introduces a 'volatile' configuration option for repos.conf. * If a repository is marked as volatile (volatile=true), then Portage will assume the repository may be changed at any time, and that it may contain e.g. local changes. Portage will not perform optimizations based on assuming its integrity. It will also not perform destructive operations if it is a git repository (it will not prioritise being able to continue to sync). * If a repository is marked as non-volatile (volatile=false), then Portage will assume that only Portage changes the repository and that it will not be modified otherwise. Portage will perform optimizations based on assuming this, like not needing to verify the metadata cache for ebuilds & eclasses. Portage will prioritise the repository continuing to work and may e.g. 'git reset --hard' or 'git clean -fdx' in order to keep git repositories working for sync purposes. (If you do not like this, please set 'volatile=true'). Bug: https://bugs.gentoo.org/528394 Bug: https://bugs.gentoo.org/887025 See: https://github.com/gentoo/portage/pull/801 See: https://github.com/gentoo/portage/pull/931 See: https://github.com/gentoo/portage/pull/939 See: https://github.com/gentoo/portage/pull/960 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/repository/config.py | 40 +++++++++++++++++++++++++++++++++++++++- man/portage.5 | 8 ++++++++ 2 files changed, 47 insertions(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2f020ae6aca04a0db68b7ceb5413e2b815d2169d commit 2f020ae6aca04a0db68b7ceb5413e2b815d2169d Author: Sam James <sam@gentoo.org> AuthorDate: 2022-12-21 01:30:22 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-12-21 01:30:55 +0000 Revert "porttree: skip metadata verification if repository is immutable (not volatile)" This reverts commit f05140beb2dd9a3577ff2042941cb3e4fbb1df31. I didn't mean to push this one yet, which we're going to handle separately. Bug: https://bugs.gentoo.org/528394 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/dbapi/porttree.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
