When the md5-cache is missing, qwhich from portage-utils 0.97 only returns meaningful results for the first package in each category. Even the meaningful results don't match the output with the md5-cache present. md5-cache present: > $ qwhich 7zip > /var/db/repos/gentoo/app-arch/7zip/7zip-24.06.ebuild > $ qwhich zip > /var/db/repos/gentoo/app-arch/zip/zip-3.0-r7.ebuild md5-cache missing: > $ qwhich 7zip > /var/db/repos/gentoo/app-arch/7zip/7zip-24.05.ebuild > /var/db/repos/gentoo/app-arch/7zip/7zip-24.06.ebuild > $ qwhich zip > /var/db/repos/gentoo/app-arch/zip/zip.ebuild I've tried really hard to get to the bottom of this, but the codebase is a lot more complex than one might imagine. Stepping through with gdb, I can see that it definitely gathers up the other directories into cat_ctx->pkg_ctxs. It seems like atom_explode_cat never sees the version because only the PN gets passed in, but I don't know why. Regarding the differing results for 7zip, you get the two lines even when specifying --latest, which doesn't seem right. Having said that, you only get a single line without --latest when the cache is present, and that doesn't seem right either.
thanks, let me check that later
in the tree with md5-cache missing, do you have a metadata/md5-cache directory?
oh and q -o might help here too, qwhich loops over all overlays by default, so it will report latest from each. That said, the difference in behaviour still makes no sense (yet).
I first hit this on the Gentoo-derivative Flatcar Linux, which hasn't bothered generating a cache in some time because it only has a relatively small set of packages in its repos. However, I can still reproduce it on Gentoo. I demonstrated this by renaming metadata/md5-cache. If I create an empty md5-cache directory, then I get no results at all. I do normally have a few overlays on my system, but these packages only exist in the main gentoo repo. If I rename /etc/portage/repos.conf so that it only sees the default main repo, the behaviour remains the same. I still get an additional result, even though "q -o" only lists one repository.
Ok, that provides some hints. portage-utils doesn't validate the (md5) cache, it just assumes it is correct. Perhaps it is time to do the extra work to ensure the cache is sane.
It's supposed to work when there is no cache. You will find that there this a lot of code for handling ebuilds directly where cachetype is CACHE_EBUILD rather than CACHE_METADATA_MD5. This code probably doesn't get tested very often.
This is actually a bug in the ebuild traversal, it indeed would return bogus after the first category. The md5-cache usage and or verification is another problem that doesn't help in this area.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=eb6d9110d09080e63944da0c1b0265156ab1e326 commit eb6d9110d09080e63944da0c1b0265156ab1e326 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2025-04-21 14:38:28 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2025-04-21 14:38:28 +0000 libq/tree: consider ebuild-tree cache as a cache, not a tree While it is certainly faster to only consider a metadata cache (such as md5-cache) it yields incorrect results on trees in flux or under development. The only correct way is to consider the ebuilds, and look at an existing cache, if it matches. This commit implements this behaviour for libq/tree, used by various q utilities. It considers an cache for an ebuild tree, and uses it for operations where it needs info from inside the ebuilds (such as DEPEND, etc.) over trying to fetch it from the ebuild itself. This is considerably slower than just considering the metadata alone, if only because we now have to traverse multiple extra directories (one for each package) as opposed to just reading the metadata. However, this is unavoidable, and hopefully can be mitigated by parallel scans in the future. Bug: https://bugs.gentoo.org/934549 Bug: https://bugs.gentoo.org/898194 Signed-off-by: Fabian Groffen <grobian@gentoo.org> libq/tree.c | 350 +++++++++++++++++++++++++++++++++++++----------------------- libq/tree.h | 1 + 2 files changed, 219 insertions(+), 132 deletions(-)
Thanks grobian. We used "equery which" as a workaround at the time, but it's still good to see improvements here.
Yeah, sorry for the almost a year delay. My time is not what it used to be, unfortunately.
