When using qwhich to search for packages in custom overlays it might return files no longer existing on disk. I can see this with pretty much every package in those repositories since i've started using q applets. It does work well with new Ebuilds being added. It doesn't matter if it's just a new version or entirely new packages. But it won't remove any delete files. I haven't found a way force it to updates. It also works well with any of the external repositories. I've used "eselect repository create" to create the repos. An entry in repos.conf looks like this: [custom-overlay] location = /var/db/repos/custom-overlay Those overlays are lacking "sync-type" and "sync-uri". Reproducible: Always
q looks at the md5-metadata cache, if there is one, can you look if repos that show "ghost files" have a metadata/md5-cache directory populated?
Thanks allot for you answer. Deleting the cache actually helps. But somehow it's not the best way to deal with it as q does not know about any version anymore after that. Not sure if this should be considered a bug then. Seems more like an issue with the state of the metadata cache not being up to date.
q does not check the timestamps of the source, versus the cache, so if the cache is present it blindly assumes it is up-to-date. Without the cache, q should be able to traverse the source tree. When you say no versions are returned does that mean q doesn't see any packages at all any more?
Somehow i am not able to recreate what it did when i wrote my last comment. It was showing me the full path to the package without the version. Like this: /home/r7l/overlays/custom-overlay/www-apps/dev-php/phpmd/phpmd.ebuild After working on some Ebuilds today, i had new cache files in md5-cache. I've just deleted it again and q isn't able to give me a single package from the entire overlay. Let me add that my overlays are symlinked to /var/db/repos in order to be functional within the scope of Portage. The actual files are in my home directory. This does work well with any package having a md5-cache file. Any other package is not existing for q. I'd guess the symlink is not the issue here. If a file is shown, it is even the correct location within my home directory. So q does work well once an up to date m5-cache file is around.
When you remove the md5-cache, do you remove the entire md5-cache directory, or only its contents? I'm asking because q treats the cache as absent only when the entire directory is missing.
Ok i see. So when only deleting the files within the directory, it will no longer show any packages. When deleting the directory itself, it shows the packages but some of them will not have a version, like i've said in my last comment. I've tested almost the entire list of packages (~20) and it showed 5 of them with versions.
Can you show me one or two unversioned examples? And also share an ls of the ebuild directory for those?
Created attachment 855978 [details] bookstack Ebuild Ebuild that will not be seen with version from qwhich.
Created attachment 855980 [details] discord-webhook Ebuild Another Ebuild that will not be shown with a version when using qwhich.
When running "ls" in dev-python/discord-webhook: Manifest discord-webhook-1.1.0.ebuild metadata.xml When running "ls" in www-apps/bookstack: Manifest bookstack-22.09.ebuild files metadata.xml The "files" directory only contains a patch file. But i don't think it's related here. Another thing i've noted is that once you work on a single Ebuild, it will create a cache file for it. At least when creating a manifest. This renders any other package unknown again. But that's not surprising as "md5-cache" exists without any files for the rest of the packages except the one just edited.
(In reply to r7l from comment #10) > When running "ls" in dev-python/discord-webhook: > > Manifest > discord-webhook-1.1.0.ebuild > metadata.xml > > When running "ls" in www-apps/bookstack: > > Manifest > bookstack-22.09.ebuild > files > metadata.xml > > The "files" directory only contains a patch file. But i don't think it's > related here. I tried this, but I get back bookstack-22.09.ebuild from qwhich bookstack. > Another thing i've noted is that once you work on a single Ebuild, it will > create a cache file for it. At least when creating a manifest. This renders > any other package unknown again. But that's not surprising as "md5-cache" > exists without any files for the rest of the packages except the one just > edited. This indeed is quite a problem for repos in flux. The code currently doesn't cater for that. I won't promise anything, but will see if it's easy to implement, and check what it does for performance.
assigning this portage-utils bug to me
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=eb6d9110d09080e63944da0c1b0265156ab1e326 commit eb6d9110d09080e63944da0c1b0265156ab1e326 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2025-04-21 14:38:28 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2025-04-21 14:38:28 +0000 libq/tree: consider ebuild-tree cache as a cache, not a tree While it is certainly faster to only consider a metadata cache (such as md5-cache) it yields incorrect results on trees in flux or under development. The only correct way is to consider the ebuilds, and look at an existing cache, if it matches. This commit implements this behaviour for libq/tree, used by various q utilities. It considers an cache for an ebuild tree, and uses it for operations where it needs info from inside the ebuilds (such as DEPEND, etc.) over trying to fetch it from the ebuild itself. This is considerably slower than just considering the metadata alone, if only because we now have to traverse multiple extra directories (one for each package) as opposed to just reading the metadata. However, this is unavoidable, and hopefully can be mitigated by parallel scans in the future. Bug: https://bugs.gentoo.org/934549 Bug: https://bugs.gentoo.org/898194 Signed-off-by: Fabian Groffen <grobian@gentoo.org> libq/tree.c | 350 +++++++++++++++++++++++++++++++++++++----------------------- libq/tree.h | 1 + 2 files changed, 219 insertions(+), 132 deletions(-)
