If the same cat/pn exists in multiple repositories, the pordbapi findname2 method searchs all repos with that cat/pn. We could make cp_list record the repo that the ebuild exists in and then findname2 could use that information to avoid unnecessary stat alls. This would be similar to the binpkg-multi-instance stuff, but in portdbapi instead of bindbapi. Since the same ebuild version might exist in multiple repositories, it might be tracking multiple instances of the same ebuild version in case it has to prefer one over the other for some reason (missing IUSE etc).
is it any news about that feature?
Created attachment 539450 [details, diff] portdbapi.findname2: avoid os.access call This simple optimization might help.
Created attachment 539452 [details, diff] portdbapi.findname2: avoid os.access call
I'm working on a patch that allows portdbapi.cp_list to return instances having a 'repo' attribute, so that most findname2 calls have not need to call os.access: https://github.com/zmedico/portage/tree/bug_650814_pordbapi_findname2_skip_access The patch is mostly complete, the only thing missing is support for caching cp_list results when the new behavior is enabled.
Comment on attachment 539452 [details, diff] portdbapi.findname2: avoid os.access call Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/49545f80246136e4c468b053e04f03fe https://github.com/gentoo/portage/pull/339
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=27eeeb6e4fc8e68efa003b2db5bbd8cc4afe336a commit 27eeeb6e4fc8e68efa003b2db5bbd8cc4afe336a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-07-13 22:19:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-07-15 05:10:51 +0000 portdbapi.cp_list: cache repo associations (bug 650814) Make portdbapi.cp_list return _pkg_str instances that have a 'repo' attribute (bindbapi.cp_list already does this), with results in ascending order by (pkg.version, repo.priority). Optimize portdbapi.findname2 to use the 'repo' attribute to enable cached results for files previously found by the portdbapi.cp_list method, avoiding filesystem access when possible. Optimize the depgraph._iter_match_pkgs_atom method by elimination of the repo loop, since portdbapi.cp_list now returns separate items when the same package version is found in multiple repos. Bug: https://bugs.gentoo.org/650814 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> pym/_emerge/depgraph.py | 12 +++--------- pym/portage/dbapi/porttree.py | 41 +++++++++++++++++++++++++++++------------ 2 files changed, 32 insertions(+), 21 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=31edfe62541d8b55c64584b056ce64ad759f6a8e commit 31edfe62541d8b55c64584b056ce64ad759f6a8e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-07-15 21:38:30 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-07-15 21:38:30 +0000 portdbapi.xmatch: return separate items for each repo Since xmatch match-all shares cache with the cp_list method, it needs to return separate items when the same package version is found in multiple repos. Fixes: 27eeeb6e4fc8 ("portdbapi.cp_list: cache repo associations (bug 650814)") Bug: https://bugs.gentoo.org/650814 pym/portage/dbapi/porttree.py | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=84431fe188ed1b3e2e7b52918329bd696d8304b1 commit 84431fe188ed1b3e2e7b52918329bd696d8304b1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-07-16 05:07:10 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-07-16 05:18:51 +0000 portdbapi.cp_list: honor porttrees modifications (bug 650814) Consumers, such as repoman and emirrordist, may modify the porttrees attribute in order to modify the effective set of repositories for all portdbapi operations. Fixes: 27eeeb6e4fc8 ("portdbapi.cp_list: cache repo associations (bug 650814)") Bug: https://bugs.gentoo.org/650814 pym/portage/dbapi/porttree.py | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)
