Bug 650814 – sys-apps/portage: portdbapi could avoid stat calls in findname2 method

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Zac Medico 2018-03-18 18:45:21 UTC]

If the same cat/pn exists in multiple repositories, the pordbapi findname2 method searchs all repos with that cat/pn. We could make cp_list record the repo that the ebuild exists in and then findname2 could use that information to avoid unnecessary stat alls. This would be similar to the binpkg-multi-instance stuff, but in portdbapi instead of bindbapi. Since the same ebuild version might exist in multiple repositories, it might be tracking multiple instances of the same ebuild version in case it has to prefer one over the other for some reason (missing IUSE etc).

[Comment 1 Vadim A. Misbakh-Soloviov (mva) (RETIRED) 2018-07-12 12:55:45 UTC]

is it any news about that feature?

[Comment 2 Zac Medico 2018-07-13 22:30:21 UTC]

Created attachment 539450 [details, diff]
portdbapi.findname2: avoid os.access call

This simple optimization might help.

[Comment 3 Zac Medico 2018-07-13 22:45:45 UTC]

Created attachment 539452 [details, diff]
portdbapi.findname2: avoid os.access call

[Comment 4 Zac Medico 2018-07-14 09:28:36 UTC]

I'm working on a patch that allows portdbapi.cp_list to return instances having a 'repo' attribute, so that most findname2 calls have not need to call os.access:

https://github.com/zmedico/portage/tree/bug_650814_pordbapi_findname2_skip_access

The patch is mostly complete, the only thing missing is support for caching cp_list results when the new behavior is enabled.

[Comment 5 Zac Medico 2018-07-15 04:11:30 UTC]

Comment on attachment 539452 [details, diff]
portdbapi.findname2: avoid os.access call

Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/49545f80246136e4c468b053e04f03fe
https://github.com/gentoo/portage/pull/339

[Comment 6 Larry the Git Cow 2018-07-15 21:18:24 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=27eeeb6e4fc8e68efa003b2db5bbd8cc4afe336a

commit 27eeeb6e4fc8e68efa003b2db5bbd8cc4afe336a
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-07-13 22:19:29 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-07-15 05:10:51 +0000

    portdbapi.cp_list: cache repo associations (bug 650814)
    
    Make portdbapi.cp_list return _pkg_str instances that have a 'repo'
    attribute (bindbapi.cp_list already does this), with results
    in ascending order by (pkg.version, repo.priority). Optimize
    portdbapi.findname2 to use the 'repo' attribute to enable cached
    results for files previously found by the portdbapi.cp_list
    method, avoiding filesystem access when possible. Optimize the
    depgraph._iter_match_pkgs_atom method by elimination of the repo
    loop, since portdbapi.cp_list now returns separate items when the
    same package version is found in multiple repos.
    
    Bug: https://bugs.gentoo.org/650814
    Reviewed-by: Brian Dolbec <[e-mail]>

 pym/_emerge/depgraph.py       | 12 +++---------
 pym/portage/dbapi/porttree.py | 41 +++++++++++++++++++++++++++++------------
 2 files changed, 32 insertions(+), 21 deletions(-)

[Comment 7 Larry the Git Cow 2018-07-15 21:40:48 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=31edfe62541d8b55c64584b056ce64ad759f6a8e

commit 31edfe62541d8b55c64584b056ce64ad759f6a8e
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-07-15 21:38:30 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-07-15 21:38:30 +0000

    portdbapi.xmatch: return separate items for each repo
    
    Since xmatch match-all shares cache with the cp_list method, it needs
    to return separate items when the same package version is found in
    multiple repos.
    
    Fixes: 27eeeb6e4fc8 ("portdbapi.cp_list: cache repo associations (bug 650814)")
    Bug: https://bugs.gentoo.org/650814

 pym/portage/dbapi/porttree.py | 19 +++----------------
 1 file changed, 3 insertions(+), 16 deletions(-)

[Comment 8 Larry the Git Cow 2018-07-16 05:27:51 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=84431fe188ed1b3e2e7b52918329bd696d8304b1

commit 84431fe188ed1b3e2e7b52918329bd696d8304b1
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-07-16 05:07:10 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-07-16 05:18:51 +0000

    portdbapi.cp_list: honor porttrees modifications (bug 650814)
    
    Consumers, such as repoman and emirrordist, may modify the porttrees
    attribute in order to modify the effective set of repositories for all
    portdbapi operations.
    
    Fixes: 27eeeb6e4fc8 ("portdbapi.cp_list: cache repo associations (bug 650814)")
    Bug: https://bugs.gentoo.org/650814

 pym/portage/dbapi/porttree.py | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)