Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 725934

Summary: emerge --complete-graph misses packages
Product: Portage Development Reporter: Joakim Tjernlund <joakim.tjernlund>
Component: UnclassifiedAssignee: Portage team <dev-portage>
Status: UNCONFIRMED ---    
Severity: normal    
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Joakim Tjernlund 2020-05-28 23:27:23 UTC
Example:
EMERGE_DEFAULT_OPTS= emerge --complete-graph --with-bdeps=y -pu world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  NS   ~] sys-kernel/gentoo-sources-5.4.43 [4.19.98, 5.4.35, 5.4.36, 5.4.37, 5.4.38, 5.4.39, 5.4.40, 5.4.41, 5.4.42]
[ebuild     U ~] sys-fs/bindfs-1.14.7 [1.14.5]
[ebuild     U ~] app-portage/mgorny-dev-scripts-5 [4]

Adding --deep gives:

EMERGE_DEFAULT_OPTS= emerge --complete-graph --with-bdeps=y --deep -pu world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] app-text/libnumbertext-1.0.5-r1 [1.0.5]
[ebuild  NS   ~] sys-kernel/gentoo-sources-5.4.43 [4.19.98, 5.4.35, 5.4.36, 5.4.37, 5.4.38, 5.4.39, 5.4.40, 5.4.41, 5.4.42]
[ebuild  N     ] virtual/ruby-ssl-11  RUBY_TARGETS="ruby24 ruby25 (-ruby26) (-ruby27)" 
[ebuild     U  ] x11-misc/shared-mime-info-1.15 [1.10-r1]
[ebuild     U  ] app-misc/pax-utils-1.2.6 [1.2.5]
[ebuild     U  ] dev-python/cffi-1.14.0-r1 [1.14.0] PYTHON_TARGETS="(-python3_9)"
[ebuild     U ~] sys-fs/bindfs-1.14.7 [1.14.5]
[ebuild     U  ] net-mail/mailutils-3.9 [3.8] PYTHON_SINGLE_TARGET="(-python3_8)" 
[ebuild     U ~] app-portage/mgorny-dev-scripts-5 [4]
Comment 1 Zac Medico gentoo-dev 2020-06-01 17:51:35 UTC
The purpose of --complete-graph is not to pull in updates. The purpose of --complete-graph is to account for the dependencies of all relevant packages. If you want to pull in all available updates then you need to use --deep.
Comment 2 Joakim Tjernlund 2020-06-02 16:39:39 UTC
(In reply to Zac Medico from comment #1)
> The purpose of --complete-graph is not to pull in updates. The purpose of
> --complete-graph is to account for the dependencies of all relevant
> packages. If you want to pull in all available updates then you need to use
> --deep.

Oh, I was looking for something cheaper that --deep to pull in updates as
it can take minutes to complete its analysis.
An option that check all installed packages without scanning the whole
gentoo(and overlays) ebuild database.
Comment 3 Zac Medico gentoo-dev 2020-06-02 16:57:37 UTC
There's no other option that does what --deep does.
Comment 4 Joakim Tjernlund 2020-06-02 16:58:59 UTC
(In reply to Zac Medico from comment #3)
> There's no other option that does what --deep does.

OK, can --deep be impl. a bit more efficient then?
Comment 5 Zac Medico gentoo-dev 2020-06-02 17:02:34 UTC
(In reply to Joakim Tjernlund from comment #4)
> (In reply to Zac Medico from comment #3)
> > There's no other option that does what --deep does.
> 
> OK, can --deep be impl. a bit more efficient then?

We'd have to profile it to see where we could improve efficiency. Also, we could parallelize things as discussed in bug 660860.
Comment 6 Joakim Tjernlund 2020-06-02 17:09:46 UTC
(In reply to Zac Medico from comment #5)
> (In reply to Joakim Tjernlund from comment #4)
> > (In reply to Zac Medico from comment #3)
> > > There's no other option that does what --deep does.
> > 
> > OK, can --deep be impl. a bit more efficient then?
> 
> We'd have to profile it to see where we could improve efficiency. Also, we
> could parallelize things as discussed in bug 660860.

One obvious optimization would be not to stat the whole Gentoo ebuild repo.
aka, start with all installed pkgs, add any additions from  profile and sets.
Then scan those pkgs.
Comment 7 Zac Medico gentoo-dev 2020-06-02 17:20:47 UTC
We use listdir instead of stat, since https://github.com/gentoo/portage/pull/219.
Comment 8 Zac Medico gentoo-dev 2020-06-02 17:25:10 UTC
Also https://github.com/gentoo/portage/pull/218, see bug 634210.
Comment 9 Zac Medico gentoo-dev 2020-06-02 17:28:35 UTC
Actually, it's the patch for bug 650814 that eliminates the stat calls via cp_list cache.
Comment 10 Joakim Tjernlund 2020-06-04 18:02:00 UTC
stracing an emerge -pDuv world:
   strace -f -p 9429 |& grep /usr/portage/ > emerge.log
has stats like these:
stat("/usr/portage/dev-lisp/roswell", {st_mode=S_IFDIR|0755, st_size=182, ...}) = 0
stat("/usr/portage/dev-lisp/c2ffi", {st_mode=S_IFDIR|0755, st_size=121, ...}) = 0
stat("/usr/portage/dev-lisp/alexandria", {st_mode=S_IFDIR|0755, st_size=89, ...}) = 0
stat("/usr/portage/dev-lisp/cl-unicode", {st_mode=S_IFDIR|0755, st_size=73, ...}) = 0
stat("/usr/portage/dev-lisp/cl-ppcre-unicode", {st_mode=S_IFDIR|0755, st_size=117, ...}) = 0
stat("/usr/portage/dev-lisp/trivial-gray-streams", {st_mode=S_IFDIR|0755, st_size=130, ...}) = 0
stat("/usr/portage/dev-lisp/flexi-streams", {st_mode=S_IFDIR|0755, st_size=112, ...}) = 0
stat("/usr/portage/dev-lisp/abcl", {st_mode=S_IFDIR|0755, st_size=67, ...}) = 0
stat("/usr/portage/dev-lisp/cl-ppcre", {st_mode=S_IFDIR|0755, st_size=101, ...}) = 0
stat("/usr/portage/dev-lisp/clisp", {st_mode=S_IFDIR|0755, st_size=139, ...}) = 0
stat("/usr/portage/dev-lisp/clozurecl", {st_mode=S_IFDIR|0755, st_size=86, ...}) = 0
stat("/usr/portage/dev-lisp/cmucl", {st_mode=S_IFDIR|0755, st_size=79, ...}) = 0
stat("/usr/portage/dev-lisp/ecls", {st_mode=S_IFDIR|0755, st_size=81, ...}) = 0
stat("/usr/portage/dev-lisp/gcl", {st_mode=S_IFDIR|0755, st_size=105, ...}) = 0
stat("/usr/portage/dev-lisp/hyperspec", {st_mode=S_IFDIR|0755, st_size=73, ...}) = 0
stat("/usr/portage/dev-lisp/sbcl", {st_mode=S_IFDIR|0755, st_size=281, ...}) = 0
stat("/usr/portage/dev-lisp/asdf", {st_mode=S_IFDIR|0755, st_size=158, ...}) = 0
stat("/usr/portage/dev-lisp/uiop", {st_mode=S_IFDIR|0755, st_size=117, ...}) = 0
stat("/usr/portage/dev-lisp/clx", {st_mode=S_IFDIR|0755, st_size=66, ...}) = 0

I don't have any dev-lisp packages installed, why then try to access these at all?
Comment 11 Zac Medico gentoo-dev 2020-06-05 04:44:34 UTC
We should be able to eliminate those stat calls which come from the _scan_cat method, introduced here:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c5a2a0edc4f4b01b16a274268431fa21f7f678b2
Comment 12 Larry the Git Cow gentoo-dev 2020-06-07 03:26:50 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=976c1133100f6da81cd8d6e13f8a723a9fc7cd85

commit 976c1133100f6da81cd8d6e13f8a723a9fc7cd85
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-06-07 02:25:11 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-06-07 02:53:27 +0000

    _better_cache._scan_cat: avoid stat calls (bug 725934)
    
    When processing category listdir results, do not use os.path.isdir to
    identify packages, in order to avoid unecessary stat calls. Instead,
    use the Atom class to validate package names. This may cause creation
    of some cache entries for non-packages, but it will not do any harm
    since these entries will never be accessed via the __getitem__ method.
    
    Bug: https://bugs.gentoo.org/725934
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/dbapi/porttree.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Comment 13 Joakim Tjernlund 2020-06-12 12:13:48 UTC
Added the patch to my portage, still takes alot of time
Managed to get BT when I was seeing alot of:
recvfrom(4, 0x55b010578f80, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

time emerge -pDuv world
--Return--
> /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None
-> pdb.set_trace()
(Pdb) bt
  /usr/lib/python-exec/python3.6/emerge(53)<module>()
-> retval = emerge_main()
  /usr/lib64/python3.6/site-packages/_emerge/main.py(1309)emerge_main()
-> return run_action(emerge_config)
  /usr/lib64/python3.6/site-packages/_emerge/actions.py(3378)run_action()
-> retval = action_build(emerge_config, spinner=spinner)
  /usr/lib64/python3.6/site-packages/_emerge/actions.py(359)action_build()
-> settings, trees, myopts, myparams, myaction, myfiles, spinner)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9922)backtrack_depgraph()
-> myaction, myfiles, spinner)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9959)_backtrack_depgraph()
-> success, favorites = mydepgraph.select_files(myfiles)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4008)select_files()
-> return self._select_files(args)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4351)_select_files()
-> return self._resolve(myfavorites)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4507)_resolve()
-> self.altlist()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7495)altlist()
-> self._resolve_conflicts()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7622)_resolve_conflicts()
-> self._process_slot_conflicts()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(1726)_process_slot_conflicts()
-> self._slot_operator_trigger_reinstalls()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2633)_slot_operator_trigger_reinstalls()
-> new_child_slot=True)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2146)_slot_operator_update_probe()
-> if not check_reverse_dependencies(dep.parent, replacement_parent):
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2104)check_reverse_dependencies()
-> if self._upgrade_available(parent):
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2455)_upgrade_available()
-> pkg.slot_atom):
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2541)_iter_similar_available()
-> graph_pkg.root_config, atom):
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(5659)_iter_match_pkgs_any()
-> pkg_type, atom, onlydeps=onlydeps):
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(5711)_iter_match_pkgs_atom()
-> myrepo=getattr(cpv, 'repo', None))
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7061)_pkg()
-> installed=installed, onlydeps=onlydeps))
  /usr/lib64/python3.6/site-packages/_emerge/Package.py(225)_gen_hash_key()
-> elements = [type_name, root, _unicode(cpv), operation]
> /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None
-> pdb.set_trace()

No idea if that helps though
Comment 14 Joakim Tjernlund 2020-06-12 12:15:50 UTC
One more BT:

(Pdb) cont
--Return--
> /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None
-> pdb.set_trace()
(Pdb) bt
  /usr/lib/python-exec/python3.6/emerge(53)<module>()
-> retval = emerge_main()
  /usr/lib64/python3.6/site-packages/_emerge/main.py(1309)emerge_main()
-> return run_action(emerge_config)
  /usr/lib64/python3.6/site-packages/_emerge/actions.py(3378)run_action()
-> retval = action_build(emerge_config, spinner=spinner)
  /usr/lib64/python3.6/site-packages/_emerge/actions.py(359)action_build()
-> settings, trees, myopts, myparams, myaction, myfiles, spinner)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9922)backtrack_depgraph()
-> myaction, myfiles, spinner)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9959)_backtrack_depgraph()
-> success, favorites = mydepgraph.select_files(myfiles)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4008)select_files()
-> return self._select_files(args)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4351)_select_files()
-> return self._resolve(myfavorites)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4507)_resolve()
-> self.altlist()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7495)altlist()
-> self._resolve_conflicts()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7622)_resolve_conflicts()
-> self._process_slot_conflicts()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(1726)_process_slot_conflicts()
-> self._slot_operator_trigger_reinstalls()
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2639)_slot_operator_trigger_reinstalls()
-> if self._slot_operator_update_probe(dep):
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2269)_slot_operator_update_probe()
-> dep.child.root, replacement_parent)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2484)_select_atoms_probe()
-> root, v, myuse=use, parent=pkg)[pkg])
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4780)_select_atoms_from_graph()
-> return self._select_atoms_highest_available(*pargs, **kwargs)
  /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4796)_select_atoms_highest_available()
-> is_valid_flag=is_valid_flag, eapi=eapi)
  /usr/lib64/python3.6/site-packages/portage/dep/__init__.py(739)use_reduce()
-> is_valid_flag=is_valid_flag)
  /usr/lib64/python3.6/site-packages/portage/dep/__init__.py(1413)__init__()
-> allow_repo=allow_repo)
  /usr/lib64/python3.6/site-packages/portage/dep/__init__.py(1363)__init__()
-> self.__dict__['cpv'] = _pkg_str(cpv)
  /usr/lib64/python3.6/site-packages/portage/versions.py(397)__init__()
-> if self.cpv_split is None:
> /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None
-> pdb.set_trace()
(Pdb)
Comment 15 Joakim Tjernlund 2020-06-12 12:20:00 UTC
I think emerging a single pkg also take a bit long to analyse, example:
time emerge -p1 dev-libs/nss
[binary     U ] dev-libs/nss-3.52.1-r1 [3.52.1]

real	0m13.738s
user	0m13.561s
sys	0m0.153s

Time is on fast DE, i7-6700 CPU @ 3.40GHz, using a nvme disk
Comment 16 Larry the Git Cow gentoo-dev 2020-06-14 23:47:26 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5330fc62703c0e2c535eb2ac8247d43faf501c8e

commit 5330fc62703c0e2c535eb2ac8247d43faf501c8e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-06-14 23:33:47 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-06-14 23:47:09 +0000

    sys-apps/portage: Bump to version 2.3.101
    
     #661518 repos.conf: Add bool sync-openpgp-key-refresh option
     #709746 New PORTAGE_LOG_FILTER_FILE_CMD variable specifies a
             command that filters build log output to a log file
     #719810 Escape percent-signs in mirror url
     #725934 _better_cache._scan_cat: Avoid stat calls
     #728046 ecompress: Prefix eqawarn messages with QA Notice
    
    Bug: https://bugs.gentoo.org/721152
    Bug: https://bugs.gentoo.org/661518
    Bug: https://bugs.gentoo.org/709746
    Bug: https://bugs.gentoo.org/719810
    Bug: https://bugs.gentoo.org/725934
    Bug: https://bugs.gentoo.org/728046
    Package-Manager: Portage-2.3.101, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest               |   1 +
 sys-apps/portage/portage-2.3.101.ebuild | 263 ++++++++++++++++++++++++++++++++
 2 files changed, 264 insertions(+)