Example: EMERGE_DEFAULT_OPTS= emerge --complete-graph --with-bdeps=y -pu world These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild NS ~] sys-kernel/gentoo-sources-5.4.43 [4.19.98, 5.4.35, 5.4.36, 5.4.37, 5.4.38, 5.4.39, 5.4.40, 5.4.41, 5.4.42] [ebuild U ~] sys-fs/bindfs-1.14.7 [1.14.5] [ebuild U ~] app-portage/mgorny-dev-scripts-5 [4] Adding --deep gives: EMERGE_DEFAULT_OPTS= emerge --complete-graph --with-bdeps=y --deep -pu world These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] app-text/libnumbertext-1.0.5-r1 [1.0.5] [ebuild NS ~] sys-kernel/gentoo-sources-5.4.43 [4.19.98, 5.4.35, 5.4.36, 5.4.37, 5.4.38, 5.4.39, 5.4.40, 5.4.41, 5.4.42] [ebuild N ] virtual/ruby-ssl-11 RUBY_TARGETS="ruby24 ruby25 (-ruby26) (-ruby27)" [ebuild U ] x11-misc/shared-mime-info-1.15 [1.10-r1] [ebuild U ] app-misc/pax-utils-1.2.6 [1.2.5] [ebuild U ] dev-python/cffi-1.14.0-r1 [1.14.0] PYTHON_TARGETS="(-python3_9)" [ebuild U ~] sys-fs/bindfs-1.14.7 [1.14.5] [ebuild U ] net-mail/mailutils-3.9 [3.8] PYTHON_SINGLE_TARGET="(-python3_8)" [ebuild U ~] app-portage/mgorny-dev-scripts-5 [4]
The purpose of --complete-graph is not to pull in updates. The purpose of --complete-graph is to account for the dependencies of all relevant packages. If you want to pull in all available updates then you need to use --deep.
(In reply to Zac Medico from comment #1) > The purpose of --complete-graph is not to pull in updates. The purpose of > --complete-graph is to account for the dependencies of all relevant > packages. If you want to pull in all available updates then you need to use > --deep. Oh, I was looking for something cheaper that --deep to pull in updates as it can take minutes to complete its analysis. An option that check all installed packages without scanning the whole gentoo(and overlays) ebuild database.
There's no other option that does what --deep does.
(In reply to Zac Medico from comment #3) > There's no other option that does what --deep does. OK, can --deep be impl. a bit more efficient then?
(In reply to Joakim Tjernlund from comment #4) > (In reply to Zac Medico from comment #3) > > There's no other option that does what --deep does. > > OK, can --deep be impl. a bit more efficient then? We'd have to profile it to see where we could improve efficiency. Also, we could parallelize things as discussed in bug 660860.
(In reply to Zac Medico from comment #5) > (In reply to Joakim Tjernlund from comment #4) > > (In reply to Zac Medico from comment #3) > > > There's no other option that does what --deep does. > > > > OK, can --deep be impl. a bit more efficient then? > > We'd have to profile it to see where we could improve efficiency. Also, we > could parallelize things as discussed in bug 660860. One obvious optimization would be not to stat the whole Gentoo ebuild repo. aka, start with all installed pkgs, add any additions from profile and sets. Then scan those pkgs.
We use listdir instead of stat, since https://github.com/gentoo/portage/pull/219.
Also https://github.com/gentoo/portage/pull/218, see bug 634210.
Actually, it's the patch for bug 650814 that eliminates the stat calls via cp_list cache.
stracing an emerge -pDuv world: strace -f -p 9429 |& grep /usr/portage/ > emerge.log has stats like these: stat("/usr/portage/dev-lisp/roswell", {st_mode=S_IFDIR|0755, st_size=182, ...}) = 0 stat("/usr/portage/dev-lisp/c2ffi", {st_mode=S_IFDIR|0755, st_size=121, ...}) = 0 stat("/usr/portage/dev-lisp/alexandria", {st_mode=S_IFDIR|0755, st_size=89, ...}) = 0 stat("/usr/portage/dev-lisp/cl-unicode", {st_mode=S_IFDIR|0755, st_size=73, ...}) = 0 stat("/usr/portage/dev-lisp/cl-ppcre-unicode", {st_mode=S_IFDIR|0755, st_size=117, ...}) = 0 stat("/usr/portage/dev-lisp/trivial-gray-streams", {st_mode=S_IFDIR|0755, st_size=130, ...}) = 0 stat("/usr/portage/dev-lisp/flexi-streams", {st_mode=S_IFDIR|0755, st_size=112, ...}) = 0 stat("/usr/portage/dev-lisp/abcl", {st_mode=S_IFDIR|0755, st_size=67, ...}) = 0 stat("/usr/portage/dev-lisp/cl-ppcre", {st_mode=S_IFDIR|0755, st_size=101, ...}) = 0 stat("/usr/portage/dev-lisp/clisp", {st_mode=S_IFDIR|0755, st_size=139, ...}) = 0 stat("/usr/portage/dev-lisp/clozurecl", {st_mode=S_IFDIR|0755, st_size=86, ...}) = 0 stat("/usr/portage/dev-lisp/cmucl", {st_mode=S_IFDIR|0755, st_size=79, ...}) = 0 stat("/usr/portage/dev-lisp/ecls", {st_mode=S_IFDIR|0755, st_size=81, ...}) = 0 stat("/usr/portage/dev-lisp/gcl", {st_mode=S_IFDIR|0755, st_size=105, ...}) = 0 stat("/usr/portage/dev-lisp/hyperspec", {st_mode=S_IFDIR|0755, st_size=73, ...}) = 0 stat("/usr/portage/dev-lisp/sbcl", {st_mode=S_IFDIR|0755, st_size=281, ...}) = 0 stat("/usr/portage/dev-lisp/asdf", {st_mode=S_IFDIR|0755, st_size=158, ...}) = 0 stat("/usr/portage/dev-lisp/uiop", {st_mode=S_IFDIR|0755, st_size=117, ...}) = 0 stat("/usr/portage/dev-lisp/clx", {st_mode=S_IFDIR|0755, st_size=66, ...}) = 0 I don't have any dev-lisp packages installed, why then try to access these at all?
We should be able to eliminate those stat calls which come from the _scan_cat method, introduced here: https://gitweb.gentoo.org/proj/portage.git/commit/?id=c5a2a0edc4f4b01b16a274268431fa21f7f678b2
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=976c1133100f6da81cd8d6e13f8a723a9fc7cd85 commit 976c1133100f6da81cd8d6e13f8a723a9fc7cd85 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-07 02:25:11 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-07 02:53:27 +0000 _better_cache._scan_cat: avoid stat calls (bug 725934) When processing category listdir results, do not use os.path.isdir to identify packages, in order to avoid unecessary stat calls. Instead, use the Atom class to validate package names. This may cause creation of some cache entries for non-packages, but it will not do any harm since these entries will never be accessed via the __getitem__ method. Bug: https://bugs.gentoo.org/725934 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/dbapi/porttree.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
Added the patch to my portage, still takes alot of time Managed to get BT when I was seeing alot of: recvfrom(4, 0x55b010578f80, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) time emerge -pDuv world --Return-- > /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None -> pdb.set_trace() (Pdb) bt /usr/lib/python-exec/python3.6/emerge(53)<module>() -> retval = emerge_main() /usr/lib64/python3.6/site-packages/_emerge/main.py(1309)emerge_main() -> return run_action(emerge_config) /usr/lib64/python3.6/site-packages/_emerge/actions.py(3378)run_action() -> retval = action_build(emerge_config, spinner=spinner) /usr/lib64/python3.6/site-packages/_emerge/actions.py(359)action_build() -> settings, trees, myopts, myparams, myaction, myfiles, spinner) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9922)backtrack_depgraph() -> myaction, myfiles, spinner) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9959)_backtrack_depgraph() -> success, favorites = mydepgraph.select_files(myfiles) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4008)select_files() -> return self._select_files(args) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4351)_select_files() -> return self._resolve(myfavorites) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4507)_resolve() -> self.altlist() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7495)altlist() -> self._resolve_conflicts() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7622)_resolve_conflicts() -> self._process_slot_conflicts() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(1726)_process_slot_conflicts() -> self._slot_operator_trigger_reinstalls() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2633)_slot_operator_trigger_reinstalls() -> new_child_slot=True) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2146)_slot_operator_update_probe() -> if not check_reverse_dependencies(dep.parent, replacement_parent): /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2104)check_reverse_dependencies() -> if self._upgrade_available(parent): /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2455)_upgrade_available() -> pkg.slot_atom): /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2541)_iter_similar_available() -> graph_pkg.root_config, atom): /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(5659)_iter_match_pkgs_any() -> pkg_type, atom, onlydeps=onlydeps): /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(5711)_iter_match_pkgs_atom() -> myrepo=getattr(cpv, 'repo', None)) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7061)_pkg() -> installed=installed, onlydeps=onlydeps)) /usr/lib64/python3.6/site-packages/_emerge/Package.py(225)_gen_hash_key() -> elements = [type_name, root, _unicode(cpv), operation] > /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None -> pdb.set_trace() No idea if that helps though
One more BT: (Pdb) cont --Return-- > /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None -> pdb.set_trace() (Pdb) bt /usr/lib/python-exec/python3.6/emerge(53)<module>() -> retval = emerge_main() /usr/lib64/python3.6/site-packages/_emerge/main.py(1309)emerge_main() -> return run_action(emerge_config) /usr/lib64/python3.6/site-packages/_emerge/actions.py(3378)run_action() -> retval = action_build(emerge_config, spinner=spinner) /usr/lib64/python3.6/site-packages/_emerge/actions.py(359)action_build() -> settings, trees, myopts, myparams, myaction, myfiles, spinner) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9922)backtrack_depgraph() -> myaction, myfiles, spinner) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(9959)_backtrack_depgraph() -> success, favorites = mydepgraph.select_files(myfiles) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4008)select_files() -> return self._select_files(args) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4351)_select_files() -> return self._resolve(myfavorites) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4507)_resolve() -> self.altlist() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7495)altlist() -> self._resolve_conflicts() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(7622)_resolve_conflicts() -> self._process_slot_conflicts() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(1726)_process_slot_conflicts() -> self._slot_operator_trigger_reinstalls() /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2639)_slot_operator_trigger_reinstalls() -> if self._slot_operator_update_probe(dep): /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2269)_slot_operator_update_probe() -> dep.child.root, replacement_parent) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(2484)_select_atoms_probe() -> root, v, myuse=use, parent=pkg)[pkg]) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4780)_select_atoms_from_graph() -> return self._select_atoms_highest_available(*pargs, **kwargs) /usr/lib64/python3.6/site-packages/_emerge/depgraph.py(4796)_select_atoms_highest_available() -> is_valid_flag=is_valid_flag, eapi=eapi) /usr/lib64/python3.6/site-packages/portage/dep/__init__.py(739)use_reduce() -> is_valid_flag=is_valid_flag) /usr/lib64/python3.6/site-packages/portage/dep/__init__.py(1413)__init__() -> allow_repo=allow_repo) /usr/lib64/python3.6/site-packages/portage/dep/__init__.py(1363)__init__() -> self.__dict__['cpv'] = _pkg_str(cpv) /usr/lib64/python3.6/site-packages/portage/versions.py(397)__init__() -> if self.cpv_split is None: > /usr/lib/python-exec/python3.6/emerge(30)debug_signal()->None -> pdb.set_trace() (Pdb)
I think emerging a single pkg also take a bit long to analyse, example: time emerge -p1 dev-libs/nss [binary U ] dev-libs/nss-3.52.1-r1 [3.52.1] real 0m13.738s user 0m13.561s sys 0m0.153s Time is on fast DE, i7-6700 CPU @ 3.40GHz, using a nvme disk
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5330fc62703c0e2c535eb2ac8247d43faf501c8e commit 5330fc62703c0e2c535eb2ac8247d43faf501c8e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-14 23:33:47 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-14 23:47:09 +0000 sys-apps/portage: Bump to version 2.3.101 #661518 repos.conf: Add bool sync-openpgp-key-refresh option #709746 New PORTAGE_LOG_FILTER_FILE_CMD variable specifies a command that filters build log output to a log file #719810 Escape percent-signs in mirror url #725934 _better_cache._scan_cat: Avoid stat calls #728046 ecompress: Prefix eqawarn messages with QA Notice Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/661518 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/719810 Bug: https://bugs.gentoo.org/725934 Bug: https://bugs.gentoo.org/728046 Package-Manager: Portage-2.3.101, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.101.ebuild | 263 ++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
