Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 628764 - sys-apps/portage: "emerge -pvuND world" consumes excessive amounts of CPU
Summary: sys-apps/portage: "emerge -pvuND world" consumes excessive amounts of CPU
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Dependencies (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-08-23 19:15 UTC by Andreas Steinmetz
Modified: 2023-12-31 15:56 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
/var/lib/portage/world (world,21.36 KB, text/plain)
2017-08-23 21:09 UTC, Andreas Steinmetz
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Steinmetz 2017-08-23 19:15:01 UTC
"emerge -pvuND world" consumes more than 10 CPU *minutes* on an x86_64 system with 2906 packages installed. To add insult to injury it is running strictly single core.
Fixing dependency problems that require multiple runs can thus take hours.
By all means emerge tends to become totally unusable due to the package analysis processing time required.
And even as the system is not the most modern one I would expect emerge to run on a system with a i7-3930K and 64GB memory reasonably fast.
Comment 1 Brian Evans (RETIRED) gentoo-dev 2017-08-23 20:46:33 UTC
It would be interesting to count how many lines are in your /var/lib/portage/world file.  The more references there, the longer the calculations take.
Comment 2 Andreas Steinmetz 2017-08-23 20:52:57 UTC
# wc -l /var/lib/portage/world
1013 /var/lib/portage/world
Comment 3 Brian Evans (RETIRED) gentoo-dev 2017-08-23 21:02:18 UTC
(In reply to Andreas Steinmetz from comment #2)
> # wc -l /var/lib/portage/world
> 1013 /var/lib/portage/world

In comparison, my desktop, developer system has this:

$ wc -l /var/lib/portage/world
287 /var/lib/portage/world
Comment 4 Andreas Steinmetz 2017-08-23 21:09:49 UTC
Created attachment 490356 [details]
/var/lib/portage/world

Have a look. I can't see something really excessive.

And anyway emerge uses 0.6 CPU seconds/entry on a reasonably fast system with respect to the amount of entries in /var/lib/portage/world which seems to me a quite a bit too much.
Comment 5 Brian Evans (RETIRED) gentoo-dev 2017-08-23 21:48:26 UTC
(In reply to Andreas Steinmetz from comment #4)
> Created attachment 490356 [details]
> /var/lib/portage/world
> 
> Have a look. I can't see something really excessive.
> 
> And anyway emerge uses 0.6 CPU seconds/entry on a reasonably fast system
> with respect to the amount of entries in /var/lib/portage/world which seems
> to me a quite a bit too much.

There is a lot of leaf nodes in that list which really should be dependencies (in other words pulled in when needed and not listed).

Just about everything in sys-devel/* dev-libs/* dev-python/* dev-perl/* x11-drivers/* and x11-libs/* should be able to be removed unless you specifically are developing new software using them. (Not using ebuilds.)  Possibly some of dev-java/* too.

The rest could be reviewed for what you absolutely need.
Comment 6 Andrei Slavoiu 2017-08-23 22:41:18 UTC
That doesn't sound right. Why would portage run time depend on number of entries in the world file and not on the number of packages in the entire tree? Shouldn't portage just treat as noop entries in the world file that were already resolved as dependencies of another package?

BTW, my world file contains 35 entries and my custom sets another 66 entries. It takes portage several minutes (usualy under 5) to build the dependency tree on my Kaveri APU. This is the output of emerge -c:
Packages installed:   1403
Packages in world:    100
Packages in system:   44
Required packages:    1403
Number removed:       0
Comment 7 Andreas Steinmetz 2017-08-23 22:59:48 UTC
The world file in my case contains

1. quite some special stuff I do in fact need (e.g. audio, java and quite some
   more)

2. for sure some cruft that piled up there in the 5 to 10 years (can't recall)
   of existence of that installation but which is problematic to reverse
   resolve manually (too error prone and time consuming)

Simply put, a "world cleaner utility" that would scan the world file for dependencies and required package use flags would be a welcome enhancement for all long term installations but that's another agenda.

The actual problem is, why emerge requires so many cycles of a 3.20GHz core for dependency tracking. A quick and dirty calculation gives about 2.000.000.000 cycles per world entry which I assume to be, well, a little too much.