Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 546246 - emerge should have info about time needed to build packages
Summary: emerge should have info about time needed to build packages
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: Normal enhancement with 5 votes (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-11 08:30 UTC by chadwick.ferguson
Modified: 2018-03-24 21:40 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description chadwick.ferguson 2015-04-11 08:30:12 UTC
It would be nice to get a readout on a package by package basis or bulk operations added up of Standard Build Units from LFS.

http://www.linuxfromscratch.org/lfs/view/development/chapter04/aboutsbus.html

What a SBU is is the amount of time it takes to compile binutils, factorized for each package.  This is especially helpful if you have a larger build like webkit an administrator might not know that particular package will take an enormous amount of time.  An administrator might choose to defer such operations or assign more metal to their distcc farm if they saw a large number here.  

This would be helpful and progressive.
Comment 1 chadwick.ferguson 2015-04-11 08:39:20 UTC
This is an example of userbase being completely stunned by how long a particular build takes https://forums.gentoo.org/viewtopic-t-922190.html
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-04-11 13:21:02 UTC
This isn't going to be simple, especially that the build time is variable based on many conditions, including not only hardware but also USE flags and current versions of dependencies.

Maybe we could establish some warning mechanism in check-reqs.eclass for packages that take really long to build. They already likely use that eclass because of memory/disk space requirements.
Comment 3 Erik Mackdanz gentoo-dev 2015-04-11 14:27:47 UTC
People would find it useful without expecting exact accuracy. The time estimate for a trip on Google Maps is always a little off, but it's still good to have the ballpark estimate.

The per-package time (including download) can be recorded in vdb on the package's first build. On subsequent builds, estimated build time can be calculated easily. Where packages don't have times, the display can read "The build estimate is 8 minutes plus 3 unknown packages". This doesn't need a comparison to binutils. This is fairly honest since the estimates only come from the same machine, but yields a lot of unknowns.

Further effort could alleviate the unknowns: devs could include an SBU in an ebuild. The ebuild tool can calculate this for a dev, assuming a full start-to-finish run is executed and the dev has a binutils time recorded in vdb.  This adds a thing for devs to remember to do, but it could be opt-in to begin with (maybe make it required in a year, after devs have their baseline (binutils time in vdb))

All the variables involved in an ebuild wouldn't change the time enough to render an estimate so inaccurate as to be useless. "test" and "doc" flags probably add, on average, 10% each to build time. The cases where a build time can triple is if, say, you have to build a binary python module for three targets instead of one. However, those packages are usually smallish.

(Pasted from a reddit comment).
Comment 4 Elias Probst 2015-04-11 20:33:38 UTC
If this ever gets implemented, the way the build-time is determined needs to be improved.
Currently qlop simply parses emerge.log which results in ridiculously high build-times e.g. when suspending the laptop during a build for half a day or so.

Another issue is building multiple packages in parallel, where each --job increment further skews the result.

So instead of simply calculating the build-time based on timestamps, the actual system or user time per package should be used (e.g. similar to "time emerge foobar").
Comment 5 Alexander Berntsen (RETIRED) gentoo-dev 2015-04-14 20:40:05 UTC
(In reply to Elias Probst from comment #4)
> If this ever gets implemented, the way the build-time is determined needs to
> be improved.

See bug 441672.
Comment 6 Patrick McMunn 2015-12-05 04:02:05 UTC
What if there were some program that parsed /var/log/emerge.log, /proc/cpuinfo, and maybe /etc/portage/make.conf in order to gather statistics and this information was uploaded to a database somewhere? It wouldn't be predictive, but it could provide useful information that could then be retrieved and give an estimated compile time for a given package based on information such as type of CPU, amount of RAM available, and number of make jobs used for the compile. Of course, I suppose there should be some kind of mechanism to detect cases where distcc, icecream, ccache, or other such programs were used so as not to include those in the statistics and thus skew results. Maybe it could even be some kind of Portage plug-in. Of course, users should be able to opt in or out as they choose, and the statistics could be anonymous.