I did a command to calculate the world compile time by listing all installed packages with qlist and using the average compile time of each of them provided by qlop. The output would be in HH:MM:SS format. I documented this in the forum (https://forums.gentoo.org/viewtopic-t-1078212.html) but not everyone reads the forum so I think it would be nice to have it officially. The command is: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' My suggestion of syntax would be qlop --world Reproducible: Always
In the same forum thread, some people had huge calculated times. Probably this is caused by some packages having it's compile times calculated wrong by qlop -t or even the files that qlop uses for this are with wrong values. Anyway, I did some commands to solve this issue: 1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation: qlist -I | xargs qlop -t | awk '{ if ($2 < 5400) secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' 2) Discovering the problematic build times (maybe report a bug?). I want to point out which packages build times are bigger than 5400 seconds (1h30min): qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'
qlop isn't perfect (and with the current portage log-format is never going to be), your script seems "trivial" enough though to be used as script (adding C-support for it would basically mimick that).
the trickery needed for 161244 is likely allowing this kind of work to be done more easily as well
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=c74a5abf3fd8db03adb531f95ecff5316d997ab3 commit c74a5abf3fd8db03adb531f95ecff5316d997ab3 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2019-02-27 20:41:45 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2019-02-27 20:50:21 +0000 qlop: rewrite from scratch This new implementation achieves a few things: - unified code path for all modes of operation (thus consistent results) - more flexibility to implement new features - simplification of the codebase Short version of what this commit changes: - existing flags -t -g have been replaced with -a and -t - -c has been been replaced with -r and no longer uses proc processing code (thus works everywhere) - addition of an ETA for running emerges (subject to improvements) - allow reading a file of atoms (e.g. /var/lib/portage/world) - summary mode -c to compute grand total, e.g. to compute world compile time Bug: https://bugs.gentoo.org/161244 Bug: https://bugs.gentoo.org/603024 Bug: https://bugs.gentoo.org/636334 Bug: https://bugs.gentoo.org/658824 Signed-off-by: Fabian Groffen <grobian@gentoo.org> man/include/qlop.desc | 25 +- man/include/qlop.optdesc.yaml | 23 +- man/qlop.1 | 66 +- qlop.c | 1479 +++++++++++++++++++++-------------------- tests/qlop/dotest | 20 +- tests/qlop/list01.good | 4 +- tests/qlop/list02.good | 6 +- tests/qlop/list03.good | 8 +- tests/qlop/list04.good | 2 +- tests/qlop/list05.good | 2 +- tests/qlop/list06.good | 4 +- tests/qlop/list07.good | 4 +- tests/qlop/list08.good | 4 +- tests/qlop/list09.good | 6 +- 14 files changed, 892 insertions(+), 761 deletions(-)
(In reply to rudregues from comment #1) > In the same forum thread, some people had huge calculated times. Probably > this is caused by some packages having it's compile times calculated wrong > by qlop -t or even the files that qlop uses for this are with wrong values. > Anyway, I did some commands to solve this issue: > > 1) Calculating total sum excluding outliers defined by the user. I want > everything bigger than 5400 seconds (1h30min) to be removed from calculation: > qlist -I | xargs qlop -t | awk '{ if ($2 < 5400) secs += $2} END { > printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' ./qlop -c [snip] dev-db/postgresql: 74 seconds average for 1 unmerge app-eselect/eselect-postgresql: 3 seconds average for 1 unmerge app-admin/pwgen: 5 seconds average for 2 unmerges sync: 80 seconds average for 1 sync total: 265109 seconds for 2086 merges, 74 unmerges, 1 sync (can't do the exclusions, but to be honest, I find that a bit weird) > 2) Discovering the problematic build times (maybe report a bug?). I want to > point out which packages build times are bigger than 5400 seconds (1h30min): > qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s > %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, > $2 % 60); }' ./qlop -mt [snip] 2019-02-27T18:31:32 >>> dev-db/sqlite-3.27.1: 115 seconds 2019-02-27T18:33:27 >>> dev-python/pyopenssl-19.0.0: 42 seconds 2019-02-27T18:34:09 >>> dev-libs/libpcre-8.43: 89 seconds 2019-02-27T18:35:38 >>> net-dns/bind-tools-9.12.3_p4: 279 seconds 2019-02-27T18:40:17 >>> dev-vcs/git-2.21.0: 166 seconds should be easy enough to grep from here to find your offenders
this is available onwards from 0.80_pre*
