Summary: | app-portage/portage-utils: `qlop -g <pkgname>` shows fewer merges than `qlop -l <pkgname>` | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Niklas Haas <gentoo> |
Component: | Third-Party Tools | Assignee: | Fabian Groffen <grobian> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | marduk, ostroffjh |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | AMD64 | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- |
Description
Niklas Haas
2016-12-18 18:17:50 UTC
qlop doesn't have a limit of 50 entries. more likely your log file is confusing it. it is not an easy to parse file. please compress & attach yours. @SpanKY: Which log file exactly are you talking about? I have a collection of files in /var/log/portage/elog that all begin with ‘app-emulation:wine’, 236 in total. Or is there some other central log I should be looking for? (In reply to Niklas Haas from comment #2) qlop doesn't look at any elog files, it looks at the central log file that portage maintains. by default, that file lives at /var/log/emerge.log. /var/log/emerge.log is too large to attach here even after XZ compression. I've uploaded it here: https://snow.haasn.xyz/~nand/log.xz I'm seeing the same problem: ~ # qlop -l llvm Sun May 27 08:45:02 2012 >>> sys-devel/llvm-2.9-r2 Thu Sep 6 00:22:35 2012 >>> sys-devel/llvm-3.0-r2 Sun Nov 25 15:33:33 2012 >>> sys-devel/llvm-3.1-r2 Sun Mar 9 23:00:36 2014 >>> sys-devel/llvm-3.3-r3 Sat Dec 27 00:15:52 2014 >>> sys-devel/llvm-3.4.2 Thu Jan 29 04:39:33 2015 >>> sys-devel/llvm-3.5.0 Sun Apr 26 05:54:31 2015 >>> sys-devel/llvm-3.5.0 Sat Jul 2 15:49:16 2016 >>> sys-devel/llvm-3.7.1-r2 Sat Oct 29 12:44:10 2016 >>> sys-devel/llvm-3.7.1-r3 Tue Dec 6 05:59:26 2016 >>> sys-devel/llvm-3.7.1-r3 Sat Apr 8 21:53:18 2017 >>> sys-devel/llvm-3.9.1-r1 ~ # qlop -gH llvm llvm: Thu Sep 6 00:12:07 2012: 10 minutes, 28 seconds llvm: Sun Nov 25 15:19:45 2012: 13 minutes, 48 seconds llvm: Sun Mar 9 22:43:53 2014: 16 minutes, 43 seconds llvm: Fri Dec 26 23:42:10 2014: 33 minutes, 42 seconds llvm: Thu Jan 29 03:38:14 2015: 1 hour, 1 minute, 19 seconds llvm: Sun Apr 26 04:24:38 2015: 1 hour, 29 minutes, 53 seconds llvm: Sat Jul 2 12:54:24 2016: 2 hours, 54 minutes, 52 seconds llvm: Sat Oct 29 09:54:53 2016: 2 hours, 49 minutes, 17 seconds llvm: Tue Dec 6 02:34:36 2016: 3 hours, 24 minutes, 50 seconds llvm: 9 times Doesn't seem to be related with the number of merges. In this case the first and last merge are missing in the second output. Using portage-utils-0.64. I have noticed the same issue, but with -g showing the first n (in this particular case, only those prior to some date. $qlop -l webkit-gtk Sun May 31 01:03:35 2015 >>> net-libs/webkit-gtk-2.4.8 Sun May 31 11:17:38 2015 >>> net-libs/webkit-gtk-2.4.8-r200 Tue Jun 9 02:15:39 2015 >>> net-libs/webkit-gtk-2.6.5 Wed Jun 24 09:47:51 2015 >>> net-libs/webkit-gtk-2.4.9 Wed Jun 24 19:53:25 2015 >>> net-libs/webkit-gtk-2.4.9-r200 Thu Jun 25 11:53:11 2015 >>> net-libs/webkit-gtk-2.6.6 Sun Sep 13 06:05:11 2015 >>> net-libs/webkit-gtk-2.8.5 Fri Dec 11 03:20:20 2015 >>> net-libs/webkit-gtk-2.4.9-r200 Wed Feb 24 21:54:03 2016 >>> net-libs/webkit-gtk-2.10.7 Wed Apr 13 11:48:53 2016 >>> net-libs/webkit-gtk-2.10.9 Thu Apr 14 05:14:33 2016 >>> net-libs/webkit-gtk-2.4.10 Fri May 13 17:41:37 2016 >>> net-libs/webkit-gtk-2.4.10 Sat May 14 10:25:46 2016 >>> net-libs/webkit-gtk-2.10.9 Wed Jul 13 07:10:24 2016 >>> net-libs/webkit-gtk-2.4.11 Tue Oct 4 17:52:01 2016 >>> net-libs/webkit-gtk-2.12.5 Sat Dec 3 20:59:28 2016 >>> net-libs/webkit-gtk-2.12.5 Sat Jan 14 02:26:09 2017 >>> net-libs/webkit-gtk-2.12.5 Tue Feb 14 10:35:34 2017 >>> net-libs/webkit-gtk-2.14.4 Sat Feb 18 06:56:29 2017 >>> net-libs/webkit-gtk-2.14.5 Thu Apr 13 23:46:45 2017 >>> net-libs/webkit-gtk-2.16.1 Tue May 16 06:52:02 2017 >>> net-libs/webkit-gtk-2.16.2 Thu Jun 1 22:42:38 2017 >>> net-libs/webkit-gtk-2.16.3 Mon Jun 26 18:48:52 2017 >>> net-libs/webkit-gtk-2.16.4 Fri Jun 30 02:35:13 2017 >>> net-libs/webkit-gtk-2.16.5 Thu Aug 3 20:24:57 2017 >>> net-libs/webkit-gtk-2.16.6 Fri Oct 27 01:06:17 2017 >>> net-libs/webkit-gtk-2.16.6 Sun Oct 29 23:24:13 2017 >>> net-libs/webkit-gtk-2.18.2 Sat Nov 25 04:43:48 2017 >>> net-libs/webkit-gtk-2.18.3 Mon Dec 11 03:44:53 2017 >>> net-libs/webkit-gtk-2.18.3 Wed Dec 27 18:36:17 2017 >>> net-libs/webkit-gtk-2.18.4 $ qlop -g webkit-gtk webkit-gtk: Sat May 30 12:36:39 2015: 44816 seconds webkit-gtk: Sun May 31 01:03:35 2015: 36843 seconds webkit-gtk: Mon Jun 8 12:02:47 2015: 51172 seconds webkit-gtk: Tue Jun 23 21:37:03 2015: 43848 seconds webkit-gtk: Wed Jun 24 09:47:51 2015: 36334 seconds webkit-gtk: Wed Jun 24 19:53:25 2015: 57586 seconds webkit-gtk: Sat Sep 12 14:39:50 2015: 55521 seconds webkit-gtk: Thu Dec 10 15:13:31 2015: 43609 seconds webkit-gtk: Tue Apr 12 14:50:41 2016: 75492 seconds webkit-gtk: Wed Apr 13 11:48:53 2016: 62740 seconds webkit-gtk: Fri May 13 04:08:48 2016: 48769 seconds webkit-gtk: Fri May 13 17:41:37 2016: 60249 seconds webkit-gtk: Tue Jul 12 17:08:55 2016: 50489 seconds webkit-gtk: 13 times rather then upload emerge.log to some other site, is there a "grep" which will include the relevant lines? Are there by chance failed merges in there? Or multiple portage runs at the same time? Possible, yes, but not likely for every case after July 2016. I wonder if something about either the last emerge shown by -g, or something in the log shortly after that, is messing up qlop so it stops reporting at that point. Again - what grep can I use to get the lines qlop is looking at (or is reading the code the only way?) I don't have an answer ready, I need to look into this myself :) Have a look at the code, here: https://gitweb.gentoo.org/proj/portage-utils.git/tree/qlop.c#n76 From line 106 it's going to loop line-by-line over the logfile. For some reason yet unknown to me, list and gauge use different code. Making both share the same code, will likely bring them in sync. On second look, no. The problem is the gauge code gets confused by the log, because there is an emerge terminating (1404894355: Started emerge on: Jul 09, 2014 10:25:54 check for llvm-3.4.2) during the merging. The gauge code tries to track parallel running emerges in order to make sense of things, but in this case it didn't know there was another emerge already running when llvm started, hence it thought the main emerge ended, which is not true. Additional question - does "ebuild" put anything into the emerge log? I have had numerous cases where an emerge failed, and instead of starting from scratch, I managed to correct the problem and then manually do the remaining ebuild steps. (often just install and qmerge - after emerge failed due to a sandbox violation, but after actually completing the compile step.) As far as I am aware, only emerge logs into emerge.log, so ebuild X operations aren't visible. Yesterday I managed to fix one case where -g didn't see a merge, but it still isn't complete. My aim is to get -l and -g to align, if possible. I managed to get one case covered, but the intrinsic problem is that the logfile isn't consistent, which is causing problems. There are 7719 "Started emerge" messages, and 7665 "terminating." messages. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=80d7de29b2c1f977e84bc653933545a6eb589694 commit 80d7de29b2c1f977e84bc653933545a6eb589694 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2018-04-04 13:13:15 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2018-04-04 13:13:15 +0000 show_merge_times: start counting parallel merges from start of the file If we start counting parallel merges only once we found something interesting, we might not be aware of other emerges at that time, drawing wrong conclusions when seeing a termination message. Bug: https://bugs.gentoo.org/603024 qlop.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-)} The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=c74a5abf3fd8db03adb531f95ecff5316d997ab3 commit c74a5abf3fd8db03adb531f95ecff5316d997ab3 Author: Fabian Groffen <grobian@gentoo.org> AuthorDate: 2019-02-27 20:41:45 +0000 Commit: Fabian Groffen <grobian@gentoo.org> CommitDate: 2019-02-27 20:50:21 +0000 qlop: rewrite from scratch This new implementation achieves a few things: - unified code path for all modes of operation (thus consistent results) - more flexibility to implement new features - simplification of the codebase Short version of what this commit changes: - existing flags -t -g have been replaced with -a and -t - -c has been been replaced with -r and no longer uses proc processing code (thus works everywhere) - addition of an ETA for running emerges (subject to improvements) - allow reading a file of atoms (e.g. /var/lib/portage/world) - summary mode -c to compute grand total, e.g. to compute world compile time Bug: https://bugs.gentoo.org/161244 Bug: https://bugs.gentoo.org/603024 Bug: https://bugs.gentoo.org/636334 Bug: https://bugs.gentoo.org/658824 Signed-off-by: Fabian Groffen <grobian@gentoo.org> man/include/qlop.desc | 25 +- man/include/qlop.optdesc.yaml | 23 +- man/qlop.1 | 66 +- qlop.c | 1479 +++++++++++++++++++++-------------------- tests/qlop/dotest | 20 +- tests/qlop/list01.good | 4 +- tests/qlop/list02.good | 6 +- tests/qlop/list03.good | 8 +- tests/qlop/list04.good | 2 +- tests/qlop/list05.good | 2 +- tests/qlop/list06.good | 4 +- tests/qlop/list07.good | 4 +- tests/qlop/list08.good | 4 +- tests/qlop/list09.good | 6 +- 14 files changed, 892 insertions(+), 761 deletions(-) this is available onwards from 0.80_pre* |