Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 161244

Summary: app-portage/portage-utils: qlop: -c should include ETA
Product: Portage Development Reporter: Harm Geerts <gentoo+bugs>
Component: Third-Party ToolsAssignee: Fabian Groffen <grobian>
Status: RESOLVED FIXED    
Severity: enhancement CC: devurandom, flameeyes, sbraz
Priority: High Keywords: Goal
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=186721
https://bugs.gentoo.org/show_bug.cgi?id=442406
https://bugs.gentoo.org/show_bug.cgi?id=658824
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: portage-utils-0.1.22-qlop-eta.patch
portage-utils-cvs-qlop-eta.patch
portage-utils-cvs-qlop-eta.patch

Description Harm Geerts 2007-01-10 01:22:17 UTC
I've made an attempt to add ETA functionality to `qlop --current`

The algorithm is the same as genlop uses with one minor difference.
If the user is allowed to read /proc/pid/environ, it will fetch the category 
from it to work around ambiguous packagenames.

The average time calculation is a stripped version of `qlop --time`. All 
timestamps are read from emerge.log and the last match is the starttime for 
the current emerge. I found this easier to work with then the process uptime.

If you like it you're free to include it, but feedback is always welcome.

Thanks for these excellent tools :)

Harm Geerts
Comment 1 Harm Geerts 2007-01-10 01:24:32 UTC
Created attachment 106286 [details, diff]
portage-utils-0.1.22-qlop-eta.patch
Comment 2 solar (RETIRED) gentoo-dev 2007-01-10 02:04:11 UTC
I'm getting failures on testing your patch local. Would you mind re doing 
your patch to cvs vs the last release. fbsd added some code to this same function.
Infact some of the code you have reword could/should probably be split 
out into a new function so less code would have to be duped.
Comment 3 Diego Elio Pettenò (RETIRED) gentoo-dev 2007-01-10 02:04:43 UTC
This is gonna get in a conflict with the change I've committed to support qlop -c on FreeBSD.

I think the cleanest way to handle this is to factor out the process data gathering (platform dependent) from the displaying and the ETA support, so that the platform specific function would just get the start date and the time elapsed, while the platform-independent code can take care of the rest.

This way, we wouldn't need to copy the code for every new platform supported (almost every platform has its different interface to get processes' data), but would just have to find the two parameters.
Actually there's just one information needed, the start_date, as you can easily find the elapsed seconds since that once you finished the processing (that might take a few seconds, I suppose, for big enough logfiles).
Comment 4 Harm Geerts 2007-01-10 21:01:18 UTC
Created attachment 106430 [details, diff]
portage-utils-cvs-qlop-eta.patch

no fair, I submitted my patch before you committed yours ;)

Couldn't test this on *BSD, but I think I got it right.

The start_date is the last match from emerge.log, it reflects when emerge started working on a package. So all we need to get the correct result is the packagename.

I think this is better then pulling the start_date from the process info, which only tells us the elapsed time of the current stage. (unpack, compile, etc.)
Comment 5 Harm Geerts 2007-01-10 21:06:54 UTC
Created attachment 106442 [details, diff]
portage-utils-cvs-qlop-eta.patch

*cough* shouldn't have used cvs diff
Comment 6 solar (RETIRED) gentoo-dev 2007-01-13 19:20:36 UTC
Flameeyes: Please cvs up and test this on fBSD. tia
Comment 7 solar (RETIRED) gentoo-dev 2007-01-15 19:02:31 UTC
Pushed 0.1.23 but excluded this patch. needinfo from bsd team.
Comment 8 solar (RETIRED) gentoo-dev 2007-01-19 19:47:39 UTC
Something is not right with your code.

pitr ~ # ./q lop -c
Segmentation fault

pitr ~ # q lop -c
 * desktop-file-utils-0.11
     started: Fri Jan 19 19:39:54 2007
     elapsed: 12 seconds

pitr ~ # ./q lop -c
Segmentation fault

The q in ./ is with your additions.


(gdb) bt full
#0  0x08074321 in strstr ()
No symbol table info available.
#1  0x08054ac5 in show_current_emerge (logfile=0x80d9ffc "/var/log/emerge.log") at qlop.c:429
        proc = (DIR *) 0x80d9f24
        de = <value optimized out>
        buf = {"[nautilus-2.16.3] sandbox\000/usr/lib/portage/bin/ebuild.sh compile", '\0' <repeats 8127 times>, 
  "PORTAGE_DEPCACHEDIR=/var/cache/edb/dep\000LESS=-R -M --shift 5\000OPENGL_PROFILE=xorg-x11\000DISTDIR=/var/tmp/portage/nautilus-2.16.3/distdir\000INFOPATH=/usr/share/info:/usr/s_64-pc-linux-gnu/3."...}
        path = "/proc/13028/environ\000\000\000\000\000\000\000\200\206\020\bØú\204ÿsÆ\b\b\002\000\000\000ø\r\205ÿÑ\237\r\b"
        p = 0xff84baaa "] sandbox"
        atom = <value optimized out>
#2  0x080558e8 in qlop_main (argc=2, argv=0xff850df8) at qlop.c:539
        i = <value optimized out>
        average = 1
        do_time = 0 '\0'
        do_list = 0 '\0'
        do_unlist = 0 '\0'
        do_sync = 0 '\0'
        do_current = 1 '\001'
        do_human_readable = 0 '\0'
        opt_logfile = 0x0
        logfile = 0x80d9ffc "/var/log/emerge.log"
        __FUNCTION__ = "qlop_main"
#3  0x0804ea25 in q_main (argc=3, argv=0xff850df4) at q.c:132
        buf = '\0' <repeats 3772 times>, "$\234\r\b\016\000\000\000\001\000\000\000 \230\017\b\000\002\000\0003\234\r\b\f\000\000\000\002\000\000\000 \232\017\b\000\004\000\000@\20\000\001\000\000\0003\024\020\b\024\000\000\000E\234\r\b\016\000\000\000\001\000\000\000àÀ\017\b\000\020\000\000¿\231\r\b\a\000\000\000\000\000\000\000k\v\205ÿ\001\000\000\000T\2000\002\000\000\000 \226\017\b\000\002\000\000]\234\r\b\a\000\000\000\001\000\000\000@ \017\b\000\020\000\000e\234\r\b\017\000\000\000\001\000\000\000 \224\017\b\000\002\000\000u\00\000\001\000\000\000 \222\017\b\000\002\000\000\204\234\r\b\006\000\000\000\001\000"...
        i = <value optimized out>
        p = <value optimized out>
        func = (APPLET) 0x80555f0 <qlop_main>
        __FUNCTION__ = "q_main"
#4  0x0805e11e in main (argc=3, argv=0xff850df4) at main.c:1035
No locals.


(gdb) bt
#0  0x08074321 in strstr ()
#1  0x08054ac5 in show_current_emerge (logfile=0x80d9ffc "/var/log/emerge.log") at qlop.c:429
#2  0x080558e8 in qlop_main (argc=2, argv=0xff850df8) at qlop.c:539
#3  0x0804ea25 in q_main (argc=3, argv=0xff850df4) at q.c:132
#4  0x0805e11e in main (argc=3, argv=0xff850df4) at main.c:1035
Comment 9 Harm Geerts 2007-01-19 21:48:36 UTC
I'm a little confused by your backtrace.
The bt is from a segfault on line 429
It says p = 0xff84baaa "] sandbox" which is from buf[0]
But on line 428 I assign p to buf[1]

Why does p still point to buf[0] ?

424: if (buf[0][0] == '[' && (p = strchr(buf[0], ']')) != NULL && strstr(buf[0], "sandbox") != NULL) {
425: 	/* try to fetch category from process env */
426: 	snprintf(path, sizeof(path), "/proc/%i/environ", pid);
427: 	if (eat_file(path, buf[1], sizeof(buf[1]))) {
428: 		p = buf[1];
429: 		while (strstr(p, "PORTAGE_BUILDDIR=") == NULL)
430: 			p = strchr(p, '\0')+1;

Comment 10 solar (RETIRED) gentoo-dev 2007-05-12 03:29:52 UTC
I must of had other changes pending or so as I'm unable to reproduce the error now

/var/cvsroot/gentoo-projects/portage-utils/qlop.c,v  <--  qlop.c
new revision: 1.38; previous revision: 1.37
Comment 11 Harm Geerts 2007-05-12 12:13:09 UTC
The bug is valid, but I'm not sure how to solve it in a decent manner.

It triggers when /proc/PID/environ is 4096 bytes and doesn't hold the variable PORTAGE_BUILDDIR
4096 bytes seems to be the maximum and the rest is trimmed off.

I made a hack that searches multiple variables until it finds a match but even then it is possible for this to trigger so I didn't want to include that either.

I didn't expect this to be included which is why I haven't responded until now. It's probably best to take out the part that searches for the category in /proc/PID/env
Comment 12 solar (RETIRED) gentoo-dev 2007-05-12 14:42:43 UTC
Ok then I've reverted the patch. If/when you have a new one that's bug free feel free to attach it here for inclusion in a future release.

/var/cvsroot/gentoo-projects/portage-utils/qlop.c,v  <--  qlop.c
new revision: 1.39; previous revision: 1.38
Comment 13 solar (RETIRED) gentoo-dev 2008-03-15 03:32:24 UTC
Reopen when ready.
Comment 14 Dennis Schridde 2010-06-07 11:36:29 UTC
Any news on this? This is the main reason for to still have genlop installed... Reopening this bug would be nice.
Comment 15 Dennis Schridde 2013-05-02 13:25:26 UTC
(In reply to comment #14)
> Any news on this? This is the main reason for to still have genlop
> installed... Reopening this bug would be nice.

ETA calculation is still missing in qlop. This issue is definitely not RESOLVED/FIXED.
Comment 16 Till Schäfer 2017-06-29 12:42:59 UTC
I find the proposal of Bug 498338 rather convincing. I.e. we should not include the ETA per default (as it requires scanning the complete emerge.log file and thus is expensive if invoked on a regular basis. E.g. i use qlop in conky), but allow the combined parameter -ct to show the expected merge time in addition to the current package build time.
Comment 17 Fabian Groffen gentoo-dev 2018-03-28 19:35:25 UTC
*** Bug 498338 has been marked as a duplicate of this bug. ***
Comment 18 Fabian Groffen gentoo-dev 2018-03-28 19:36:47 UTC
*** Bug 569104 has been marked as a duplicate of this bug. ***
Comment 19 Louis Sautier (sbraz) gentoo-dev 2019-01-31 11:59:25 UTC
I'd love seeing this implemented :) qlop is generally way faster than genlop
Comment 20 Fabian Groffen gentoo-dev 2019-01-31 12:11:18 UTC
I'm not that optimistic it would be easy to get here, but technically speaking from the log we'd be able to harvest unfinished merges - that as a start
Comment 21 Larry the Git Cow gentoo-dev 2019-02-27 20:53:29 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage-utils.git/commit/?id=c74a5abf3fd8db03adb531f95ecff5316d997ab3

commit c74a5abf3fd8db03adb531f95ecff5316d997ab3
Author:     Fabian Groffen <grobian@gentoo.org>
AuthorDate: 2019-02-27 20:41:45 +0000
Commit:     Fabian Groffen <grobian@gentoo.org>
CommitDate: 2019-02-27 20:50:21 +0000

    qlop: rewrite from scratch
    
    This new implementation achieves a few things:
    - unified code path for all modes of operation (thus consistent results)
    - more flexibility to implement new features
    - simplification of the codebase
    
    Short version of what this commit changes:
    - existing flags -t -g have been replaced with -a and -t
    - -c has been been replaced with -r and no longer uses proc processing
      code (thus works everywhere)
    - addition of an ETA for running emerges (subject to improvements)
    - allow reading a file of atoms (e.g. /var/lib/portage/world)
    - summary mode -c to compute grand total, e.g. to compute world compile
      time
    
    Bug: https://bugs.gentoo.org/161244
    Bug: https://bugs.gentoo.org/603024
    Bug: https://bugs.gentoo.org/636334
    Bug: https://bugs.gentoo.org/658824
    Signed-off-by: Fabian Groffen <grobian@gentoo.org>

 man/include/qlop.desc         |   25 +-
 man/include/qlop.optdesc.yaml |   23 +-
 man/qlop.1                    |   66 +-
 qlop.c                        | 1479 +++++++++++++++++++++--------------------
 tests/qlop/dotest             |   20 +-
 tests/qlop/list01.good        |    4 +-
 tests/qlop/list02.good        |    6 +-
 tests/qlop/list03.good        |    8 +-
 tests/qlop/list04.good        |    2 +-
 tests/qlop/list05.good        |    2 +-
 tests/qlop/list06.good        |    4 +-
 tests/qlop/list07.good        |    4 +-
 tests/qlop/list08.good        |    4 +-
 tests/qlop/list09.good        |    6 +-
 14 files changed, 892 insertions(+), 761 deletions(-)
Comment 22 Fabian Groffen gentoo-dev 2019-06-06 07:32:40 UTC
this is available onwards from 0.80_pre*