Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 289724 - [PATCH-provided] much faster --pretend processing for app-portage/genlop
Summary: [PATCH-provided] much faster --pretend processing for app-portage/genlop
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage Tools Team
Keywords: PATCH
Depends on:
Reported: 2009-10-19 14:00 UTC by Jean-Philippe Robichaud
Modified: 2023-07-18 21:58 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---

patch to improve the speed of genlop -p (fast-genlop.patch,3.95 KB, patch)
2009-10-19 14:01 UTC, Jean-Philippe Robichaud
Details | Diff
compressed emerge.log (emerge.log.gz,893.65 KB, text/plain)
2010-11-08 16:46 UTC, Dennis Nolte

Note You need to log in before you can comment on or make changes to this bug.
Description Jean-Philippe Robichaud 2009-10-19 14:00:37 UTC
When one does emerge -pvuDN @world | genlop -p, it can take forever to come back with the result.  This is especially true if you have many packages on the list and a long emerge.log (mine dates back from early 2008) 

Looking at the genlop code, I saw that the emerge.log is re-read for each package on the pretend list.  This is bad, so I fixed it.

I've done the following:

# eix -I -C kde --format "<category>/<name>" > /tmp/tt
# wc -l /tmp/tt
179 /tmp/tt
# time cat /tmp/tt | genlop -pq
(running this last command multiple time and looking at average)

The results:
Original genlop 0.30.8-r2
real    0m42.382s
user    0m41.289s
sys     0m1.033s 

Patched version
real    0m0.610s
user    0m0.526s
sys     0m0.071s

The patch was build against the latest genlop (0.30.8-r2)
Comment 1 Jean-Philippe Robichaud 2009-10-19 14:01:45 UTC
Created attachment 207550 [details, diff]
patch to improve the speed of genlop -p
Comment 2 Jean-Philippe Robichaud 2009-11-05 04:02:12 UTC
I'm now sure how to proceed for this to get some attention.  So I just add to the CC list email addresses I see in the genlop Changelog file?
Comment 3 Dennis Nolte 2010-11-06 17:06:38 UTC
i used your patch in genlop-0.30.8-r2 , works nice.

 however i get following warning output :

emerge -p mc |genlop -p
These are the pretended packages: (this may take a while; wait...)

[ebuild   R   ] app-misc/mc-4.7.4-r1 
Use of uninitialized value in numeric gt (>) at /usr/bin/genlop line 592, <_GEN_1> line 98084.
                            mc [     1m  7s]

system is portage (default/linux/amd64/10.0/desktop/kde, gcc-4.4.5, glibc-2.12.1-r3, 2.6.36-gentoo x86_64)
System uname: Linux-2.6.36-gentoo-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q6600_@_2.40GHz-with-gentoo-2.0.1

Reproduceable: allways,

with regards
Comment 4 Jean-Philippe Robichaud 2010-11-08 02:54:53 UTC
hum, interesting.

Could you gzip and attach your /var/log/emerge.log?  Perhaps a package name or a special line is causing the precedent regular expression to crash...
Comment 5 Dennis Nolte 2010-11-08 16:46:03 UTC
Created attachment 253639 [details]
compressed emerge.log

As requested, the gzip compressed emerge.log

If you need anything else, let me know.

Comment 6 Jean-Philippe Robichaud 2010-11-08 17:39:29 UTC
hum, your emerge.log is strange (no offense!)

starting at line 97346, you have:

1288857271:  >>> emerge (17 of 76) kde-base/ksysguard-4.5.3 to /         
1288857272:  === (17 of 76) Compiling/Merging (kde-base/ksysguard-4.5.3::/usr/portage/kde-base/ksysguard/ksysguard-4.5.3.ebuild)
1288857366:  === (19 of 76) Merging (kde-base/plasma-runtime-4.5.3::/usr/portage/kde-base/plasma-runtime/plasma-runtime-4.5.3.ebuild)
1288857369:  >>> AUTOCLEAN: kde-base/plasma-runtime:4.5                  
1288857369:  === Unmerging... (kde-base/plasma-runtime-4.5.2)
1288857370:  >>> unmerge success: kde-base/plasma-runtime-4.5.2    
1288857371:  === (19 of 76) Post-Build Cleaning (kde-base/plasma-runtime-4.5.3::/usr/portage/kde-base/plasma-runtime/plasma-runtime-4.5.3.ebuild)                                                          
1288857371:  ::: completed emerge (19 of 76) kde-base/plasma-runtime-4.5.3 to /
1288857371:  >>> emerge (20 of 76) kde-base/kdepimlibs-4.5.3 to /

As you can see, there are no "emerge (19 of 76) ..." line and the whole "18 of 78" block is missing too.  

I can only speculate on why your emerge.log has such content,  I'm not sure how it could have happen: fs corruption, emerge crashing, temporaty out-of-disk-space issue?  I don't know.  

All in all, the problem isn't in the genlop tool itself.

Have a nice day!
Comment 7 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-07-18 21:58:27 UTC
Anyone fancy rebasing it?