| Summary: | portage 2.2rc8 needs too much time and memory to emerge packages | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Joerg Plate <plate> |
| Component: | [OLD] Unspecified | Assignee: | Portage team <dev-portage> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | CC: | dschridde+gentoobugs, ikelos, kingjon3377, nikolay.pershyn, pesa, please.no.spam.here, rogerx.oss, tiago |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | AMD64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 240320 | ||
|
Description
Joerg Plate
2008-08-30 20:12:45 UTC
1GB RAM, ~400MB free, emerge used 45% memory. CPU is 2.4GHz. So you are not hitting swap space at all? Also, memory and time comparisons on your same system using older portage could be useful (go back to whatever version of portage you were using previously that behaved better) I did some quick profiling with portage-2.2_rc8 and portage-2.1.5.6 using
python -m cProfile /usr/bin/emerge -1 x11-misc/xrestop
Postprocessing the profiling output with pstats module shows the following stats.
Exact command was "p.sort_stats('time', 'cum').print_stats(10)"
portage-2.2_rc8:
Ordered by: internal time, cumulative time
List reduced from 1457 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
141 13.935 0.099 13.935 0.099 {built-in method poll}
3 1.770 0.590 8.677 2.892 vartree.py:148(rebuild)
80854 1.308 0.000 1.308 0.000 {posix.lstat}
93336 0.958 0.000 1.614 0.000 posixpath.py:56(join)
508567 0.621 0.000 0.621 0.000 {method 'setdefault' of 'dict' objects}
19031/18981 0.575 0.000 4.850 0.000 posixpath.py:410(realpath)
6 0.557 0.093 0.557 0.093 {built-in method load}
433665 0.505 0.000 0.505 0.000 {method 'append' of 'list' objects}
347558 0.468 0.000 0.468 0.000 {method 'startswith' of 'str' objects}
80694 0.386 0.000 2.016 0.000 posixpath.py:156(islink)
portage-2.1.5.6:
Ordered by: internal time, cumulative time
List reduced from 1006 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
137 11.230 0.082 11.230 0.082 {select.select}
27 0.349 0.013 0.349 0.013 {posix.waitpid}
30519/254 0.263 0.000 0.773 0.003 /usr/lib64/python2.5/copy.py:144(deepcopy)
155219 0.254 0.000 0.254 0.000 {getattr}
73722/36868 0.243 0.000 0.527 0.000 /usr/bin/emerge:1378(__eq__)
168262 0.209 0.000 0.209 0.000 {method 'get' of 'dict' objects}
32985 0.196 0.000 0.294 0.000 /usr/lib64/python2.5/sets.py:356(_update)
4238 0.194 0.000 1.311 0.000 /usr/lib64/portage/pym/portage.py:7216(aux_get)
10196 0.193 0.000 0.193 0.000 {posix.stat}
143960/134706 0.184 0.000 0.207 0.000 {len}
You can notice that the calls to rebuild() [from vartree.py] are new in 2.2.
The callers are unmerge(), _preserve_libs() and treewalk().
Function was called by...
ncalls tottime cumtime
vartree.py:148(rebuild) <- 1 0.596 2.907 vartree.py:1693(unmerge)
1 0.588 2.899 vartree.py:2305(_preserve_libs)
1 0.586 2.871 vartree.py:2576(treewalk)
And the slowest function called by rebuild() is realpath() [from posixpath internal python module].
Function called...
ncalls tottime cumtime
vartree.py:148(rebuild) -> 37772 0.102 0.501 posixpath.py:117(dirname)
18886 0.570 4.826 posixpath.py:410(realpath)
1 0.000 0.000 util.py:85(grabfile)
3 0.000 0.000 vartree.py:131(getPreservedLibs)
3 0.014 0.073 vartree.py:686(cpv_all)
3 0.000 0.000 vartree.py:802(flush_cache)
2535 0.091 0.266 vartree.py:880(aux_get)
37772 0.069 0.069 {filter}
18886 0.022 0.022 {len}
252419 0.289 0.289 {method 'append' of 'list' objects}
37772 0.053 0.053 {method 'replace' of 'str' objects}
504838 0.612 0.612 {method 'setdefault' of 'dict' objects}
59193 0.145 0.145 {method 'split' of 'str' objects}
40307 0.051 0.051 {method 'strip' of 'str' objects}
portage 2.2rc16 Stuck at ">>> Installing mail-client/evolution-2.24.2" for an hour now... Free memory while still running: 128MB Shouldn't the "Product field" be changed from "Gentoo Linux" to "Portage Development"? I've been monitoring memory consumption for the past month here and finally realized, it's Portage who's caching large amounts of memory here. Although, ATop doesn't slate Portage (Python) as using the memory. But after syncing and/or compiling (using Portage/Python), large amounts of memory are apparently consumed by comparing "free -m" output. (I've finally concluded this after dropping to tty and DWM!) My best guess, it's Python and/or un-optimized Python coding. Might be a function or two in Portage not optimized to make proper use of CPU/Memory resources?? Is Bug #236748 a duplicate? Bug #240320 seems like a tracker and hasn't been touched for sometime either. BTW, using =sys-apps/portage-2.2_rc38 here. |
