| Summary: | repoman is much slower in portage-2.1_pre10 | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Mr. Bones. (RETIRED) <mr_bones_> |
| Component: | Core | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED WORKSFORME | ||
| Severity: | normal | CC: | halcy0n |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
Mr. Bones. (RETIRED)
2006-04-30 17:50:43 UTC
So, just to document my findings here, I ran category level scans for both sys-kernel and kde-misc and didn't find much difference between pre9 and pre10. The pre10 runs were both slightly longer, but not enough to make anything stand out above the noise level in the profiler runs. It seems that this problem is only really bad for certain types of ebuilds or categories (considering that Halcyon also experienced the performance problem when doing a full repo scan). So, it seems like it may be necessary to profile scans of the full repo. BTW, to profile repoman, here's how I did it: time python -m profile `which repoman` scan http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats is the python profiler output (In reply to comment #2) > http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats is the python profiler > output Thanks. Ideally though, the same exact scan needs to be performed with both pre9 and pre10 so that the stats can be directly compared against eachother (pre9 is the control sample and pre10 is the experimental sample). If I do `python -c 'import pstats; pstats.Stats("repoman-stats.pstats").print_stats()'` it shows me that this run was only about 1 hour which suggests that it's not a full repo scan. Is this from a specific category or something? No idea how the portage profiling thing works but it actually ran for: real 2144m0.165s user 1944m33.336s sys 129m14.177s and sucked up 99%+ of the cpu the whole time. This was a repoman full for the whole tree. (In reply to comment #3) > (In reply to comment #2) > > http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats is the python profiler > > output > > Thanks. Ideally though, the same exact scan needs to be performed with both > pre9 and pre10 so that the stats can be directly compared against eachother > (pre9 is the control sample and pre10 is the experimental sample). It was a full run- meet the python profiler, occasionally known to overflow the time count and go negative ;) Reverse the stats, and you'll see it. Here is the profile run for pre9-r5: http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats-2.1_pre9-r5 After I ran an `emerge --regen`, everything is fine for me. I added two profile runs of the dev-util category for both pre9 and pre10 as requested by ferringb to http://dev.gentoo.org/~mr_bones_/ (In reply to comment #7) > After I ran an `emerge --regen`, everything is fine for me. Hrm. This bug looks like another manifestation of bug 124041. Mr. Bones, are you able to reproduce the problem if you make sure to run `emerge --regen` before the repoman scan? I always run repoman for the whole tree like this: clear ; emerge --sync ; emerge --regen ; time ~/repoman.new full > ~/repoman.out Last night I dedicated my desktop system to repoman benchmarks and got practically identical results for 2.1_pre9-r5 vs. 2.1_pre10-r3 when running `repoman full` on the entire repo:
2.1_pre9-r5 2.1_pre10-r3
real 132m19.175s 133m35.277s
user 91m48.656s 92m45.168s
sys 16m15.321s 15m57.328s
Given these numbers (and Halcy0n's findings), Mr. Bones, it seems that something on your system is skewing your benchmarks. In order to ensure valid benchmark results in my test, I took several precautions:
1) ran `emerge --regen` to ensure that the cache was fully up to date
2) removed all load from the system (paused the cron daemon and didn't run anything that would put IO or cpu load on the system except repoman)
3) installed the version of portage that would be tested
4) ran `time repoman full > ~/repoman.out`
5) repeated steps 3 and 4 with a different version of portage, and the same exact portage tree / cache (no sync in between).
Then go back to versions before pre9. As I *just* mentioned in the previous comment, I always --regen before running. (In reply to comment #12) > Then go back to versions before pre9. The performance difference is likely due to additional QA checks that have been added to repoman in the 2.1_pre series. We don't want to sacrifice QA for performance, so the only option would be to optimize the QA checks for better performance. I'm satisfied with the current performance level, but people are welcome to contribute repoman optimization patches. Actually, here's one that you might test (I haven't tried it myself): http://gentooexperimental.org/~ferringb/portage-on-steroids-again.patch |