Here is a patch that changes the structure of "config.pusedict". The idea is to organize its content in two levels: instead of being a {depkey->[list of flags]} simple dictionnary, it's now a {catpkg->{depkey->[list of flags]}} dictionnary of dictionnaries. That speed up "config.setcpv()" because it allows to easily avoid some unecessary calls to "portage_dep.best_match_to_list()" which were resulting in many calls to "match_from_list()" (100000+ here, but that really depends on your packages.use length). Here are some experimental results on "emerge -pe world", with a 98 lines packages.use file: Before the patch: ================ % time em -pe --nospinner world >/dev/null real 0m9.587s user 0m8.941s sys 0m0.560s ================ And here are the relevant pstats (also for a '-pe world'): ================ Function called... /usr/lib/portage/pym/portage_dep.py:585(match_to_list) /usr/lib/portage/pym/portage_dep.py:610(match_from_list)(112700) 6.110 /usr/lib/portage/pym/portage_dep.py:596(best_match_to_list) /usr/lib/portage/pym/portage_dep.py:585(match_to_list)(1150) 5.650 ================ ncalls tottime percall cumtime percall filename:lineno(function) 1150 0.020 0.000 7.870 0.007 /usr/lib/portage/pym/config.py:452(setcpv) ================ After the patch: ================ % time em -pe --nospinner world >/dev/null real 0m8.397s user 0m7.726s sys 0m0.588s ================ And now the same pstats look like this: ================ Function called... /usr/lib/portage/pym/portage_dep.py:585(match_to_list) /usr/lib/portage/pym/portage_dep.py:610(match_from_list)(83) 1.020 /usr/lib/portage/pym/portage_dep.py:596(best_match_to_list) /usr/lib/portage/pym/portage_dep.py:585(match_to_list)(83) 0.000 ================ ncalls tottime percall cumtime percall filename:lineno(function) 1150 0.010 0.000 2.320 0.002 /usr/lib/portage/pym/config.py:461(setcpv) ================ Nothing that impressive, but that's still a >10% improvement. Reproducible: Always Steps to Reproduce:
Created attachment 53795 [details, diff] 20050318--config_py--pusedict_optimisation.patch The patch is against today HEAD.
Surprised this one missed for so long... Couple of questions: + if puse_depkey not in self.pusedict[puse_cp]: + self.pusedict[puse_cp][puse_depkey] = [] + self.pusedict[puse_cp][puse_depkey].extend(mypusedict[puse_depkey]) Why extending? + self.mycp = portage_dep.dep_getkey(mycpv) Why keeping mycp around?
Created attachment 56088 [details, diff] config_py--pusedict_optimisation.patch That's 2 very good questions :) About self.mycp, i really don't know why i wrote that. And about the list extension, iirc, i was not using grabdict_package at the beginning but simply grabfile, and it was possible that a key appears several times, so i guess that's where it comes from. Anyway, here is a cleaner patch.
Putting a hold on feature requests for portage as they are drowning out the bugs. Most of these features should be available in the next major version of portage. But for the time being, they are just drowning out the major bugs and delaying the next version's progress. Any bugs that contain patches and any bugs for etc-update or dispatch-conf can be reopened. Sorry, I'm just not good enough with bugzilla. ;)
Reopening since this one is not a feature request, but a simple improvement proposal with its patch. I think i've addressed your previous comments with the second attachment, so you may want to have a second look at it. Note that this is against an old HEAD tho, not Brian's rewrite (i've not had time to look at this version yet).
Hmm, seems this one actually has been integrated in 2.0 long time ago. Closing now that 2.1 is officially dead, there's nothing left to do here.
