Bug 436084 – emerge performance bad due to unnecessary binary package scanning

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Konstantin Münning 2012-09-24 10:08:20 UTC]

Hello!

Quite a while now I am seeing a continuously decreasing performance of emerge. One of the worst performance killers seems to be that emerge almost always scans binary packages in PKGDIR even if no binary packages are requested (no -k|-K|-g|-G flag). This includes operations which do not need binary packages at all like emerge --depclean.

If there is no important reason to keep current behavior it would be great to make emerge scan PKGDIR only when binary packages are requested.

[Comment 1 Zac Medico 2012-09-24 15:20:36 UTC]

This is fixed in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=c4703d79878e4e0eb8e2b36e49c0bdee835b847e

[Comment 2 Zac Medico 2012-09-26 04:37:43 UTC]

This is fixed in 2.1.11.22 and 2.2.0_alpha131.

[Comment 3 Konstantin Münning 2012-09-26 14:40:15 UTC]

That's great. Tested 2.1.11.22 and now an emerge -pv portage takes 5 minutes instead of 14, so a huge improvement.

Are there any further plans to improve performance? After package scanning a worthwhile optimization target seem to be dependency calculations. If there is anything I could do, let me know but I have no experience with python profiling.

[Comment 4 Zac Medico 2012-09-26 16:17:33 UTC]

For python profiles, I have a couple of scripts that you can use to get started:

  http://dev.gentoo.org/~zmedico/tmp/profile.sh
  http://dev.gentoo.org/~zmedico/tmp/dump_pstats.py

Usage is like this:

   profile.sh emerge.pstats emerge <args>
   dump_pstats.py emerge.pstats > emerge.pstats.txt

The second command converts the profiling stats to a human-readable form, which is easy to analyze.