Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 276813 - sys-apps/portage Performance improvement
Summary: sys-apps/portage Performance improvement
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 835380 210077 288499
  Show dependency tree
 
Reported: 2009-07-06 19:02 UTC by Marat Radchenko
Modified: 2023-06-14 01:26 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
cache_heavy_regexes.patch (cache_heavy_regexes.patch,2.85 KB, patch)
2009-07-07 06:27 UTC, Marat Radchenko
Details | Diff
faster_loops_in_setcpv_v1.patch (faster_loops_in_setcpv_v1.patch,2.84 KB, patch)
2009-07-10 07:18 UTC, Marat Radchenko
Details | Diff
faster_loops_in_setcpv_v2.patch (faster_loops_in_setcpv_v2.patch,3.71 KB, patch)
2009-07-10 07:28 UTC, Marat Radchenko
Details | Diff
faster_loops_in_setcpv_v4.patch (faster_loops_in_setcpv_v4.patch,3.62 KB, patch)
2009-07-13 19:02 UTC, Marat Radchenko
Details | Diff
use_readlines_when_reading_whole_file.patch (use_readlines_when_reading_whole_file.patch,1.35 KB, patch)
2009-07-17 06:59 UTC, Marat Radchenko
Details | Diff
Removes getattr call from Task (avoid_slow_reflection.patch,506 bytes, patch)
2009-09-06 13:06 UTC, Marat Radchenko
Details | Diff
build tuple in one go in catpkgsplit instead of array + extend + tuple() (catpkgsplit_remove_array_extend.patch,665 bytes, patch)
2009-09-06 13:07 UTC, Marat Radchenko
Details | Diff
implements isvalidatom using regex (regex_isvalidatom.patch,5.26 KB, patch)
2009-09-07 20:25 UTC, Marat Radchenko
Details | Diff
Simplifies isvalidatom regex and adds comments (verbose_regex.patch,2.18 KB, patch)
2009-09-08 16:32 UTC, Marat Radchenko
Details | Diff
Fixed verbose regex (verbose_regex_v2.patch,2.26 KB, patch)
2009-09-08 16:59 UTC, Marat Radchenko
Details | Diff
even simpler regex (verbose_regex_v3.patch,2.25 KB, patch)
2009-09-08 17:16 UTC, Marat Radchenko
Details | Diff
Even more simpler regex (verbose_regex_v4.patch,2.25 KB, patch)
2009-09-08 17:56 UTC, Marat Radchenko
Details | Diff
Turns Package.metadata into a dict (dict_pkg_metadata.patch,3.60 KB, patch)
2009-09-19 07:51 UTC, Marat Radchenko
Details | Diff
Use existing atom instances instead of doing Atom(Atom) (avoid_unnecessary_atom_creation.patch,13.67 KB, patch)
2009-09-19 13:18 UTC, Marat Radchenko
Details | Diff
Adds caching to module_wrapper (cache_wrappers.patch,1.64 KB, patch)
2009-09-20 10:36 UTC, Marat Radchenko
Details | Diff
Makes Atom inherit from str and increases Atom creation rate from 1.2k per second to 1.6k (atom_str.patch,4.27 KB, patch)
2009-09-20 12:57 UTC, Marat Radchenko
Details | Diff
Simplify match_to_list down to single list comprehension (simplify_match_to_list.patch,581 bytes, patch)
2009-09-21 07:08 UTC, Marat Radchenko
Details | Diff
Optimizes unicode wrappers by avoiding redundant isinstance checks (faster_unicode_wrappers.patch,2.34 KB, patch)
2009-09-22 06:42 UTC, Marat Radchenko
Details | Diff
Improves vartree.getpath timing from 1.2 msec to 100 usec (faster_vartree_getpath.patch,866 bytes, patch)
2009-09-23 06:10 UTC, Marat Radchenko
Details | Diff
Improves porttree.auxget timings from 11msec down to 5.8msec (faster_porttree_auxget.patch,4.30 KB, patch)
2009-09-23 18:20 UTC, Marat Radchenko
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Marat Radchenko 2009-07-06 19:02:43 UTC
In its current shape, portage has serious performance problems (mostly cpu-wise). Is has acceptable speed on desktop/server but on embedded or low-end devices, using portage is a real pain.

This bug is intended to track patches/ideas related to performance improvements.
Comment 1 Marat Radchenko 2009-07-07 06:27:51 UTC
Created attachment 196987 [details, diff]
cache_heavy_regexes.patch

cache_heavy_regexes.patch uses dict-based cache to reuse extremely heavy regexes from Package.regex and dbapi._iter_match_use. Additionally, _iter_match_use no longer builds one huge regex, but splits it into implicit/explicit. Implicit part (very large) is same fo all packages so it avoids heavy recompilation.

`emerge -vpuND world system` goes down from 4:00 to 3:35 mins on my 400mhz armv6 box with this patch.
Comment 2 Marat Radchenko 2009-07-07 07:13:03 UTC
cache_heavy_regexes.patch was applied (in modified form) as r13801.
Comment 3 Marat Radchenko 2009-07-10 07:18:49 UTC
Created attachment 197434 [details, diff]
faster_loops_in_setcpv_v1.patch

faster_loops_in_setcpv_v1.patch improves performance of setcpv by using list comprehensions instead of manual loops wherever possible. It reduces `emerge -puND world system` from 3:35 to 3:20.
Comment 4 Marat Radchenko 2009-07-10 07:28:21 UTC
Created attachment 197437 [details, diff]
faster_loops_in_setcpv_v2.patch

faster_loops_in_setcpv_v2.patch is an alternative approach to setcpv performance improvement (see previous comment).

Instead of doing len(use_expand_split) * len(iuse_implicit) loops, this patch iterates over iuse_implicit once, builds a dict of prefix/[iuse_implicit] and then iterates over entries of this dict.

It reduces `emerge -puND world system` from 3:35 to 3:05.
Comment 5 Zac Medico gentoo-dev 2009-07-11 20:25:28 UTC
(In reply to comment #3)
> Created an attachment (id=197434) [edit]
> faster_loops_in_setcpv_v1.patch

Thanks, this is in svn r13815.


(In reply to comment #4)
> Created an attachment (id=197437) [edit]
> faster_loops_in_setcpv_v2.patch

+		use_expand_iuses = dict()
+		for item in iuse_implicit:
+			s = item.split("_", 1)
+			if len(s) > 1:
+				prefix = s[0]
+				l = use_expand_iuses.get(prefix, set())
+				if not l:
+					use_expand_iuses[prefix] = l
+				l.add(item)

This part seems to assume that prefix does not contain underscore, which is not true for things like video_cards.
Comment 6 Zac Medico gentoo-dev 2009-07-12 23:06:18 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > Created an attachment (id=197437) [edit]
> > faster_loops_in_setcpv_v2.patch
>
> This part seems to assume that prefix does not contain underscore, which is not
> true for things like video_cards.

In svn r13823 I've committed a version of this optimization which accounts for the above issue.
Comment 7 Marat Radchenko 2009-07-13 19:02:48 UTC
Created attachment 197827 [details, diff]
faster_loops_in_setcpv_v4.patch

faster_loops_in_setcpv_v4.patch makes setcpv even more faster
Comment 8 Zac Medico gentoo-dev 2009-07-16 21:45:16 UTC
(In reply to comment #7)
> Created an attachment (id=197827) [edit]
> faster_loops_in_setcpv_v4.patch
> 
> faster_loops_in_setcpv_v4.patch makes setcpv even more faster

How much faster is it? It's a lot less readable. Also, I think those x[:len(y)] == y comparisons should really be x[:len(y)+1] == y + "_".
Comment 9 Marat Radchenko 2009-07-17 06:59:30 UTC
Created attachment 198253 [details, diff]
use_readlines_when_reading_whole_file.patch

use_readlines_when_reading_whole_file.patch

This patch makes `emerge -puND world system` 5% faster by using file.readline()  instead of list(file).
Comment 10 Zac Medico gentoo-dev 2009-07-19 22:02:38 UTC
(In reply to comment #9)
> Created an attachment (id=198253) [edit]
> use_readlines_when_reading_whole_file.patch

Thanks, that's in svn r13834.
Comment 11 Zac Medico gentoo-dev 2009-08-03 23:12:31 UTC
This is fixed in 2.2_rc34.
Comment 12 Marat Radchenko 2009-09-06 13:06:13 UTC
Created attachment 203267 [details, diff]
Removes getattr call from Task
Comment 13 Marat Radchenko 2009-09-06 13:07:58 UTC
Created attachment 203269 [details, diff]
build tuple in one go in catpkgsplit instead of array + extend + tuple()
Comment 14 Marat Radchenko 2009-09-06 13:24:40 UTC
Just ideas:
1. Split load_emerge_config into separate config/vartree reads. It'll speedup things that do not actually need vartree (--help as one example). Low priority.
2. Rewrite isvalidatom & friends with regexes. It currently uses ~20% of all init time (before depgraph creation).
3. Make emerge functions bypass unicode module and function wrappers introduced recently. That's 20% more of init time.
Comment 15 Zac Medico gentoo-dev 2009-09-07 02:59:19 UTC
(In reply to comment #12)
> Created an attachment (id=203267) [edit]
> Removes getattr call from Task
(In reply to comment #13)
> Created an attachment (id=203269) [edit]
> build tuple in one go in catpkgsplit instead of array + extend + tuple()

Thanks, those are in svn r14209 and r14211.
Comment 16 Marat Radchenko 2009-09-07 20:25:43 UTC
Created attachment 203406 [details, diff]
implements isvalidatom using regex

Attached patch reimplements isvalidatom using regex. It reduces time spent in isvalidatom from 10 to 2 seconds doing 'emerge -vp paludis' (43->35 'user' total). It also should improve repoman dependency checking.

The only known problem is over relaxed pkg name part, which is supposed to reject pkg name that 'ends in a hyphen followed by one or more digits' '~foo/bar-1-0.5', but allows it instead. However previous implementation also  allowed it, so no regression problem is introduced.
Comment 17 Alec Warner archtester gentoo-dev Security 2009-09-07 21:21:28 UTC
(In reply to comment #16)
> Created an attachment (id=203406) [edit]
> implements isvalidatom using regex
> 
> Attached patch reimplements isvalidatom using regex. It reduces time spent in
> isvalidatom from 10 to 2 seconds doing 'emerge -vp paludis' (43->35 'user'
> total). It also should improve repoman dependency checking.
> 
> The only known problem is over relaxed pkg name part, which is supposed to
> reject pkg name that 'ends in a hyphen followed by one or more digits'
> '~foo/bar-1-0.5', but allows it instead. However previous implementation also 
> allowed it, so no regression problem is introduced.
> 

I would modify the test to actually fail on foo-1-3-5 whatever.  If the code is 'broken' the test should also be broken.  I believe I wrote code that you could use to mark the test as an expected failure so it can be fixed later.
Comment 18 Zac Medico gentoo-dev 2009-09-07 22:39:54 UTC
(In reply to comment #16)
> Created an attachment (id=203406) [edit]
> implements isvalidatom using regex

Thanks, that's is svn r14213.
Comment 19 Marat Radchenko 2009-09-08 16:32:03 UTC
Created attachment 203509 [details, diff]
Simplifies isvalidatom regex and adds comments

Attached patch greatly simplifies atom regex (winning even more performance) and turns it in verbose mode with comments. Added more corner case tests.
Comment 20 Marat Radchenko 2009-09-08 16:59:47 UTC
Created attachment 203512 [details, diff]
Fixed verbose regex

I somehow messed up with previous patch, this is a corrected version.
Comment 21 Marat Radchenko 2009-09-08 17:16:03 UTC
Created attachment 203515 [details, diff]
even simpler regex
Comment 22 Marat Radchenko 2009-09-08 17:56:07 UTC
Created attachment 203519 [details, diff]
Even more simpler regex

Sorry for spam :( this is the last version.
Comment 23 Zac Medico gentoo-dev 2009-09-08 18:12:25 UTC
(In reply to comment #22)
> Created an attachment (id=203519) [edit]

Thanks, this is in svn r14219.
Comment 24 Marat Radchenko 2009-09-19 07:51:24 UTC
Created attachment 204572 [details, diff]
Turns Package.metadata into a dict

Attached patch increases Package creation speed by turning Package.metadata into a dict and avoiding slow reflection.
Comment 25 Zac Medico gentoo-dev 2009-09-19 12:59:18 UTC
(In reply to comment #24)
> Created an attachment (id=204572) [edit]
> Turns Package.metadata into a dict

Thanks, this is in svn r14280.
Comment 26 Marat Radchenko 2009-09-19 13:18:47 UTC
Created attachment 204597 [details, diff]
Use existing atom instances instead of doing Atom(Atom)

Attached patch makes code to use existing atom instances instead of doing Atom(Atom). Also, cpv_getkey is done with regex now.

This patch reduces atom creations from 70k to 18k on emerge -puND world here (having 1.3k per second, that saves 40 seconds!).
Comment 27 Zac Medico gentoo-dev 2009-09-19 17:07:05 UTC
(In reply to comment #26)
> Created an attachment (id=204597) [edit]
> Use existing atom instances instead of doing Atom(Atom)

Thanks, that's in svn r14282.
Comment 28 Marat Radchenko 2009-09-20 10:36:41 UTC
Created attachment 204673 [details, diff]
Adds caching to module_wrapper

Attached patch adds caching to module_wrapper.
That reduces time of 'emerge -vp paludis' from 35 to 32 seconds
Comment 29 Marat Radchenko 2009-09-20 12:57:08 UTC
Created attachment 204685 [details, diff]
Makes Atom inherit from str and increases Atom creation rate from 1.2k per second to 1.6k

Attached patch makes Atom inherit from str and increases Atom creation rate from 1.2k per second to 1.6k.

This eliminates the need in str methods in Atom.
Comment 30 Daniel Robbins 2009-09-20 18:29:32 UTC
Thanks very much for the Portage improvements, Marat! :) Keep up the good work and keep sending us improvements please :)
Comment 31 Zac Medico gentoo-dev 2009-09-20 19:35:39 UTC
(In reply to comment #28)
> Created an attachment (id=204673) [edit]
> Adds caching to module_wrapper

(In reply to comment #29)
> Created an attachment (id=204685) [edit]
> Makes Atom inherit from str and increases Atom creation rate from 1.2k per
> second to 1.6k

Thanks, those are in svn r14299 and r14300.
Comment 32 Marat Radchenko 2009-09-21 07:08:33 UTC
Created attachment 204770 [details, diff]
Simplify match_to_list down to single list comprehension

Attached patch simplifies match_to_list down to single list comprehension.
Comment 33 Zac Medico gentoo-dev 2009-09-21 15:29:43 UTC
(In reply to comment #32)
> Created an attachment (id=204770) [edit]
> Simplify match_to_list down to single list comprehension

Thanks, this is in svn r14325.
Comment 34 Marat Radchenko 2009-09-22 06:42:31 UTC
Created attachment 204883 [details, diff]
Optimizes unicode wrappers by avoiding redundant isinstance checks

Attached patch optimizes unicode wrappers by avoiding redundant isinstance checks.
Comment 35 Zac Medico gentoo-dev 2009-09-22 18:37:49 UTC
(In reply to comment #34)
> Created an attachment (id=204883) [edit]
> Optimizes unicode wrappers by avoiding redundant isinstance checks

Thanks, this is in svn r14376.
Comment 36 Marat Radchenko 2009-09-23 06:10:06 UTC
Created attachment 204996 [details, diff]
Improves vartree.getpath timing from 1.2 msec to 100 usec

Attached patch improves vartree.getpath timing from 1.2 msec to 100 usec. Such big difference is caused by unicode wrappers.
Comment 37 Zac Medico gentoo-dev 2009-09-23 06:36:23 UTC
(In reply to comment #36)
> Created an attachment (id=204996) [edit]
> Improves vartree.getpath timing from 1.2 msec to 100 usec

Thanks, that's in svn r14392.
Comment 38 Marat Radchenko 2009-09-23 18:20:46 UTC
Created attachment 205039 [details, diff]
Improves porttree.auxget timings from 11msec down to 5.8msec

Attached patch improves porttree.auxget timings from 11msec down to 5.8msec. This results in reducing 'emerge -vp paludis' from 30secs to 26.5secs
Comment 39 Zac Medico gentoo-dev 2009-09-24 05:49:02 UTC
(In reply to comment #38)
> Created an attachment (id=205039) [edit]
> Improves porttree.auxget timings from 11msec down to 5.8msec

Thanks, this is in svn r14398.
Comment 40 Zac Medico gentoo-dev 2009-10-11 00:55:38 UTC
This is fixed in 2.1.7.