Bug 90680 – equery can require an excessive amount of memory

• Status: RESOLVED FIXED
• Product: Portage Development

[Description Paul Varner (RETIRED) 2005-04-27 21:44:13 UTC]

I was playing with 'equery hasuse' to answer a question on the gentoo-user ML and had it fill up 1GB of RAM and almost all of 1GB of swap before I killed it. Investigation shows that the find_all_* functions in helpers.py return a list of package objects.

Unfortunately, when dealing with a large number of packages, the memory required to return this list is extremely large.

I am attaching a patch the changes the behavior of these functions to return a list of package names instead, and I have modified equery and etcat to instantiate the package objects as needed from the list of names returned.

These changes resulted in very little memory growth when dealing with the entire portage tree.

[Comment 1 Paul Varner (RETIRED) 2005-04-27 21:45:09 UTC]

Created attachment 57449 [details, diff]
patch to limit memory usage in equery

[Comment 2 Paul Varner (RETIRED) 2005-04-27 21:49:05 UTC]

Created attachment 57450 [details, diff]
patch to limit memory usage in equery

blech, screwed up the patch for etcat.

[Comment 3 Paul Varner (RETIRED) 2005-04-30 00:36:08 UTC]

Comment on attachment 57450 [details, diff]
patch to limit memory usage in equery

I found the underlying cause of the issue.  I will submit a patch later.

[Comment 4 Paul Varner (RETIRED) 2005-04-30 01:29:00 UTC]

Created attachment 57657 [details, diff]
package.py patch

Underlying cause was package.py creating a copy of portage.config object for
every package object created.  This patch solves the memory usage and
significantly speeds up equery as well.  I'll attach full benchmarks but an
equery hasuse -p perl dropped from 14 minutes to 46 seconds

[Comment 5 Paul Varner (RETIRED) 2005-04-30 01:29:47 UTC]

Created attachment 57658 [details]
Patch benchmarks

[Comment 6 Marius Mauch (RETIRED) 2005-04-30 02:45:35 UTC]

Problem is that we need this copy as the setcpv() (required to reflect package.use settings) call makes the config instance package specific :-/

[Comment 7 Marius Mauch (RETIRED) 2005-04-30 03:23:08 UTC]

Hmm, I should have actually checked the patch first ...
It should work, but I'm not really happy with that if statement, just thinking about a more general solution (in portage.config) right now.

[Comment 8 Paul Varner (RETIRED) 2005-07-19 09:21:34 UTC]

*** Bug 99517 has been marked as a duplicate of this bug. ***

[Comment 9 Paul Varner (RETIRED) 2005-09-16 09:49:01 UTC]

Created attachment 68588 [details, diff]
package.py.patch

[Comment 10 Paul Varner (RETIRED) 2005-09-20 10:34:39 UTC]

Created attachment 68874 [details, diff]
gentoolkit patch

Updated patch to add semaphore access to the global portage.config object

[Comment 11 Paul Varner (RETIRED) 2005-09-20 18:17:17 UTC]

Created attachment 68900 [details, diff]
Updated gentoolkit patch

Changed Sempaphore to Lock, removed reset() call.

[Comment 12 Paul Varner (RETIRED) 2005-09-22 08:54:44 UTC]

Fix is in subversion

[Comment 13 Paul Varner (RETIRED) 2005-09-23 15:49:40 UTC]

Fix is in gentoolkit-0.2.1_pre8

[Comment 14 Paul Varner (RETIRED) 2006-01-17 19:10:28 UTC]

Fixed in gentoolkit-0.2.1