Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 115600 - emerge -S is slow
Summary: emerge -S is slow
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All All
: Low enhancement (vote)
Assignee: Portage team
Depends on:
Reported: 2005-12-14 15:42 UTC by Eugene Wong
Modified: 2005-12-17 16:19 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Eugene Wong 2005-12-14 15:42:37 UTC
I noticed that the ebuilds still contain descriptions. For example, each ebuild 
within /usr/portage/kde-base/kdebase/ has
'DESCRIPTION="KDE base packages: the desktop, panel, window manager, konqueror.

This is redundant in that each ebuild basically has the same description. With 
each "emerge --sync", the user is downloading the same information over and over 

Also, having only 1 small file to read means that it will be faster for the user 
to search for keywords within the descriptions.

I recommend that you put all of the information within a file called "desc". So, 
continuing with the KDE example above, we would remove that line from all the 
files, and create a file called, /usr/portage/kde-base/kdebase/desc. When users 
search for a keyword in the description of this software, they will only have to 
search within 1 file as opposed to 10.

Reproducible: Always
Steps to Reproduce:
This is not a bug, but here are the steps to search descriptions.
1. emerge -S the
Actual Results:  
I haven't tried it in a while, because it takes so long. It's faster for me to 
take a guess and just open an ebuild file in a text editor. It's also faster to 
use the internet to search for a description.

Expected Results:  
It should have quickly found the keyword in the description.
Comment 1 SpanKY gentoo-dev 2005-12-14 15:58:42 UTC
bugzilla is not the appropriate forum for this

e-mail gentoo-dev mailing list
Comment 2 Brian Harring (RETIRED) gentoo-dev 2005-12-15 00:38:13 UTC
Will save the effort...

Centralizing DESCRIPTION metadata out of ebuilds and into a single file... no. 
Cache exists to hold metadata per ebuild- the cache is slow (your complaints
about emerge -S) due to 2 factors- default cache backend is single file per
ebuild (look in $PORTDIR/metadata/cache for an example)- this however isn't the
major killer.  

Major killer is that an emerge -S operation has to access *all* entries in the
cache, regardless of the backend.  The interface for accessing it internally
isn't well defined enough to allow handing off to the backend in a sane way-
fex, you would think that a RDBMS backend would be fast, it's not.  Why? 
Because of the design, portage has to go entry by entry, thus what could be a
single select becomes 20,000.

The cache subsystem is modularized, and self contained- each backend stores
however it wants.  Adding a file as you suggested forces _all_ backends to now
use a mandated file, which is rather bad- breaks flexibility, but more
importantely, unless the interface for querying the backend is improved, it
doesn't give you _any_ speed gains without building hacks into the source.

We're not building anymore hacks into portage btw. ;-)

So... that's why centralizing DESCRIPTION metadata isn't going to magically make
things faster- what it would do however, is hork up ebuild design.  Instead of
having ebuilds standalone (sans their eclasses), now the description metadata
resides in a global file that has to be shipped around with the ebuild.  Not a
good thing.

So... cache interface needs improvement.  If that's done, then improving the
actual backend (say the backend actually description metadata centralized) would
have an actual gain.
Comment 3 Eugene Wong 2005-12-15 12:24:35 UTC
Hi again.

Thanks for both of the replies. I didn't realize that this wasn't the place to 
make suggestions. I appreciate the details. I didn't realize that portage was so 
complex. :^)
Comment 4 Brian Harring (RETIRED) gentoo-dev 2005-12-16 04:40:08 UTC
>Thanks for both of the replies. I didn't realize that this wasn't the place to 
>make suggestions. I appreciate the details. I didn't realize that portage was so 
Nothing wrong with suggestions- if my statement comes off forcefull, well, bit
of history is the reason.  -S *is* slow, but fixing it needs be done in a manner
that doesn't break the cache modularization.  So... just dumped the details,
roads people have pushed for (and the faults of 'em).

So... yeah, nothing wrong with suggestions.  Just noticed the bug, and figured
it was time to jot down another recap of the problem, solutions, etc. :)
Comment 5 solar (RETIRED) gentoo-dev 2005-12-16 07:51:13 UTC
Several other tools exist that are better suited for the job of simple searching 
over the portage internal methods. portage-utils:qsearch, esearch and eix 
to name a few. Most users have discovered these tools.
IMO I think the -S remains mostly for reference at this point. 
If anything should be done with it it would probably be full removal.
Comment 6 Brian Harring (RETIRED) gentoo-dev 2005-12-16 19:45:42 UTC
Full removal won't occur.  The tools that exist, _all_ have shortcomings- either they read from the cache directly, or they maintain their own cache.

The further we get with the cache backend and restriction framework, the more likely the cache backend will be changed by users.  Result in that case would be all of the external tools no longer working (whether on disk change, or api).

So... yeah, external tools now, but -S isn't going away- it gets souped up when restriction framework is mainlined.
Comment 7 Eugene Wong 2005-12-17 16:19:04 UTC
(In reply to comment #4)
> if my statement comes off forcefull, well, bit
> of history is the reason.

No, it wasn't. I was saying sorry in reply to the other fellow who said that this is off-topic. I hate it when people post off-topic, and I felt bad about it.

I appreciated all the details that you gave.