Bug 11981 – -S / --searchdesc speedup?!

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Torgeir Hansen 2002-12-11 15:15:11 UTC]

Currently, emerge -S is a nice feature to find packages that a simple -s won't dig up for us (me:)
However, emerge -S takes way, WAY to long time to complete!
I'm currently using gentoo on a variety of systems, from a dual ppro 180 mhz to 800Mhz amd and a 850Mhz PIII laptop - and if I get my way, soon I'll be using it on my 175Mhz MIPS aswell! ;]
some have ide-drive(s) and some have a scsi-drive(s), but the systems have all one thing in common;
it takes way too long!

My suggestion is to speed it up, either with a desc.dat or whatever which can be distributed as the portage-files via rsync,
or by a on-line search application.
The latter is my preferance, as it would probably be the fastest way to search - if build properly :)

[Comment 1 Sven Vermeulen (RETIRED) 2003-01-11 04:48:32 UTC]

I'm currently testing various methodes for improving the speed of emerge's "-s",
"--searchdesc", "--upgrade --pretend world", qpkg's "-f" etc... 

The problem is all solutions I have tested yet have some setbacks. Or they make
the system more complicated, or they save information on double locations, or
they require considerate resources (more RAM, f.i.). 

I'll post my results on gentoo-dev after I have finished testing all situations,
but this can still take a lot of time (only 5 of the 12 testcases finished yet).

Some ideas that are already finished testing:
- /usr/portage is on a tmpfs-system (with on-write-disk-access so that it
doesn't disappear when rebooten)
- /usr/portage-files in a mysql-database (+ on disk for when mysql doesn't work)
- portage using specialcrafted binary-file, possibly mounted using loopback and
some special filesystem (requires kerneladjustments) (like rpm's db3 + some
special mount filesystem so that they are seen as regular ebuilds)
- descriptions, rdepend, depend, homepage in one seperate file for all ebuilds
(or mixed: double information storage), configure/make information in seperate
files (as are the ebuilds now)
- caching of the needed information into a temporary file that Portage can use,
and builds during an "emerge (r)sync" (finished: information storage in cache
using B- and B*-trees, still testing other methods)

[Comment 2 Alain Penders (RETIRED) 2003-02-03 01:14:40 UTC]

Any other news on the testing?

[Comment 3 Sven Vermeulen (RETIRED) 2003-02-03 07:51:36 UTC]

Well, at work we're currently implementing Portage with MySQL so that several systems use 1 central database. The setbacks is that Portage won't work if the network's out or the server's down. The advantages are that it's very fast (the server also compiles the packages, but that's another thing).

Having the system work with tmpfs on /usr/portage and /var/db/pkg (with write-back mechanism) is very fast, but it consumes lots of memory (about 250 meg).

Portage using a binary format (such as db3) makes the system less flexible since it's not that easy to upgrade. That is a serious drawback, especially for Gentoo since it has a fast development cycle.

Caching is the same as using a binary format, but it's more flexible (the cache is generated each time the respective emerge-command is executed). However, emerge (r)syncs and installation of software is a little slower (several seconds).

All other options have been found to be not interesting or flawed. Currently MySQL is on second phase research.

[Comment 4 Nicholas Jones (RETIRED) 2003-02-17 06:44:51 UTC]

Athlon700 -- portage-2.0.47-r2

'emerge -s' is less than 2 seconds.

time emerge -S test >/dev/null

real 31.711 seconds
user 23.269 seconds
sys   4.945 seconds