Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 109016

Summary: ebuild for preload
Product: Gentoo Linux Reporter: Sebastian Bergmann (RETIRED) <sebastian>
Component: New packagesAssignee: Jeremy Olexa (darkside) (RETIRED) <darkside>
Status: RESOLVED FIXED    
Severity: enhancement CC: bugzie, coldwind, honk-online, hurikhan77+bgo, sharpshopter
Priority: High Keywords: EBUILD
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: sys-apps/preload-0.2
files/init.d-preload
sys-apps/preload-0.2
preload-init
preload-conf

Description Sebastian Bergmann (RETIRED) gentoo-dev 2005-10-12 08:35:46 UTC
preload is an adaptive readahead daemon. It monitors applications that users
run, and by analyzing this data, predicts what applications users might run, and
fetches those binaries and their dependencies into memory for faster startup times.

Reproducible: Always
Steps to Reproduce:
Comment 1 Sebastian Bergmann (RETIRED) gentoo-dev 2005-10-12 08:36:15 UTC
Created attachment 70459 [details]
sys-apps/preload-0.2
Comment 2 Sebastian Bergmann (RETIRED) gentoo-dev 2005-10-12 08:36:39 UTC
Created attachment 70460 [details]
files/init.d-preload
Comment 3 Jakub Moc (RETIRED) gentoo-dev 2005-10-12 15:12:14 UTC
License is invalid, src_compile() is redundant and no need to 'dodoc COPYING
INSTALL' :-)
Comment 4 Sebastian Bergmann (RETIRED) gentoo-dev 2005-10-12 22:11:11 UTC
Created attachment 70529 [details]
sys-apps/preload-0.2
Comment 5 Abraham Smith 2005-11-09 09:39:25 UTC
Here is an updated init script, which includes a "reload" option, and another
option, "dump", for dumping the state file, as given in the man page.  

I also see no reason for "after xdm", since there's no real problem with having
it startup before X...

I also added a conf.d for logfile, etc.

I hope this package gets a maintainer real-soon-now, as this is really a neat
program!
Comment 6 Abraham Smith 2005-11-09 09:40:04 UTC
Created attachment 72499 [details]
preload-init
Comment 7 Abraham Smith 2005-11-09 09:40:21 UTC
Created attachment 72500 [details]
preload-conf
Comment 8 Piotr Szymaniak 2006-03-03 14:31:18 UTC
Also an ebuild at:
https://svn.breakmygentoo.net/bmg-main/sys-apps/preload/
Comment 9 Sebastian Bergmann (RETIRED) gentoo-dev 2006-07-01 10:54:51 UTC
Closing as LATER as I do not know whether UPSTREAM is still alive and I also could not see any real benefit from using it.
Comment 10 Jakub Moc (RETIRED) gentoo-dev 2007-01-10 21:00:35 UTC
*** Bug 161404 has been marked as a duplicate of this bug. ***
Comment 11 Andrey A. Ugolnik 2008-03-01 16:48:31 UTC
Version 0.4 of preload available. Please update ebuild.
http://sourceforge.net/projects/preload
Comment 12 Kai Krakow 2008-03-02 05:06:49 UTC
(In reply to comment #11)
> Version 0.4 of preload available. Please update ebuild.
> http://sourceforge.net/projects/preload

I have such an ebuild available in my overlay (which I created before I found this (closed) bug:

http://svn.hurikhan.ath.cx/gentoo/trunk
Comment 13 David D. Huff Jr. 2008-04-01 15:22:57 UTC
Thanks  Kai! These dev wannabes think that software has to updated every six months to have any real meaning or purpose.


Gentoo is becoming more and more a user supported platform because those in charge are not capable of making any meaningful judgments, even about software.
Comment 14 Malte E. 2008-05-06 09:59:33 UTC
In case you are still looking at this, I am also interested in getting this into portage. ATM I am using it from Kai Krakow's overlay.
Comment 15 Kai Krakow 2008-05-06 15:41:58 UTC
I'd volunteer as the ebuild maintainer if that would be of interest and someone introduced me how to become involved...
Comment 16 Kai Krakow 2008-05-13 12:18:26 UTC
Just for anyone who's interested. My ebuild in http://svn.hurikhan.ath.cx/gentoo/trunk has been updates and fixes the shutdown problem of the init-script.
Comment 17 David D. Huff Jr. 2008-05-13 19:11:05 UTC
(In reply to comment #16)
> Just for anyone who's interested. My ebuild in
> http://svn.hurikhan.ath.cx/gentoo/trunk has been updates and fixes the shutdown
> problem of the init-script.
> 

I have been using it, thank you for your efforts. I have also had great success using on a Linux Mint laptop. It is the Ubuntu folks that are reviving it's use.

Will be giving -r3 a whirl as soon as I am done typing here.

Thanks again and yes that would be great if you could be be the maintainer for this, your scripts work perfectly.
Comment 18 Kai Krakow 2008-07-24 21:42:53 UTC
Overlay: http://svn.hurikhan.ath.cx/gentoo/trunk

preload-0.6: version bumped
MAKE SURE YOU ARE AT LEAST AT R91 OF THE OVERLAY

This version includes patches by me for the following problems:
- preload-0.6 can become a fork-bomp
- child processes weren't always cleaned up and stuck in zombie state
- child process counter was prone to race conditions

SF bug link: https://sourceforge.net/tracker/index.php?func=detail&aid=2026901&group_id=143398&atid=755420

Read this first:
----------------

Please please please etc-update your configuration before restarting preload or your system performance may suffer (preload-0.6 is more aggressive by being multithreaded but reduces memory usage in exchange). A future ebuild will include a warning about this.

The fork bomb problem brought my system down several times. I am not sure if this is fixed 100%. Please log out from X, sync your fs and restart preload. REMOVE(!!) it from all runlevels for the first test. Alt+SysRq+U,S,B if it hangs. Please give feedback.

Preload uses ionice by default now if installed. It switches to idle IO priority with my patches by default (original tarball used normal priority).

If preload exits for no reason your preload.state file may be corrupted. Delete it in this case and restart preload. It will be recreated but your usage pattern is lost and must be retrained.
Comment 19 Kai Krakow 2008-07-28 19:25:42 UTC
Overlay: http://svn.hurikhan.ath.cx/gentoo/trunk

preload-0.6.2: version bumped

Fixes:

* Suggestions accepted upstream to keep control of forks
* Incorporated Abraham Smith's ideas into init script
* Includes patch to restart forking burst a bit more early (still testing)
* Should fix zombie problems once and for all
Comment 20 Kai Krakow 2008-07-29 07:57:22 UTC
Overlay: http://svn.hurikhan.ath.cx/gentoo/trunk

preload-0.6.3: version bumped

* Fixes a brain dump of mine which ends in a busy loop
* Fixes a brain dump of the developer which prevents loading the state file

preload-0.6.2: removed due to brain damages ;-)
Comment 21 Pino Silvaggio 2008-07-29 17:34:17 UTC
with preload 6.2 and 6.3 it just stops responding when the first readahead object is found. In my case it's firefox and the cpu tops 100%. I can repoduce this behavior by kill -9 preload and removing the state file. Then restart preload with -V 9, monitor the log and it hangs again.
Comment 22 Kai Krakow 2008-07-29 19:47:09 UTC
(In reply to comment #21)
> with preload 6.2 and 6.3 it just stops responding when the first readahead
> object is found. In my case it's firefox and the cpu tops 100%. I can repoduce
> this behavior by kill -9 preload and removing the state file. Then restart
> preload with -V 9, monitor the log and it hangs again.

This is exactly the problem which should have been fixed with 0.6.3. I could not reproduce it. Did you etc-update the config file? Does it help lowering the   processes option in preload.conf?

I will have a look at the source code if I spot any left-over problems...
Comment 23 Kai Krakow 2008-07-29 20:21:15 UTC
Pino,

I added a vanilla useflag (svn-up the overlay first please) which removes a patch of mine. Can you please check with and without the flag if the problem is reproducable?

If reproducable please strace the preload main process and send an excerpt to me which looks like it's causing the bug (email preferred).
Comment 24 Pino Silvaggio 2008-07-30 01:59:52 UTC
Well well, I re-emerged and now it seems to be working. Not even using vanilla.

I will test further it been now running for 2 hours.

Did you change anything besides adding the vanilla use?
Comment 25 Pino Silvaggio 2008-07-30 02:00:21 UTC
BTW, thanks for providing us preload.
Comment 26 Kai Krakow 2008-07-30 10:46:29 UTC
(In reply to comment #24)
> Well well, I re-emerged and now it seems to be working. Not even using vanilla.
> 
> I will test further it been now running for 2 hours.
> 
> Did you change anything besides adding the vanilla use?

Nope, but I may have missed uploading a fix for my patches with the first overlay checkin. So if you upped during this small time window you may had it missing. I shouldn't be doing such things when I am in a hurry. ;-)

BTW: Upstream is interested in performance differences between vanilla version and patched version. My patch starts preload io bursts a bit earlier instead of waiting for the previous io burst to completely finish. So if anyone wants to try out please post here.

Comment 27 Pino Silvaggio 2008-07-31 20:00:33 UTC
Ok, I am happy to report everything runs A1.
Comment 28 Pino Silvaggio 2008-08-03 23:07:05 UTC
Ok, after using it for a while...

When booting preload reads ahead takes completely takes over and I have to wait for it to finish. I have 8GB RAM and there's 2.3GB of stuff loading. It takes about 20 sec. Previous versions 0.4.x didn't have this behaviour. Isn't preload supposed to be loading stuff when idle or in very load priority?

Also if I try to use preload --nice X it gives me a seg fault.

I am on x86_64 core2 Q9450 with gcc-4.3.
Comment 29 Kai Krakow 2008-08-04 06:54:08 UTC
The 0.6 version starts up to 30 processes in parallel to do the preload which let's the kernel optimize the block read order. As soon as 30 processes are reached the preloading stalls until half the processes are reached. Then preload continues until the queue of files is finished. If you use the vanilla version, preload waits until no child processes are left before continueing. You can try if it helps.

So for tuning: You can change the process count in the config file to use few or more parallel readers. You can also change the tick value which is currently set to 20 seconds. Which means: It waits 20 seconds to ananlyze your running programs and 10 seconds later it will start preloading based on this pattern. You can also try to lower preloads memory usage. And yes: preload uses lower io priority if you have "ionice" installed. The "--nice" switch may be a bug. I'll check it.

It is also known that (by its design) preload may reduce boot time. In turn it speeds up application startup.
Comment 30 Kai Krakow 2008-08-04 06:59:20 UTC
(In reply to comment #29)
> It is also known that (by its design) preload may reduce boot time. In turn it
> speeds up application startup.

Of course that should have read "may increase boot time"...
Comment 31 Pino Silvaggio 2008-08-04 18:06:53 UTC
ionice is nice and all but doesn't work on x86_64

or am I hallucinating ?
Comment 32 Pino Silvaggio 2008-08-12 17:06:14 UTC
Specs:

Gentoo 2008.0 - x86_64 - gcc 4.3.1
Intel Q9450
8GB RAM DDR3
KDE 3.5.9

Using version 0.6.x for some time now:

Boot time is 3 times longer. Usually takes 15-20 sec.
With preload it takes 1 min at least.
Didn't get this behavior with 0.4.x.

ionice is only for x86 so I can not use it.

Using preload crashed my computer a couple of times using
gtk apps like thunderbird or openoffice.
Removing preload fixed the problem.
I can reproduce but it doesn't always crash the same way.

Using --nice X with preload gives a segfault.

Should preload be less aggressive and maybe wake up on idle
(option?). This could complement "nice".
Comment 33 Kai Krakow 2008-08-13 10:45:46 UTC
(In reply to comment #32)
> Using version 0.6.x for some time now:
> 
> Boot time is 3 times longer. Usually takes 15-20 sec.
> With preload it takes 1 min at least.
> Didn't get this behavior with 0.4.x.

I'm using preload with baselayout-2 and parallel startup. There seems to be no difference for me with or without preload (I didn't measure it however). Did you include sbin directories into the configs? You can also try increasing the cycle value in the config to about 40 - which means preload will not do anything before 20 seconds passed after invocation.

> ionice is only for x86 so I can not use it.

I didn't know that - using it on amd64 and it compiled and let's me set priorities on processes. Is there any link you can provide to deep further into the topic please? Thanks. BTW: I'm using the CFS scheduler of the kernel.

> Using preload crashed my computer a couple of times using
> gtk apps like thunderbird or openoffice.
> Removing preload fixed the problem.
> I can reproduce but it doesn't always crash the same way.

Does it crash by spawning too much processes? That bug should have been fixed in my latest ebuild. Otherwise: Try lowering the max processes setting in the config file. It defaults to 30. "0" effectively should disable spawning and return to 0.4 behaviour.

> Using --nice X with preload gives a segfault.

I looked into it and found nothing wrong with the code. It's reported upstream. Meanwhile, use just "-n" which works.

> Should preload be less aggressive and maybe wake up on idle
> (option?). This could complement "nice".

Well it just uses the readahead() advise to let the kernel decide when to schedule reading the files - that's usually not aggressive. Also it should throw away existing files in the cache. Maybe there's an integer overflow in the memory calculation? You offer an extreme amount of RAM in the end... ;-) I'll check that...
Comment 34 Kai Krakow 2008-08-13 10:48:46 UTC
> Also it should
> throw away existing files in the cache.

Of course "it should NOT throw away" files in the cache... Sorry for the typo.
Comment 35 Behdad Esfahbod 2008-08-19 16:40:17 UTC
Fixed the --nice segfault.
Comment 36 Kai Krakow 2008-08-20 20:02:33 UTC
Overlay: http://svn.hurikhan.ath.cx/gentoo/trunk

preload-0.6.3-r1: version bumped

* Updated copyright line
* Fixed segfault on --nice cmdline

Thanx to Behdad for adding the info about his updated upstream CVS...
Comment 37 Pino Silvaggio 2008-08-24 04:18:52 UTC
Alright, this is weird.

Since my last post I've been trying to fix a behaviour which contradicts everything I know about preload (as explained by Kai).

If I enable preload then when booting my hard drive would go crazy mad.

I tried everything including switching from CFQ to anything else. Tried different kernels with different settings to no avail. Well, after getting annoyed I changed filesystems and guess what? Now it runs perfect.

I noticed crappy performance with reiserfs and multithreaded access. So now I use JFS and everything runs A1.

I can't explain it.
Comment 38 Pino Silvaggio 2008-08-24 04:18:52 UTC
Alright, this is weird.

Since my last post I've been trying to fix a behaviour which contradicts everything I know about preload (as explained by Kai).

If I enable preload then when booting my hard drive would go crazy mad.

I tried everything including switching from CFQ to anything else. Tried different kernels with different settings to no avail. Well, after getting annoyed I changed filesystems and guess what? Now it runs perfect.

I noticed crappy performance with reiserfs and multithreaded access. So now I use JFS and everything runs A1.

I can't explain it.
Comment 39 Kai Krakow 2008-08-24 23:10:55 UTC
ReiserFS uses the "Big Kernel Lock" and is known to behave bad in multi CPU environments. So that's probably the explanation.
Comment 40 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2008-10-23 19:47:24 UTC
I maintain this now.
Comment 41 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2008-10-23 19:49:17 UTC
FYI, preload is in portage now. I will be looking over Kai's mods and incorporate some or all of them to the current ebuild.
Comment 42 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2008-10-23 19:56:00 UTC
(bugzilla limitations, can't reassign and resolve...sorry for spam)

So, I am conversing with Kai in email about the current ebuild that is in the overlay that he maintains. I am welcome to additional bug reports but I think given its current state that the outstanding issues have been fixed? I hope so, look forward to additional portage updates soon here for preload.