preload is an adaptive readahead daemon. It monitors applications that users
run, and by analyzing this data, predicts what applications users might run, and
fetches those binaries and their dependencies into memory for faster startup times.
Steps to Reproduce:
Created attachment 70459 [details]
Created attachment 70460 [details]
License is invalid, src_compile() is redundant and no need to 'dodoc COPYING
Created attachment 70529 [details]
Here is an updated init script, which includes a "reload" option, and another
option, "dump", for dumping the state file, as given in the man page.
I also see no reason for "after xdm", since there's no real problem with having
it startup before X...
I also added a conf.d for logfile, etc.
I hope this package gets a maintainer real-soon-now, as this is really a neat
Created attachment 72499 [details]
Created attachment 72500 [details]
Also an ebuild at:
Closing as LATER as I do not know whether UPSTREAM is still alive and I also could not see any real benefit from using it.
*** Bug 161404 has been marked as a duplicate of this bug. ***
Version 0.4 of preload available. Please update ebuild.
(In reply to comment #11)
> Version 0.4 of preload available. Please update ebuild.
I have such an ebuild available in my overlay (which I created before I found this (closed) bug:
Thanks Kai! These dev wannabes think that software has to updated every six months to have any real meaning or purpose.
Gentoo is becoming more and more a user supported platform because those in charge are not capable of making any meaningful judgments, even about software.
In case you are still looking at this, I am also interested in getting this into portage. ATM I am using it from Kai Krakow's overlay.
I'd volunteer as the ebuild maintainer if that would be of interest and someone introduced me how to become involved...
Just for anyone who's interested. My ebuild in http://svn.hurikhan.ath.cx/gentoo/trunk has been updates and fixes the shutdown problem of the init-script.
(In reply to comment #16)
> Just for anyone who's interested. My ebuild in
> http://svn.hurikhan.ath.cx/gentoo/trunk has been updates and fixes the shutdown
> problem of the init-script.
I have been using it, thank you for your efforts. I have also had great success using on a Linux Mint laptop. It is the Ubuntu folks that are reviving it's use.
Will be giving -r3 a whirl as soon as I am done typing here.
Thanks again and yes that would be great if you could be be the maintainer for this, your scripts work perfectly.
preload-0.6: version bumped
MAKE SURE YOU ARE AT LEAST AT R91 OF THE OVERLAY
This version includes patches by me for the following problems:
- preload-0.6 can become a fork-bomp
- child processes weren't always cleaned up and stuck in zombie state
- child process counter was prone to race conditions
SF bug link: https://sourceforge.net/tracker/index.php?func=detail&aid=2026901&group_id=143398&atid=755420
Read this first:
Please please please etc-update your configuration before restarting preload or your system performance may suffer (preload-0.6 is more aggressive by being multithreaded but reduces memory usage in exchange). A future ebuild will include a warning about this.
The fork bomb problem brought my system down several times. I am not sure if this is fixed 100%. Please log out from X, sync your fs and restart preload. REMOVE(!!) it from all runlevels for the first test. Alt+SysRq+U,S,B if it hangs. Please give feedback.
Preload uses ionice by default now if installed. It switches to idle IO priority with my patches by default (original tarball used normal priority).
If preload exits for no reason your preload.state file may be corrupted. Delete it in this case and restart preload. It will be recreated but your usage pattern is lost and must be retrained.
preload-0.6.2: version bumped
* Suggestions accepted upstream to keep control of forks
* Incorporated Abraham Smith's ideas into init script
* Includes patch to restart forking burst a bit more early (still testing)
* Should fix zombie problems once and for all
preload-0.6.3: version bumped
* Fixes a brain dump of mine which ends in a busy loop
* Fixes a brain dump of the developer which prevents loading the state file
preload-0.6.2: removed due to brain damages ;-)
with preload 6.2 and 6.3 it just stops responding when the first readahead object is found. In my case it's firefox and the cpu tops 100%. I can repoduce this behavior by kill -9 preload and removing the state file. Then restart preload with -V 9, monitor the log and it hangs again.
(In reply to comment #21)
> with preload 6.2 and 6.3 it just stops responding when the first readahead
> object is found. In my case it's firefox and the cpu tops 100%. I can repoduce
> this behavior by kill -9 preload and removing the state file. Then restart
> preload with -V 9, monitor the log and it hangs again.
This is exactly the problem which should have been fixed with 0.6.3. I could not reproduce it. Did you etc-update the config file? Does it help lowering the processes option in preload.conf?
I will have a look at the source code if I spot any left-over problems...
I added a vanilla useflag (svn-up the overlay first please) which removes a patch of mine. Can you please check with and without the flag if the problem is reproducable?
If reproducable please strace the preload main process and send an excerpt to me which looks like it's causing the bug (email preferred).
Well well, I re-emerged and now it seems to be working. Not even using vanilla.
I will test further it been now running for 2 hours.
Did you change anything besides adding the vanilla use?
BTW, thanks for providing us preload.
(In reply to comment #24)
> Well well, I re-emerged and now it seems to be working. Not even using vanilla.
> I will test further it been now running for 2 hours.
> Did you change anything besides adding the vanilla use?
Nope, but I may have missed uploading a fix for my patches with the first overlay checkin. So if you upped during this small time window you may had it missing. I shouldn't be doing such things when I am in a hurry. ;-)
BTW: Upstream is interested in performance differences between vanilla version and patched version. My patch starts preload io bursts a bit earlier instead of waiting for the previous io burst to completely finish. So if anyone wants to try out please post here.
Ok, I am happy to report everything runs A1.
Ok, after using it for a while...
When booting preload reads ahead takes completely takes over and I have to wait for it to finish. I have 8GB RAM and there's 2.3GB of stuff loading. It takes about 20 sec. Previous versions 0.4.x didn't have this behaviour. Isn't preload supposed to be loading stuff when idle or in very load priority?
Also if I try to use preload --nice X it gives me a seg fault.
I am on x86_64 core2 Q9450 with gcc-4.3.
The 0.6 version starts up to 30 processes in parallel to do the preload which let's the kernel optimize the block read order. As soon as 30 processes are reached the preloading stalls until half the processes are reached. Then preload continues until the queue of files is finished. If you use the vanilla version, preload waits until no child processes are left before continueing. You can try if it helps.
So for tuning: You can change the process count in the config file to use few or more parallel readers. You can also change the tick value which is currently set to 20 seconds. Which means: It waits 20 seconds to ananlyze your running programs and 10 seconds later it will start preloading based on this pattern. You can also try to lower preloads memory usage. And yes: preload uses lower io priority if you have "ionice" installed. The "--nice" switch may be a bug. I'll check it.
It is also known that (by its design) preload may reduce boot time. In turn it speeds up application startup.
(In reply to comment #29)
> It is also known that (by its design) preload may reduce boot time. In turn it
> speeds up application startup.
Of course that should have read "may increase boot time"...
ionice is nice and all but doesn't work on x86_64
or am I hallucinating ?
Gentoo 2008.0 - x86_64 - gcc 4.3.1
8GB RAM DDR3
Using version 0.6.x for some time now:
Boot time is 3 times longer. Usually takes 15-20 sec.
With preload it takes 1 min at least.
Didn't get this behavior with 0.4.x.
ionice is only for x86 so I can not use it.
Using preload crashed my computer a couple of times using
gtk apps like thunderbird or openoffice.
Removing preload fixed the problem.
I can reproduce but it doesn't always crash the same way.
Using --nice X with preload gives a segfault.
Should preload be less aggressive and maybe wake up on idle
(option?). This could complement "nice".
(In reply to comment #32)
> Using version 0.6.x for some time now:
> Boot time is 3 times longer. Usually takes 15-20 sec.
> With preload it takes 1 min at least.
> Didn't get this behavior with 0.4.x.
I'm using preload with baselayout-2 and parallel startup. There seems to be no difference for me with or without preload (I didn't measure it however). Did you include sbin directories into the configs? You can also try increasing the cycle value in the config to about 40 - which means preload will not do anything before 20 seconds passed after invocation.
> ionice is only for x86 so I can not use it.
I didn't know that - using it on amd64 and it compiled and let's me set priorities on processes. Is there any link you can provide to deep further into the topic please? Thanks. BTW: I'm using the CFS scheduler of the kernel.
> Using preload crashed my computer a couple of times using
> gtk apps like thunderbird or openoffice.
> Removing preload fixed the problem.
> I can reproduce but it doesn't always crash the same way.
Does it crash by spawning too much processes? That bug should have been fixed in my latest ebuild. Otherwise: Try lowering the max processes setting in the config file. It defaults to 30. "0" effectively should disable spawning and return to 0.4 behaviour.
> Using --nice X with preload gives a segfault.
I looked into it and found nothing wrong with the code. It's reported upstream. Meanwhile, use just "-n" which works.
> Should preload be less aggressive and maybe wake up on idle
> (option?). This could complement "nice".
Well it just uses the readahead() advise to let the kernel decide when to schedule reading the files - that's usually not aggressive. Also it should throw away existing files in the cache. Maybe there's an integer overflow in the memory calculation? You offer an extreme amount of RAM in the end... ;-) I'll check that...
> Also it should
> throw away existing files in the cache.
Of course "it should NOT throw away" files in the cache... Sorry for the typo.
Fixed the --nice segfault.
preload-0.6.3-r1: version bumped
* Updated copyright line
* Fixed segfault on --nice cmdline
Thanx to Behdad for adding the info about his updated upstream CVS...
Alright, this is weird.
Since my last post I've been trying to fix a behaviour which contradicts everything I know about preload (as explained by Kai).
If I enable preload then when booting my hard drive would go crazy mad.
I tried everything including switching from CFQ to anything else. Tried different kernels with different settings to no avail. Well, after getting annoyed I changed filesystems and guess what? Now it runs perfect.
I noticed crappy performance with reiserfs and multithreaded access. So now I use JFS and everything runs A1.
I can't explain it.
ReiserFS uses the "Big Kernel Lock" and is known to behave bad in multi CPU environments. So that's probably the explanation.
I maintain this now.
FYI, preload is in portage now. I will be looking over Kai's mods and incorporate some or all of them to the current ebuild.
(bugzilla limitations, can't reassign and resolve...sorry for spam)
So, I am conversing with Kai in email about the current ebuild that is in the overlay that he maintains. I am welcome to additional bug reports but I think given its current state that the outstanding issues have been fixed? I hope so, look forward to additional portage updates soon here for preload.