There is always too much talk about speeding up system boot and this need for speed grew after other systems (mainly Windows XP and MacOS X) adopted some techiniques. Gentoo have the most advanced and fast Linux boot, it provides a great dependency system that supports parallel service starts. But its far from ideal. There are some on going researches to speed things, from kernel to user space. The kernel part was already identified and is being worked by CE Linux, sponsored by Sony, Fujistu and others and patches are going main stream as issues are being squashed. But user space is up to gentoo to adopt. Is this side of things that I would like to focus. I would like to propose a group to study, discuss, develop and test these techiniques. Some references: - http://www.finux.org/Reprints/Reprint-Bird-OLS2004.pdf - http://tree.celinuxforum.org/pubwiki/moin.cgi/BootupTimeReductionHowto Discussion in Gentoo World: - http://bugs.gentoo.org/show_bug.cgi?id=55329 - http://bugs.gentoo.org/show_bug.cgi?id=64724 - http://forums.gentoo.org/viewtopic.php?t=131142 Ideas: - Scripts should be tested to conform to busybox shell and this shell should be used (http://tree.celinuxforum.org/pubwiki/moin.cgi/OptimizeRCScripts) - rc-update should create an optimized file with every dependency calculated, comments and white spaces removed and in future dead code eliminated. The later requires too much work, but the former are a matter of a sed expression. With this we can create an optimized BSD-style file which will be used instead of using the current process, avoiding hundreds of forks to run the heavy bash. This is the easiest part. - modules-update should be patched with http://bugs.gentoo.org/show_bug.cgi?id=55329 - Provide means of cache and read ahead files used at boot time: http://bugs.gentoo.org/show_bug.cgi?id=64724 - Use a ram/initrd or any other really fast file system in memory with files need at boot time. Maybe provide a tool that check for access time of every file and its size and put them in this memory disc until some limit size is reach in the access time order. This will eliminate HD head moviments and make the system boot really fast. - Clarify users about patches that may reduce boot times but are not included since they are risky: * Force IDE no probe: http://tree.celinuxforum.org/pubwiki/moin.cgi/IDENoProbe * Reduce IDE probe delay: http://groups.google.com/groups?q=g:thl2568235731d&selm=2nL5c-4tK-13%40gated-at.bofh.it - provide a tool to help configure a kernel for a given hardware, with just need modules. This is low priority since many know how to configure the kernel, but if done right could be very handy. It could make use of lspci, lsusb, /proc, /sys and stuff like that. This could be integrated to genkernel package as an auxiliar tool. What's need: - a project/formal group, either in gentoo, sourceforge or something that provide us some list, bugzilla, repository, ... - people interested. Reproducible: Always Steps to Reproduce:
Gustavo, are you interested in coding up some of this stuff?
Sure. I'll start ASAP. But some things are already done (see the cited bugs in this bugzilla). I'm specially interested in caching and optimizing the boot scripts, I'll start with them. My plan is to first produce a concatenation of every used boot script and check the possible conflicts, put just the start section, then I'll create a regular expression to vanish with comments and unused spaces (tricky due strings, but I see that python have a shell lexer ready), then play with parallel starts. Then I'll look at the whole script and ensure it can run with busybox, after that I'll start to time steps and possible fix some pipes and evaluation ("`command`" and "$(command)"). The dead code elimination will be left, since it's more tricky and there are not too many code that would benefit from that. Other possible, but really tricky, optimization is to cache evaluation results and other results. I mean, if I do $(cmd), put that return into a variable then use it later. The same with "[ -f $SOMEPATH/somefile ]", it could go to a variable and then be reused. But since some scripts alter the environment, it's tricky to do. A possible solution is to create some helper functions: - cached_test(): this will be a test ("[" command) but the value will be computed just once and then cached. - cached_eval(): this will be an evaluation ("`cmd`" or "$(cmd)") but the results will be cached too. This is quite easy given we implement a fast hash function, maybe in busy box itself. The tricky part is to identify which part shouldn't be cached. Should I post things here in this bug report or will you set up a working group?
What could be a rough estimate of the time reduction, relatively to the non-kernel part?
Ah, something I forgot to mention before: The easiest way to make linux boot faster without changing anything on distros (I mean, it's transparent) is to use software suspend with a checkpoint of the system at the desired point. It could be after the X or the window manager is up and running. This would be the optimum solution since Kernel does everything to us and it's fast. But it would require a separated resume partition, the swap could not be used since the system will write it during the system operation. What's missing to this work is a way to fall back when something fails (ie: format the resume partition) or the something changes in the mean time (some app changed) and recreate another checkpoint after the system is in a good state. This should be transparent to user. When to recreate the checkpoint is the problem here. Maybe it can be done using "FAM" on the cached files and/or integrated in portage, every installed package should clean the old checkpoint. But this doesn't eliminate the other optimizations I mentioned, since when there's no resume partition it would boot the old way and the swsusp doesn't always work until now. But it's a great area that should be worked as well (maybe later, when swsusp2 is in mainline 2.6).
Comment #3 From Gregorio Guidi 2004-10-31 10:54 PST: > What could be a rough estimate of the time reduction, relatively to > the non-kernel part? They (CELinux) claim to cut times in 50% using an optimized busybox 1.0, without parallel starts or other advanced techiniques discussed here. IMHO looking at RC scripts and ensure they can be used with busybox and remove the cruft is really easy. The hard part is to write a tool that refactory the code eliminating function, and stuff like that, making the whole thing small and consise and without redundant fork()s. But this not that difficult. Other parts that can help improve times is to have every need tool in a compressed rom filesystem, since the slowest part is to read things from HD. If things are already in memory the whole thing is really fast. Check the whole thing related to RC scripts at http://tree.celinuxforum.org/pubwiki/moin.cgi/OptimizeRCScripts, but be aware their focus is CE devices, disconsider the XIP (eXecute In Place).
Gus, for now post in here
notes for bash -> busybox transition: - from reading the reference (http://tree.celinuxforum.org/pubwiki/moin.cgi/OptimizeRCScripts) it seems that the optimizations are not in the stock-busybox. This should be taken into account. If both, busybox-ash and bash work with the init-process this is not too much of a problem. - some features available in bash are not in ash -- basically we'll need to transform the init-scripts to _plain_ bourne-shell. - as an interim-step moving not directly to busybox but to ash/dash may be worth a thought -- a while ago i did some testing with ash/dash for "dev-util/larch", not remembering/finding the exact numbers but the speedup was impressive.
About busybox and busybox optimized they talk about, they enchanced the number of possible built-ins, since they explain: "The built-ins are simply invoked as functions, and applets are invoked by means of the
About busybox and busybox optimized they talk about, they enchanced the number of possible built-ins, since they explain: "The built-ins are simply invoked as functions, and applets are invoked by means of the fork/exec system calls." Then they say: " - The set of shell commands and utilities is implemented as built-ins. - The invoked cat command at the beginning of pipes is eliminated and file descriptors are passed only into the next command of the pipe. " Now in consideration to features not in ash, if ash supports function I see not barrier in gentoo scripts. They're plain easy and AFAIK every command can be rewritten as a simpler sh one. Generally is just a matter of eliminating bash-ims, some gnu options and stuff like that. Talking about the speedup, I use bash with nls support and many stuff which are not need in boot... And imagine a 500KB binary that is already in memory and doesn't have to fork() for every command! It has to be fast! Now one new thing: one other possible optimization is to check if it's possible to save the intermediary representation, eliminating the parse phase... If we can reach this together with busybox (which have most commands as functions) it's almost a compiled shell script! :D
People at CK-kernel list informed that http://www.fefe.de/ have some work in this area and a paper about it. Worth reading.
Another thing that can be done is to start X early in the boot process. There is no reason why some daemons such as ssh, apache, postgresql, mysql and many others have to start before X.
We need to be carefull when to start X. I want a faster boot, while start X before other services like sshd can make believe it's started faster it can actually start slower. One thing that will be done is check every dependency of the startup scripts so you can launch in parallel.
Check bug 69854, I think it could be a dependency for this one. Any help will be appreciated.
Thanks Paul, It was added as a dependency. Any fix/patch so far?
Update on bash -> ash translation: I've started to do that and found that many gentoo scripts are really big and bloated. Also, as noticed in bug #69854, the parallel is not really parallel and most bloat come from this, how to solve dependencies and this kind of stuff. We (some friends and I) realized that doing this dependency stuff in shell is insane and more trouble than anything, so one friend is coding the stuff in C that will hopefully make it really parallel and eliminate the bloat from shell scripts.
I just filed bug 70689, can you add it to the dependencies? It does not speed up boot per se, but it gives me kdm login prompt 20 seconds sooner. Out of 2 minutes, so far I have: bug 55329 saves me 11 seconds bug 69854 saves me 20 seconds bug 70689 saves me 20 seconds so total I have reduced my boot time in 51 seconds, or 42% !!!! ( more to come stay tuned :) )
Not sure how relevant this is, but I did a small bit of benchmarking on my desktop system (parallel startup enabled, baselayout 1.11.6-r1, almost no boot services). I added a bit of code to /sbin/rc and /lib/rcscripts/sh/rc-services.sh to write timestamps to /dev/boottime (yes, silly, but the first thing mounted rw :). Unfortunately I didn't back up my patch or data, but still: - Most of my services started in 1 second or less. - The first "slow" one is /etc/init.d/clock. This calls /sbin/hwclock twice, which takes about 4 seconds total. It also is not run in parallel, holding everything up. If you have the hardware clock set to UTC and have something like ntp to keep the time synced, so you do not really need --adjust, it seems to be safe to comment those two /sbin/hwclock calls out, saving 4 seconds. - The second "slow" one is /etc/init/net.lo (and .eth0, same thing). This sources about 3333 lines of bash in /lib/rcscripts/net.modules.d/, most of which I didn't need. Removing unneeded modules from that dir saved a bunch of further seconds. This isn't much, but my boot was already pretty fast. Current time is 5 seconds from grub to init and about 10 from init to a login prompt (no X). Those 10 seconds were 15 before my modifications.
Please interested in this bug may want to check out http://www.klika.si/ziga/bootchart/
regarding #17 (sorry if Off topic) Is there an ebuild or easy way to instal bootchart on Gentoo?
I ported http://www.klika.si/ziga/bootchart/ to gentoo, sent the patches to Ziga and he merged them. To install, download from cvs, emerge baselayout 1.11.6-r1 and sysstat. Run install.sh, reboot, run /etc/bootchart/bootlog stop, put the logs in http://www.klika.si/ziga/bootchart/, save the png or svg. For your viewing pleasure, here are some charts I generated. You can view them by emerging librsvg and using rsvg-view. This is the default boot sequence: http://paulpach.no-ip.com/~paul/bootlogs/default/bootchart.log.svgz This is with RC_PARALLEL="yes" http://paulpach.no-ip.com/~paul/bootlogs/parallel/bootchart.log.svgz This is with bug 69854 http://paulpach.no-ip.com/~paul/bootlogs/paul_parallel/bootchart.log.svgz This is with bug 69854, bug 55329, bug 70689 http://paulpach.no-ip.com/~paul/bootlogs/paul_everything/bootchart.log.svgz This is with bug 69854, bug 55329, bug 70689, no coldplug and alsa in default instead of boot http://paulpach.no-ip.com/~paul/bootlogs/no-coldplug/bootchart.log.svgz I won't spoil it, you have to see the charts for yourselfs. Anyone has an idea of what to do next?
http://www.fooishbar.org/blog//tech/ubuntu/fastBootMiniBoF-2004-12-09-13-45.html has some nice info on what Ubuntu's been doing to speed up its boot. Code may show up at some point in the near future in its deb-src places.
Two things. One, how would this handle hardware detection? Would it require a ramdisk on startup? Secondly, concerning the "snapshot" idea in #4: it strikes me that the swap partition could indeed be used at startup. The boot process as I see it could be something like this: something at boot checks if the swap partition is formatted in the correct format to be a snapshot suspend image, and if so boots that snapshot immediately, and runs mkswap on the swap partition and addswap's it. If the swap is actually a swap partition, it goes to the regular bootup process and at a certain point (maybe before or just after X starts) it creates a snapshot and saves it somewhere in the filesystem. Then, on shutdown the last script simply dd's the snapshot over the swap partition. Since we're not talking about more than the total memory of the system, I think the time involved to mkswap and then dd at shutdown will be minimal. This has the advantage of being able to be used almost immediately on already-installed system (no new partition needed). Also, it automagically responds semi-correctly to a crashed system. If the machine crashed, it never got to write the snapshot over the swap and thus goes right to the default boot. This way if your machine crashes you automatically go into non-snapshot mode. The image should probably be recreated when runlevels change and other events.
In reply to Comment #21: The hardware detection should be done later if possible, or it should be faster, just a "compare with old table and check if something changed." About the swap, it's a good idea also, but copy +1Gb (I use 1Gb as swap) is too slow... I have a 160Gb HD, so I don't care to use extra gig to make it faster. Anyway, I like to see people in other distros are also addressing some problems, see link from Comment #20 as reference. Next week I should (re)start my work on the parallel launcher of init-scripts, then try to rewite initscripts in ash (busybox) instead of bash and let's see the improvements. Also, about hw detection, it's becoming faster... I don't have a link atm, but I read somewhere it was rewritten in C.
I've been thinking about something that might help improve bootup speed. I saw that my system starts netmount at start, though I don't use it. This causes that a script is run, (if I understand that right) reads from a file what to mount and does this. Quite logical, but not the fastest way. So the idea is: Create a tool that is run after editing a configfile that is able to build a script depending on the changes and by doing an rc-update adding or removing it. Like "crontab -e" does ... or that tool from SuSE (iirc). This would result in running-time optimized scripts, I think, since the values in /etc/conf.d/ would be in the scripts. If I'm wrong in my asumptions, correct me.
Netmount doesn't need to be slow if you don't use it, if it is, bugreport it. AFAIK, /usr and other dirs are expected to be non-local, that's why we need the netboot at early in the process. IMHO, it shouldn't be moved, since it can break many things, but it should be fixed and made fast.
I've been looking through different techniques described here and in bugs this one depends on. The parallel rc-scripts thing hasn't given me much - 1s speed increase or so, which in my case is ca. 2%. I have yet to look into the readahead stuff, but taking into consideration http://kerneltrap.org/node/2157 I'm not very optimistic about this. What I think has a real potential of dramatic boot speed improvements is the usage of software suspension. Some may call it cheating (it's not a real, full boot, after all), but if it lets us get from the bootloader to a login prompt in 5 secs I guess most people won't care too much :) I've done some preliminary tests with Software Suspend 2 (which in my experience is the fastest suspend system currently available for Linux) and got encouraging results. I have been doing stuff by hand so far, so I've yet to write the necessary scripts and patches for baselayout. The speedup in both bootup (ca. 650%) and shutdown/reboot (ca. 300%) is dramatic, and I seriously doubt we could get similar results with any modifications to the rc-scripts of even the programs that are started themselves (and even if we could, it would require a huge amount of work). However, I'm not sure what I've been doing here is what you had originally in mind. In all my tests I've simply exited X (the nvidia module has to be unloaded, or things get messy on my sys), made a snapshot and later used it to resume. Now, from what I understood from comment #4, you want to take a snapshot at some point during boot and then use it to resume. This is where things get tricky. A snapshot generated by any suspend system is not just a simple image with all running processes. It contains an image of the in-kernel data structs which are related to the state of the filesystems at the time the snapshot was taken. Generating a snapshot, then running the system and then using it to resume is a Bad Idea, and it will most likely result in a corrupted fs (I had a problem of this kind once, and it ended in --rebuild-tree - fortunately, no data was lost; and the system was running for only a minute after taking the snapshot!) So, if we want to take a snapshot during boot, we would have to somehow disentangle all fs-related data from it. This may not be impossible, but it certainly sounds difficult :)
A few questions: 1. If all scripts will be compiled into one script. Won't the gain from going to busybox ash be small, after all there's just one process of it then. 2. If the scripts will be compiled. Doesn't that remove the whole idea of having scripts in the first place? Wouldn't it be better to whip together a C-based library and have the boot "scripts" in c, a "boot-update" script would compile the scripts giving us the power of GCC to optimze the boot process. 3. Most of the time when I reboot a Linux box, it's because I have changed something about the boot process. Cheating with susped wouldn't help at all here. Besides, If you can use suspend you allready know you don't want to reboot. 4. Are we talking about optimizing the boot runlevel or should "default" also be optimized? 5. I've been playing with runit to replace sysvinit on my boxes. As this project seems to go over the whole boot process of Gentoo, I urge you: please make the choice of boot/daemon system a user choice (USE-flag?/profile?). runit is currently set up to run the boot runlevel just as sysvinit with "/sbin/rc sysinit" and "/sbin/rc boot" but instead of running "/sbin/rc default" it will run and manage "/etc/runit/runsvdir/default/*/run" there is nothing demanding that those "run" files must be shell scripts... it would be trivial to have a /etc/runit/Makefile to compile the boot system.
In reply to Comment #26: 1) No, since busybox ash + CELinux optimizations make some common commands built-in (they're functions ratter commands) and thus are faster (an asm call versus load another program and execute it). 2) I'm not sure where you saw about compiled scripts. I suggested it once but soon gone back since it would bring us too much trouble for nothing. 3) Many points were raised about cheating with software suspend (at least saving the initial state and using it again later, every boot). The major one and showstopper is the fact that it would corrupt filesystem and other memory structs... maybe a patch to avoid those structs could be done, but it's too much hack IMHO. 4) Every runlevel, IMHO. Worth noting that many distros contributed many optimizations to services that boot often and so the boot is becoming faster. Examples are: cups, xorg. Bug reporting to services you know take too much time to startup (without a good reason) is one way to help.
Hi! I just wanted to point you to initng (http://initng.thinktux.net/), a sysvinit replacement which also aims to reduce bootup speed by parallelizing startup of services. Best regards Christian
development/discussion threads such as this should happen on mailing lists, not bugzilla otherwise i dont really see anything here that needs to be done ... file specific bugs for specific issues
*** Bug 115583 has been marked as a duplicate of this bug. ***
Well, this may be off-topic but I've experienced amazing results when booting with runit. Everything starts in parallel, even X so one has good feeling "it's up quick". ...and you get service supervision as an extra value for free. :-)
assuming the process doesnt suck, it might not be a bad idea to document how to use different init's other than sysvinit we have runit/initng/daemontools for starters
