Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 64724

Summary: implement readahead(2) into userspace to speedup boot process
Product: Gentoo Linux Reporter: Elan Ruusamäe <glen>
Component: [OLD] baselayoutAssignee: Robin Johnson <robbat2>
Severity: enhancement CC: base-system,, bugz, bytewise, dberkholz, ernstp, flash3001, funtoos, gentoo.bugzilla, hiyuh.root, kensington, kkrizka, lanius, m.debruijne, masterdriverz, rockoo, spock, teutzz
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 69579    
Attachments: readahead.c, extracted from kernel-utils-2.4-9.1.131.src.rpm
/etc/init.d/readahead_early, add it to 'boot' runlevel.
/etc/init.d/readahead, add it to 'default' runlevel.
/etc/readahead.early.files, original from FC2
/etc/readahead.files, original from FC2
script that will strace(1) executed program for open and exec calls
script that will put multiple *.log files, produced by
script that will make filelist for all 'boot' services
/etc/readahead.files, for my workstation. could suit for you, and could not.
bootchart with readahead-list
bootchart without readahead-list

Description Elan Ruusamäe 2004-09-20 00:56:24 UTC
i posted to forum thread, that RedHat, Fedora Core 2 to be exact uses readahead(2) to speed up boot process.

so the idea is to have something similiar in gentoo too.

Reproducible: Always
Steps to Reproduce:
Comment 1 Elan Ruusamäe 2004-09-20 00:59:58 UTC
Created attachment 39996 [details]
readahead.c, extracted from kernel-utils-2.4-9.1.131.src.rpm

it has no copyrights, so i don't know what it's license is. probably same as
the whole package has, License: GPL/OSL.
Comment 2 Elan Ruusamäe 2004-09-20 01:03:20 UTC
Created attachment 39997 [details]
/etc/init.d/readahead_early, add it to 'boot' runlevel.

probably should adjust the dependencies, so that it gets invoked as very first
thing. (maybe it does, i haven't checked :))
Comment 3 Elan Ruusamäe 2004-09-20 01:05:42 UTC
Created attachment 39998 [details]
/etc/init.d/readahead, add it to 'default' runlevel.

similiar like, readahead_early, but readahead to memory progams that you will
most probably execute when you log in (kdm, kde, gnome, mozilla).
Comment 4 Elan Ruusamäe 2004-09-20 01:06:33 UTC
Created attachment 39999 [details]
/etc/readahead.early.files, original from FC2
Comment 5 Elan Ruusamäe 2004-09-20 01:07:02 UTC
Created attachment 40000 [details]
/etc/readahead.files, original from FC2
Comment 6 Elan Ruusamäe 2004-09-20 01:13:22 UTC
you could perhaps filter out filenames, that don't exist on your system:

# a=$(<readahead.early.files ); echo "$a" |xargs ls -1 2>/dev/null  > readahead.early.files

NOTE: You MUST run it twice, otherwise you will just truncate your filelist :)

also, you could grab the files being used by strace:
# strace -eopen -ff /etc/init.d/checkfs status 2>&1 | egrep -o 'open\([^)]+\)' |awk -F'"' '{print $2}'|sort -u

Comment 7 Elan Ruusamäe 2004-09-21 11:08:06 UTC
Created attachment 40108 [details]
script that will strace(1) executed program for open and exec calls

it also needs file called 'filter', what files to exclude from list:

Comment 8 Elan Ruusamäe 2004-09-21 11:10:39 UTC
Created attachment 40109 [details]
script that will put multiple *.log files, produced by

this script also needs patterns to do some filtering:

# require full path:
Comment 9 Elan Ruusamäe 2004-09-21 11:14:11 UTC
Created attachment 40110 [details]
script that will make filelist for all 'boot' services

it stopped working for me for some unknown reason. the straced progams open
/dev/vc/1 and keep waiting there. but it did work before...
Comment 10 Elan Ruusamäe 2004-09-21 11:15:52 UTC
Created attachment 40111 [details]
/etc/readahead.files, for my workstation. could suit for you, and could not.
Comment 11 Elan Ruusamäe 2004-09-21 11:18:01 UTC
Created attachment 40112 [details]
Comment 12 Elan Ruusamäe 2004-09-21 11:21:34 UTC
to test out the readahead setup:
1. compile readahead.c, install it /usr/sbin
2. create /etc/readahead.files and /etc/readahead.early.files
3. save /etc/init.d/00readahead and /etc/init.d/00readahead_early
4. add 00readahead_early to 'boot' runlevel (rc-update add 00readahead_early boot)
5. add 00readahead to 'default' runlevel (rc-update add 00readahead default)

measure time and listen your hdd activity :)
Comment 13 Gustavo Sverzut Barbieri 2004-10-31 00:41:02 UTC
Hello people,

I want to setup a group to work on gentoo boot process and try some techiniques to speed it up, making it the fastest boot around (it's already the fastest linux boot) and this bugreport will be one of the building blocks.

Check it out and join me:
Comment 14 Ernst Sjöstrand 2004-12-16 19:34:31 UTC
Less dangerous Python-script that you can run after a nice boot and put in readahead.files

import os

fd = os.popen('lsof -F n')

list = []

for line in fd:
	line = line[1:]
	line = line[:-1]
	if line[0] == '/' and line[-1] != '/' and line[:4] != '/dev' and line[:5] != '/proc' and line not in list:
		list.append( line )
		print line
Comment 15 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2005-01-17 20:33:20 UTC
here is an alternative implementation of readahead.c

and a possibly better way to track down which files to use readhead on:
Comment 16 Henk Poley 2005-02-01 04:42:42 UTC
Just for giggles I tried setting it up as shown in #12 and ran the python script from #14, so.. I could experience a boot time as "fast" as the Sun Ultra 5 workstations at the uni's. Whow it was slow. Probably the python script makes the file list too big to be loaded in 1GB RAM. After 11 minutes I stopped my stopwatch, had already logged in with gdm by that time, and went to eat some. When I came back readahead was still pounding against the diskcache, this was at around 38 minutes uptime..

So I would recommend against blindly using the list generated with the script from #14 :-P
Comment 17 Henk Poley 2005-02-01 05:41:04 UTC
Tried again, this time I removed the avi files from the list that were apparently open at the time of the lsof capture.. (several 600+ MB files.. doh)

Also removed all lines with "/home/" and "kde" in it from readahead.early.files, and the ones with "/tmp/" and "(deleted)" from both readahead lists.

Ran with and without readahead and also once without backgrounding the readahead task. As far as I can clock with my stopwatch only the non-backgrounded run improved boottime with about 10s. 

Next I should try getting the early readahead earlier in the bootprocess. IT currently seems to be one of the last ones started in the 'boot' runlevel. And maybe getting to run so I have more precise readings.
Comment 18 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2005-03-20 23:25:47 UTC
ok, I've assembled a source package that I'm maintaining upstream for this, based off the readahead-list.c from Erich Schubert that I linked to.
"emerge sys-apps/readahead-list" to get it.

I'd be very greatful for better scripts to generate the list of files for readahead to use. I suggest you consider using the linux accounting to build a list (hack accton very early into /sbin/rc)...

I've included scripts of my own design and used them to build the initial dataset I used.
Comment 19 Robert 2005-03-25 10:47:40 UTC
I've emerged readahead-list (on baselayout-1.9.4-r6) and put it into my boot runlevel.
It complains about a circular dependency "ibefore" in checkroot and readahead-list.
Other than that it doesn't seem to make a differenc, I've tested with bootchart (from Is there anything else I need to do to test this?
Comment 20 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2005-03-25 12:44:58 UTC
bytewise: could you please post up your bootchart graphic results with readahead-list?
Comment 21 Robert 2005-03-25 12:54:51 UTC
Created attachment 54466 [details]
bootchart with readahead-list

This is in inkscape format - hope that's ok. If not I can repost.
Note that the timeline seems to be off somehow, some of the processes start
before 0 secons.
Comment 22 Robert 2005-03-25 12:56:42 UTC
Created attachment 54467 [details]
bootchart without readahead-list

Just for comparison... here the timeline is correct again.
Comment 23 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2005-03-25 13:33:22 UTC
bytewise: thanks, the inkscape files are just svgz which is easy to view.
one thing I find strange, is that readahead-list itself isn't in your after view there.
Comment 24 Robert 2005-03-26 06:41:43 UTC
Well I have the impression it's not really running at all... Could you tell me something with respect to my comment #19?
Comment 25 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2005-04-28 23:04:54 UTC
Robert: sorry about the long response time.
Check out the new version, it should work fine.

The only thing that this bug now remains open for, is for somebody to provide better ways of building the initial lists.
Comment 26 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2006-03-28 21:38:27 UTC
bytewise: did the newer version work? could you post up newer bootchart output?

anybody have any requests to keep this bug open?
Comment 27 SpanKY gentoo-dev 2006-07-31 21:14:10 UTC
*** Bug 142338 has been marked as a duplicate of this bug. ***
Comment 28 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2006-11-14 03:35:42 UTC
no response from any users on their graphs or custom readahead input, so I assume there are no problems, and I can close this bug now.