Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 55329 - don't do modules-update on every boot, just when needed
Summary: don't do modules-update on every boot, just when needed
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] baselayout (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Gentoo's Team for Core System packages
: 100342 (view as bug list)
Depends on:
Blocks: 69579
  Show dependency tree
Reported: 2004-06-27 07:39 UTC by Paul Pacheco
Modified: 2005-07-26 03:07 UTC (History)
22 users (show)

See Also:
Package list:
Runtime testing required: ---

recalculate module dependency if something changes (fastermodules.patch,737 bytes, patch)
2004-06-27 07:42 UTC, Paul Pacheco
Details | Diff
New version not using find (patch2,723 bytes, patch)
2004-08-17 14:44 UTC, Paul Pacheco
Details | Diff
patch for modules-update (modules-update.patch,6.14 KB, patch)
2004-09-11 08:05 UTC, Paul Pacheco
Details | Diff
New modules-update.patch (modules-update.patch,6.67 KB, patch)
2004-09-12 05:25 UTC, Paul Pacheco
Details | Diff
bootmisc-localmount-speed.patch (bootmisc-localmount-speed.patch,892 bytes, patch)
2004-09-26 07:38 UTC, Heinrich Wendel (RETIRED)
Details | Diff
Refactor dependency checking into a function (modules-update.patch,7.13 KB, patch)
2004-10-27 08:05 UTC, Paul Pacheco
Details | Diff
bootmisc.patch (bootmisc.patch,450 bytes, patch)
2004-11-04 10:48 UTC, Paul Pacheco
Details | Diff
modules-update.patch (modules-update.patch,7.48 KB, patch)
2004-11-11 07:50 UTC, Paul Pacheco
Details | Diff
modules-update.patch (modules-update.patch,7.86 KB, patch)
2004-11-17 06:17 UTC, Paul Pacheco
Details | Diff
modules-update.patch (modules-update.patch,7.08 KB, patch)
2004-12-04 13:06 UTC, Paul Pacheco
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Pacheco 2004-06-27 07:39:34 UTC
modules-update takes a long time and it is being called every time you boot.
We only need to run this when something in /etc/modules.d change.

Here is a patch that does exactly that. It saves 9 seconds of boot time on my machine.

Reproducible: Always
Steps to Reproduce:
1. reboot system

Actual Results:  
calculating module dependencies takes a long time, and is totally unnecesary 

Expected Results:  
calculating module dependencies should be executed only when something 
in /etc/modules.d changes
Comment 1 Paul Pacheco 2004-06-27 07:42:02 UTC
Created attachment 34274 [details, diff]
recalculate module dependency if something changes

apply this patch to /etc/init.d/modules and enjoy faster boot time.
Comment 2 Norberto Bensa 2004-06-28 12:06:49 UTC
Didn't make any difference here :-/
Comment 3 Paul Pacheco 2004-07-08 09:41:40 UTC
Did you see the message:

"Module dependencies are up-to-date - do not recalculating"?
(sorry for the grammar error).

when booting?
normally it says:

"Calculating module dependencies"
and it stays there some good 9 seconds.

Try running the following command:

/usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf

It should give you nothing. If it prints any file, then either you changed something in modules.d since you last rebooted, or there is a file with a modification time in the future. Do a ls -l in that file to see the date and put it here. Also, run modules-update, check if you have any errors.

Comment 4 Norberto Bensa 2004-07-08 09:56:58 UTC
Yes, I know the logic behind your patch. But it seems something is changing one of the files in /etc/modules.d

nbensa@venkman:~$ /usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf
nbensa@venkman:~$ ll /etc/modules.conf
-rw-r--r--  1 root root 4297 Jul  5 12:37 /etc/modules.conf
nbensa@venkman:~$ ll /etc/modules.d
total 28
-rw-r--r--  1 root root   19 Feb 27 15:55 8139too
-rw-r--r--  1 root root 1330 Jun 29 22:32 aliases
-rw-r--r--  1 root root 1080 Feb 29 15:30 alsa
-rw-r--r--  1 root root  152 May  4 01:16 creative-webcam
-rw-r--r--  1 root root  122 Jun 29 22:32 i386
-rw-r--r--  1 root root  479 Jul  5 12:35 nvidia
-rw-r--r--  1 root root   36 Apr  2  2003 svgalib
nbensa@venkman:~$ uptime
 13:55:35 up 3 days,  1:18,  1 user,  load average: 0.35, 0.22, 0.20
nbensa@venkman:~$ date
Thu Jul  8 13:55:46 ART 2004

I still don't know WHY nvidia changes its modified time :-/
Comment 5 Paul Pacheco 2004-07-08 13:53:27 UTC
I see.

Does it get modified before you reboot or after? Try booting from the livecd, mounting the partition and see if it is modified.

if it has a newer date than modules.conf, it means it is modified when the system is shooting down.

If it has an older date than modules.conf, it means it is modified when the system boots up.

what package does nvidia belong to:
qpkg -I -fp /etc/modules.d/nvidia

maybe grep the init.d files to see if any of them touch the nvidia file:
cd /etc/init.d
grep -r "nvidia" .

and maybe we can try to find out why it modifies it and if it is necesary.
Comment 6 Norberto Bensa 2004-07-09 00:48:35 UTC
I'm an idiot, it was modified on shutdown. I upgraded my kernel together with nvidia-kernel (the package that owns /etc/modules.d/nvidia) *BUT*  after many reboots, it still doesn't work as expected:

	if [ -z "${CDBOOT}" ]
	    # do we need to recalculate module dependencies?
	    if [ ! -x /usr/bin/find ] || [ -n "`/usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf`" ]

ALWAYS comes here and I don't know why. The logic says it should go to "Modules dependencies up-to-date" but it doesn't.

	        ebegin "Calculating module dependencies"
	        /sbin/modules-update &>/dev/null
	        eend $? "Failed to calculate dependencies"
	        einfo "Module dependencies are up-to-date - do not recalculating."
Comment 7 Paul Pacheco 2004-07-09 10:42:12 UTC
If it is not the nvidia stuff, can you figure out why the find gives something and what does it give?

try this:
        if [ -z "${CDBOOT}" ]
            # do we need to recalculate module dependencies?
            if [ ! -x /usr/bin/find ] || [ -n "`/usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf`" ]

		echo "the files that are new are"
		/usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf -exec ls -l {} \;

		echo "showing modules.conf"
		ls -l /etc/modules.conf

                ebegin "Calculating module dependencies"
                /sbin/modules-update &>/dev/null
                eend $? "Failed to calculate dependencies"
                einfo "Module dependencies are up-to-date."
Comment 8 Norberto Bensa 2004-07-09 11:06:21 UTC
Got it.

find is in /usr. I have /usr as a separated partition. It's mounted AFTER /etc/init.d/modules. Hmm. I guess your patch will not work for many people :-(
Comment 9 Paul Pacheco 2004-08-17 06:56:22 UTC
Any issue with this patch? any reason why is hasn't gone in? I'll fix it if it needs fixing.
Comment 10 Norberto Bensa 2004-08-17 07:57:03 UTC
The issue: comment #8

Comment 11 Paul Pacheco 2004-08-17 13:59:19 UTC
Well, the worst than can happen is #8, that is, it will keep working the way it works today.

It will only make a difference for people who have /usr in the same partition, and the difference being gentoo boots faster.

So nothing breaks for anyone, and for most people it is faster. So why is #8 a reason not to apply the patch?
Comment 12 Paul Pacheco 2004-08-17 14:12:02 UTC
And the patch correctly identifies when find can not be found, and just reverts to the current behavior, so if /usr is not mounted, the patch works flawlessly.
Comment 13 Paul Pacheco 2004-08-17 14:44:08 UTC
Created attachment 37630 [details, diff]
New version not using find

I did not think #8 was a reason not to apply the patch. 
But since it is, here is another patch that avoids using find and anything in

This approach just uses bash builtins to find if there is a newer file in
/etc/modules.d than /etc/modules.conf

The only problem with this is that I assume there aren't subdirectories in
/etc/modules.d. The first patch didn't have that problem. 
Is that an issue?
Comment 14 Norberto Bensa 2004-08-17 18:33:23 UTC
I don't think so. Or is it supposed /etc/modules.d to have subdirs?

I haven't tested your new patch yet, but I'll do ASAP.

Thank you :)

Comment 15 Norberto Bensa 2004-08-18 11:33:12 UTC
Yup. It works, and it's fast!
Comment 16 Stefan Huszics 2004-08-27 23:01:00 UTC
C'mon devs, don't leave this patch hanging for no good reason. It cuts of about 15 sec on my GF's old K6-2 500 bootup so it's seems like a nobrainer to me to add this. :)
Comment 17 Norberto Bensa 2004-08-27 23:20:59 UTC
Doesn't really work. I need to "depmod -ae" with every kernel upgrade :(
Comment 18 Stefan Huszics 2004-08-27 23:40:06 UTC
> I need to "depmod -ae" with every kernel upgrade :(

Havn't tried upgrading the kernel yet, but what version of kernel & depmod are you running anyway? Depmod for kernels 2.5.48 and above doesn't even have the -a option (according to the manpage). Could running an 2.4(?) kernel and/or old depmod have anything to do with your problems?
Comment 19 Norberto Bensa 2004-08-27 23:57:08 UTC
I didn't '-a' was removed. Anyway:

$ uname -a
Linux venkman 2.6.9-rc1-mm1 #4 Fri Aug 27 16:42:36 ART 2004 i686 Pentium III (Coppermine)GenuineIntel GNU/Linux

$ qpkg -I -v modul
sys-apps/module-init-tools-3.0-r2 *

Comment 20 Paul Pacheco 2004-08-31 07:55:11 UTC
Digging deeper, I found that the botle neck is running /sbin/generate-modprobe.conf.

This script takes /etc/modules.conf and generates /etc/modproble.conf.

Running depmod does not take any noticeable time.

So there are a few posible approaches:
1) modify modules-update to avoid running generate-modprobe.conf unless something in /etc/modules.d changes.

2) splitting modules-update and taking depmod out of there so that modules-update can be skipped and depmod called.

I like 2 better, but I would like to hear from you before I do any coding.
Comment 21 Sven Wegener gentoo-dev 2004-09-02 20:39:23 UTC
Re: Comment #20

I would prefer to modify modules-update to only update /etc/modules.conf if a
file in /etc/modules.d is newer than /etc/modules.conf. And only call
generate-modprobe.conf if /etc/modules.conf is newer then /etc/modprobe.conf.
And only run depmod if /etc/modprobe.conf is newer than /lib/modules/$(uname -r)/modules.dep.

Optional support via the force option that is already in modules-update to
force updating all files.

This way we have a full depency, can leave modules-update in /etc/init.d/modules
and the benefit of gaining some speed is not only at bootup time, but als when
running modules-update while the system is running.

Comment 22 Paul Pacheco 2004-09-03 06:11:16 UTC
Sounds good to me.

I will do this some time this week.
Comment 23 Paul Pacheco 2004-09-11 08:05:40 UTC
Created attachment 39398 [details, diff]
patch for modules-update

This is a patch for modules-update implementing Comment #21

I have left some messages like "Generating xxx" so it is easy for you to test
when things are being generated. Once you approve of the patch, I will remove
those messages.
Comment 24 Paul Pacheco 2004-09-12 05:25:51 UTC
Created attachment 39435 [details, diff]
New modules-update.patch

This improved patch avoids making a .old file unless a new file is generated.
The previous version allways did a .old even if the file was not touched.
Comment 25 Paul Pacheco 2004-09-17 08:49:52 UTC
Have you had time to try the last patch? 
does it have any problem?
Is there anything I should do?
Comment 26 Sven Wegener gentoo-dev 2004-09-19 06:23:50 UTC
I'm currently looking into some other open bugs that have to do with modules support.
Bug 48736, 51371, 59188

So far the patch looks good, works as expected.
Comment 27 Heinrich Wendel (RETIRED) gentoo-dev 2004-09-26 07:24:42 UTC
works fine for me as well.

azarah: what about including it in baselayout?
Comment 28 Heinrich Wendel (RETIRED) gentoo-dev 2004-09-26 07:37:51 UTC
to speed up boot further, env-update should only be called if there are changes in /etc/env.d and mount should mount the filesystems parallel, the attached patch fixes that
Comment 29 Heinrich Wendel (RETIRED) gentoo-dev 2004-09-26 07:38:10 UTC
Created attachment 40449 [details, diff]
Comment 30 Stefan Huszics 2004-09-26 07:54:33 UTC
Um, somewhere in the back of my head there is a warningbell going off about mounting in parallell. Eg what happens if in your fstab you have eg 

/dev/hdb1 /opt
/dev/hdb2 /opt/whatever

Since in parallell what finishes in which order is pretty random, isn't there a very high risk of mounts failing? A marginal bootup speedincrease that potentially breaks your system seems like a bad idea to me.
Comment 31 Paul Pacheco 2004-09-27 06:45:28 UTC
from the mount man page:
-F     (Used  in  conjunction with -a.)  Fork off a new incarnation of mount for each device.
              This will do the mounts on different devices or different  NFS  servers  in  parallel.
              This  has  the advantage that it is faster; also NFS timeouts go in parallel. A disad-
              vantage is that the mounts are done in undefined order.  Thus,  you  cannot  use  this
              option if you want to mount both /usr and /usr/spool.

so the /usr thing would break many installations.

Also the if [ /etc/env.d -nt /etc/profile.env ]; then does not work because if you touch one file inside /etc/env.d, the directory itself does not change.
Try this:

touch 05gcc
if [ /etc/env.d -nt /etc/profile.env ] ; then echo "The directory was modified" ; fi

Check my last patch to see what I did with /etc/init.d which is similar.
Comment 32 Paul Pacheco 2004-09-27 06:50:01 UTC
I meant to say what I did with /etc/modules.d

Comment 33 Paul Pacheco 2004-10-11 07:47:57 UTC
There is no apparent movement on the other bugs, and some of them conflict with this one, giving one extra reason why this should be merged before work starts on them.

Is there any reason why the modules-update patch has not been merged? What can I do to get it merged?

Comment 34 Heinrich Wendel (RETIRED) gentoo-dev 2004-10-25 08:25:37 UTC
touch 05gcc
if [ /etc/env.d -nt /etc/profile.env ] ; then echo "The directory was modified" ; fi

that works for me, i'm using reiser3
Comment 35 Norberto Bensa 2004-10-25 08:47:46 UTC
Doesn't for me. Reiser3.

$ sudo /sbin/
$ sudo touch /etc/env.d/05gcc
$ if [ /etc/env.d -nt /etc/profile.env ] ; then echo "The directory was modified" ; fi
Comment 36 Heinrich Wendel (RETIRED) gentoo-dev 2004-10-25 09:11:23 UTC
that does not work for me as well, because 05gcc already exists, so you just do nothing, but if you try it with 04gcc it should work
Comment 37 Paul Pacheco 2004-10-25 09:38:34 UTC
Given that the stuff should also run when any file changes and not just when a new file is created, you should also check for that.

The modules-update.patch seems to work for everybody. Is there any reason it has not being merged?
Comment 38 Heinrich Wendel (RETIRED) gentoo-dev 2004-10-25 10:10:23 UTC
it's looks very complicated/difficult

when a file in the directory changes the directory timestap changes as well, just try it out
Comment 39 Paul Pacheco 2004-10-25 14:31:49 UTC
Using reiserfs3

Cut and pasted straight out of my terminal:

$ ls -ld env.d
drwxr-xr-x  4 root root 1008 Oct 22 08:56 env.d
$ touch env.d/05gcc
$ ls -ld env.d
drwxr-xr-x  4 root root 1008 Oct 22 08:56 env.d
$ touch env.d/pepe
$ ls -ld env.d
drwxr-xr-x  4 root root 1032 Oct 25 16:31 env.d

So the modification time does not change unless a new file is created. The stuff should work even when an existing file is changed and no new file is created.
Comment 40 Heinrich Wendel (RETIRED) gentoo-dev 2004-10-25 14:40:39 UTC
ok, conviced, it seems that my vim was causing the directory timestamp to change
Comment 41 Paul Pacheco 2004-10-27 06:58:59 UTC
On my first version, I used find and it was much simpler. in #8, Norberto pointed out that find is not allways available at that time. That is why I had to do the bash sourcery to find if any file has changed. 

If you have a simpler way to do it, I would like to know.
Comment 42 Paul Pacheco 2004-10-27 08:05:28 UTC
Created attachment 42697 [details, diff]
Refactor dependency checking into a function

Prettier version. Now dependency checking is factored inside a function
"dependencyChanged" that can be reused for any .d type folder.

You should be able to use the function for env.d like this:

if dependencyChanged /etc/profile.env /etc/env.d /etc/env.d/* ; then

please let me know if there is anything I can improve on.

To apply the patch:

cd /sbin
patch -p0 < /tmp/modules-update.patch

It should patch modules-update and
Comment 43 Gustavo Sverzut Barbieri 2004-10-31 00:40:46 UTC
Hello people,

I want to setup a group to work on gentoo boot process and try some techiniques to speed it up, making it the fastest boot around (it's already the fastest linux boot) and this bugreport will be one of the building blocks.

Check it out and join me:
Comment 44 Paul Pacheco 2004-11-01 06:02:59 UTC
Please devs, give me some feed back on this.
Comment 45 Gustavo Sverzut Barbieri 2004-11-02 08:41:18 UTC
Some things about the bootmisc-localmount-speed.patch:

 - Is the mount -aF safe? It seems not. Maybe it deserves a patch to 'mount' so it can be made safe and everyone benefits from that?

 - if [ /etc/env.d -nt /etc/profile.env ]... isnt' that supposed to be changed? You said it didn't work and would use dependencyChanged instead.

Also, if we can use busybox (#69579) we can compile it with 'find' support, so easy (and maybe speed) this.
Comment 46 Paul Pacheco 2004-11-02 09:16:59 UTC
The bootmisc-localmount-speed.patch was submited by Heinrich Wendel in comment #29, therefore is his baby and I did not want to touch it. I agree it has problems with the mount and I suggested to use dependencyChanged for fixing the env.d issue.

What I would like feedback on is on the other patch, the one named "Refactor dependency checking into a function". Which is the one I made and the one I would like to get accepted.

Heinrich, I will fix your patch if that is ok with you and you are not planing on doing it yourself.
Comment 47 Paul Pacheco 2004-11-04 10:48:13 UTC
Created attachment 43295 [details, diff]

This one fixes the issues with bootmisc-localmount-speed.patch 

I don't do anything with mount and the directory env.d is properly checked for
changes. It depends on "Refactor dependency checking into a function"
Comment 48 SpanKY gentoo-dev 2004-11-04 11:07:08 UTC
this bug is just about modules-update, stop posting unrelated (and obselete) patches
Comment 49 Paul Pacheco 2004-11-11 07:50:02 UTC
Created attachment 43725 [details, diff]

This one is the same as "Refactor dependency checking into a function" Except
that it applies to baselayout-1.11.6-r1 and removes a touch in
/etc/init.d/modules which kills the benefits of this patch.

To apply, save in /tmp/modules-update.patch and apply

cd /
patch -p0 -b < /tmp/modules-update.patch
Comment 50 Paul Pacheco 2004-11-17 06:17:30 UTC
Created attachment 44151 [details, diff]

Report actions with ebegin and eend.

Also, dependencyChanged is now recursive
Comment 51 Kathy Wills 2004-11-17 18:18:31 UTC
The patch did not apply cleanly. I had to patch by hand based on the modules.rej file.
Comment 52 Paul Pacheco 2004-11-18 06:28:16 UTC
Did you have a current /etc/init.d/modules? Did you do etc-update when you last upgraded baselayout?
what version of baselayout did you have? The patch is made against baselayout-1.11.6-r1 which is marked ~x86.

Can you post your modules.rej?
Comment 53 Paul Pacheco 2004-12-04 13:06:49 UTC
Created attachment 45288 [details, diff]

This is the same thing but applied to baselayout-1.11.7-r1
Comment 54 Heinrich Wendel (RETIRED) gentoo-dev 2004-12-29 10:37:08 UTC
base-system: i use this patch since it's available and it works fine, any chance to include it soon?
Comment 55 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2005-01-23 22:39:56 UTC
agriffis: could this please get in the next version of baselayout?
It massively helps boot times on slow machines (30 second improvement for me on an NFSroot box).
Comment 56 Heinrich Wendel (RETIRED) gentoo-dev 2005-05-16 15:30:03 UTC
uberlord: what about this?
Comment 57 Roy Marples (RETIRED) gentoo-dev 2005-05-19 08:30:47 UTC
I've merged this into baselayout CVS

Will be in baselayout-1.12.0-alpha3
Comment 58 SpanKY gentoo-dev 2005-07-14 07:06:38 UTC
baselayout-1.12.x has this, no plans currently to merge it back into

for those who want it, you should be able to just copy the modules init.d 
script and sbin modules-update script from baselayout-1.12.x into your
baselayout-1.11.x system, but no guarantees ;p
Comment 59 Sven Wegener gentoo-dev 2005-07-26 03:07:56 UTC
*** Bug 100342 has been marked as a duplicate of this bug. ***