modules-update takes a long time and it is being called every time you boot. We only need to run this when something in /etc/modules.d change. Here is a patch that does exactly that. It saves 9 seconds of boot time on my machine. Reproducible: Always Steps to Reproduce: 1. reboot system Actual Results: calculating module dependencies takes a long time, and is totally unnecesary Expected Results: calculating module dependencies should be executed only when something in /etc/modules.d changes
Created attachment 34274 [details, diff] recalculate module dependency if something changes apply this patch to /etc/init.d/modules and enjoy faster boot time.
Didn't make any difference here :-/
Did you see the message: "Module dependencies are up-to-date - do not recalculating"? (sorry for the grammar error). when booting? normally it says: "Calculating module dependencies" and it stays there some good 9 seconds. Try running the following command: /usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf It should give you nothing. If it prints any file, then either you changed something in modules.d since you last rebooted, or there is a file with a modification time in the future. Do a ls -l in that file to see the date and put it here. Also, run modules-update, check if you have any errors.
Yes, I know the logic behind your patch. But it seems something is changing one of the files in /etc/modules.d nbensa@venkman:~$ /usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf nbensa@venkman:~$ ll /etc/modules.conf -rw-r--r-- 1 root root 4297 Jul 5 12:37 /etc/modules.conf nbensa@venkman:~$ ll /etc/modules.d total 28 -rw-r--r-- 1 root root 19 Feb 27 15:55 8139too -rw-r--r-- 1 root root 1330 Jun 29 22:32 aliases -rw-r--r-- 1 root root 1080 Feb 29 15:30 alsa -rw-r--r-- 1 root root 152 May 4 01:16 creative-webcam -rw-r--r-- 1 root root 122 Jun 29 22:32 i386 -rw-r--r-- 1 root root 479 Jul 5 12:35 nvidia -rw-r--r-- 1 root root 36 Apr 2 2003 svgalib nbensa@venkman:~$ uptime 13:55:35 up 3 days, 1:18, 1 user, load average: 0.35, 0.22, 0.20 nbensa@venkman:~$ date Thu Jul 8 13:55:46 ART 2004 I still don't know WHY nvidia changes its modified time :-/
I see. Does it get modified before you reboot or after? Try booting from the livecd, mounting the partition and see if it is modified. if it has a newer date than modules.conf, it means it is modified when the system is shooting down. If it has an older date than modules.conf, it means it is modified when the system boots up. what package does nvidia belong to: qpkg -I -fp /etc/modules.d/nvidia maybe grep the init.d files to see if any of them touch the nvidia file: cd /etc/init.d grep -r "nvidia" . and maybe we can try to find out why it modifies it and if it is necesary.
I'm an idiot, it was modified on shutdown. I upgraded my kernel together with nvidia-kernel (the package that owns /etc/modules.d/nvidia) *BUT* after many reboots, it still doesn't work as expected: if [ -z "${CDBOOT}" ] then # do we need to recalculate module dependencies? if [ ! -x /usr/bin/find ] || [ -n "`/usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf`" ] then ALWAYS comes here and I don't know why. The logic says it should go to "Modules dependencies up-to-date" but it doesn't. ebegin "Calculating module dependencies" /sbin/modules-update &>/dev/null eend $? "Failed to calculate dependencies" else einfo "Module dependencies are up-to-date - do not recalculating." fi fi
If it is not the nvidia stuff, can you figure out why the find gives something and what does it give? try this: if [ -z "${CDBOOT}" ] then # do we need to recalculate module dependencies? if [ ! -x /usr/bin/find ] || [ -n "`/usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf`" ] then echo "the files that are new are" /usr/bin/find /etc/modules.d -type f -newer /etc/modules.conf -exec ls -l {} \; echo "showing modules.conf" ls -l /etc/modules.conf ebegin "Calculating module dependencies" /sbin/modules-update &>/dev/null eend $? "Failed to calculate dependencies" else einfo "Module dependencies are up-to-date." fi fi
Got it. find is in /usr. I have /usr as a separated partition. It's mounted AFTER /etc/init.d/modules. Hmm. I guess your patch will not work for many people :-(
Any issue with this patch? any reason why is hasn't gone in? I'll fix it if it needs fixing.
The issue: comment #8
Well, the worst than can happen is #8, that is, it will keep working the way it works today. It will only make a difference for people who have /usr in the same partition, and the difference being gentoo boots faster. So nothing breaks for anyone, and for most people it is faster. So why is #8 a reason not to apply the patch?
And the patch correctly identifies when find can not be found, and just reverts to the current behavior, so if /usr is not mounted, the patch works flawlessly.
Created attachment 37630 [details, diff] New version not using find I did not think #8 was a reason not to apply the patch. But since it is, here is another patch that avoids using find and anything in /usr. This approach just uses bash builtins to find if there is a newer file in /etc/modules.d than /etc/modules.conf The only problem with this is that I assume there aren't subdirectories in /etc/modules.d. The first patch didn't have that problem. Is that an issue?
I don't think so. Or is it supposed /etc/modules.d to have subdirs? I haven't tested your new patch yet, but I'll do ASAP. Thank you :)
Yup. It works, and it's fast!
C'mon devs, don't leave this patch hanging for no good reason. It cuts of about 15 sec on my GF's old K6-2 500 bootup so it's seems like a nobrainer to me to add this. :)
Doesn't really work. I need to "depmod -ae" with every kernel upgrade :(
> I need to "depmod -ae" with every kernel upgrade :( Havn't tried upgrading the kernel yet, but what version of kernel & depmod are you running anyway? Depmod for kernels 2.5.48 and above doesn't even have the -a option (according to the manpage). Could running an 2.4(?) kernel and/or old depmod have anything to do with your problems?
I didn't '-a' was removed. Anyway: $ uname -a Linux venkman 2.6.9-rc1-mm1 #4 Fri Aug 27 16:42:36 ART 2004 i686 Pentium III (Coppermine)GenuineIntel GNU/Linux $ qpkg -I -v modul sys-apps/module-init-tools-3.0-r2 *
Digging deeper, I found that the botle neck is running /sbin/generate-modprobe.conf. This script takes /etc/modules.conf and generates /etc/modproble.conf. Running depmod does not take any noticeable time. So there are a few posible approaches: 1) modify modules-update to avoid running generate-modprobe.conf unless something in /etc/modules.d changes. 2) splitting modules-update and taking depmod out of there so that modules-update can be skipped and depmod called. I like 2 better, but I would like to hear from you before I do any coding.
Re: Comment #20 I would prefer to modify modules-update to only update /etc/modules.conf if a file in /etc/modules.d is newer than /etc/modules.conf. And only call generate-modprobe.conf if /etc/modules.conf is newer then /etc/modprobe.conf. And only run depmod if /etc/modprobe.conf is newer than /lib/modules/$(uname -r)/modules.dep. Optional support via the force option that is already in modules-update to force updating all files. This way we have a full depency, can leave modules-update in /etc/init.d/modules and the benefit of gaining some speed is not only at bootup time, but als when running modules-update while the system is running. Comments?
Sounds good to me. I will do this some time this week.
Created attachment 39398 [details, diff] patch for modules-update This is a patch for modules-update implementing Comment #21 I have left some messages like "Generating xxx" so it is easy for you to test when things are being generated. Once you approve of the patch, I will remove those messages.
Created attachment 39435 [details, diff] New modules-update.patch This improved patch avoids making a .old file unless a new file is generated. The previous version allways did a .old even if the file was not touched.
Have you had time to try the last patch? does it have any problem? Is there anything I should do?
I'm currently looking into some other open bugs that have to do with modules support. Bug 48736, 51371, 59188 So far the patch looks good, works as expected.
works fine for me as well. azarah: what about including it in baselayout?
to speed up boot further, env-update should only be called if there are changes in /etc/env.d and mount should mount the filesystems parallel, the attached patch fixes that
Created attachment 40449 [details, diff] bootmisc-localmount-speed.patch
Um, somewhere in the back of my head there is a warningbell going off about mounting in parallell. Eg what happens if in your fstab you have eg /dev/hdb1 /opt /dev/hdb2 /opt/whatever Since in parallell what finishes in which order is pretty random, isn't there a very high risk of mounts failing? A marginal bootup speedincrease that potentially breaks your system seems like a bad idea to me.
from the mount man page: -F (Used in conjunction with -a.) Fork off a new incarnation of mount for each device. This will do the mounts on different devices or different NFS servers in parallel. This has the advantage that it is faster; also NFS timeouts go in parallel. A disad- vantage is that the mounts are done in undefined order. Thus, you cannot use this option if you want to mount both /usr and /usr/spool. so the /usr thing would break many installations. Also the if [ /etc/env.d -nt /etc/profile.env ]; then does not work because if you touch one file inside /etc/env.d, the directory itself does not change. Try this: env-update touch 05gcc if [ /etc/env.d -nt /etc/profile.env ] ; then echo "The directory was modified" ; fi Check my last patch to see what I did with /etc/init.d which is similar.
I meant to say what I did with /etc/modules.d
There is no apparent movement on the other bugs, and some of them conflict with this one, giving one extra reason why this should be merged before work starts on them. Is there any reason why the modules-update patch has not been merged? What can I do to get it merged?
============== env-update touch 05gcc if [ /etc/env.d -nt /etc/profile.env ] ; then echo "The directory was modified" ; fi ============== that works for me, i'm using reiser3
Doesn't for me. Reiser3. $ sudo /sbin/env-update.sh Password: $ sudo touch /etc/env.d/05gcc $ if [ /etc/env.d -nt /etc/profile.env ] ; then echo "The directory was modified" ; fi $
that does not work for me as well, because 05gcc already exists, so you just do nothing, but if you try it with 04gcc it should work
Given that the stuff should also run when any file changes and not just when a new file is created, you should also check for that. The modules-update.patch seems to work for everybody. Is there any reason it has not being merged?
it's looks very complicated/difficult when a file in the directory changes the directory timestap changes as well, just try it out
Using reiserfs3 Cut and pasted straight out of my terminal: $ ls -ld env.d drwxr-xr-x 4 root root 1008 Oct 22 08:56 env.d $ touch env.d/05gcc $ ls -ld env.d drwxr-xr-x 4 root root 1008 Oct 22 08:56 env.d $ touch env.d/pepe $ ls -ld env.d drwxr-xr-x 4 root root 1032 Oct 25 16:31 env.d So the modification time does not change unless a new file is created. The stuff should work even when an existing file is changed and no new file is created.
ok, conviced, it seems that my vim was causing the directory timestamp to change
On my first version, I used find and it was much simpler. in #8, Norberto pointed out that find is not allways available at that time. That is why I had to do the bash sourcery to find if any file has changed. If you have a simpler way to do it, I would like to know.
Created attachment 42697 [details, diff] Refactor dependency checking into a function Prettier version. Now dependency checking is factored inside a function "dependencyChanged" that can be reused for any .d type folder. You should be able to use the function for env.d like this: if dependencyChanged /etc/profile.env /etc/env.d /etc/env.d/* ; then ... fi please let me know if there is anything I can improve on. To apply the patch: cd /sbin patch -p0 < /tmp/modules-update.patch It should patch modules-update and functions.sh
Hello people, I want to setup a group to work on gentoo boot process and try some techiniques to speed it up, making it the fastest boot around (it's already the fastest linux boot) and this bugreport will be one of the building blocks. Check it out and join me: http://bugs.gentoo.org/show_bug.cgi?id=69579
Please devs, give me some feed back on this.
Some things about the bootmisc-localmount-speed.patch: - Is the mount -aF safe? It seems not. Maybe it deserves a patch to 'mount' so it can be made safe and everyone benefits from that? - if [ /etc/env.d -nt /etc/profile.env ]... isnt' that supposed to be changed? You said it didn't work and would use dependencyChanged instead. Also, if we can use busybox (#69579) we can compile it with 'find' support, so easy (and maybe speed) this.
The bootmisc-localmount-speed.patch was submited by Heinrich Wendel in comment #29, therefore is his baby and I did not want to touch it. I agree it has problems with the mount and I suggested to use dependencyChanged for fixing the env.d issue. What I would like feedback on is on the other patch, the one named "Refactor dependency checking into a function". Which is the one I made and the one I would like to get accepted. Heinrich, I will fix your patch if that is ok with you and you are not planing on doing it yourself.
Created attachment 43295 [details, diff] bootmisc.patch This one fixes the issues with bootmisc-localmount-speed.patch I don't do anything with mount and the directory env.d is properly checked for changes. It depends on "Refactor dependency checking into a function"
this bug is just about modules-update, stop posting unrelated (and obselete) patches
Created attachment 43725 [details, diff] modules-update.patch This one is the same as "Refactor dependency checking into a function" Except that it applies to baselayout-1.11.6-r1 and removes a touch in /etc/init.d/modules which kills the benefits of this patch. To apply, save in /tmp/modules-update.patch and apply cd / patch -p0 -b < /tmp/modules-update.patch
Created attachment 44151 [details, diff] modules-update.patch Report actions with ebegin and eend. Also, dependencyChanged is now recursive
The patch did not apply cleanly. I had to patch by hand based on the modules.rej file.
Did you have a current /etc/init.d/modules? Did you do etc-update when you last upgraded baselayout? what version of baselayout did you have? The patch is made against baselayout-1.11.6-r1 which is marked ~x86. Can you post your modules.rej?
Created attachment 45288 [details, diff] modules-update.patch This is the same thing but applied to baselayout-1.11.7-r1
base-system: i use this patch since it's available and it works fine, any chance to include it soon?
agriffis: could this please get in the next version of baselayout? It massively helps boot times on slow machines (30 second improvement for me on an NFSroot box).
uberlord: what about this?
I've merged this into baselayout CVS Will be in baselayout-1.12.0-alpha3
baselayout-1.12.x has this, no plans currently to merge it back into 1.11.x for those who want it, you should be able to just copy the modules init.d script and sbin modules-update script from baselayout-1.12.x into your baselayout-1.11.x system, but no guarantees ;p
*** Bug 100342 has been marked as a duplicate of this bug. ***
