Our situation is that we have a boot drive on the onboard ICH5 SATA controller
and the remaining 14 disks attached to a 3ware 9500 series controller. We have
to maintain the order of probing for the devices such that /dev/sda always is
the ICH5 SATA controller drive, and /dev/sd[bcdefghijklmno] are on the 3ware.
As a result, our 126.96.36.199 vanilla kernel has the ata_piix module builtin and
the 3w-9xxx module as a boot-time-loadable via modules.autoload.
Ever since the upgrade to 2005.0 from 2004.3, this is what happens on boot:
a. udev loads successfully
b. misc other things happen
c. autoload modules load in the following order:
d. mdadm addon startup script attempts to run but fails
e. checkfs attempts to fsck all the device partitions /dev/sd[b-o]2 and
fails saying they can't be opened.
f. single user mode gets entered to correct the problem
g. logging in and immediately doing mdadm -As says "/dev/md0 has been started"
and all of the device nodes exist as they should be.
The problem turns out to be that when 3w-9xxx gets loaded, udev gets informed
but udev is way too slow to add all the /dev/sd[b-o] devices, the modules
obviously return and checkfs or mdadm startup happens, but since the device
nodes arent there right away after the module load, everything fails.
In 2004.3, everything worked as expected because even though udev may have
been slow, the initial device-node tarball extraction helped to maintain
all the devices on /dev as all the devices were tarred up during halt.sh.
In 2005.0/1.11.12-r4 baselayout, halt.sh tars up ONLY those devices that
were NOT created by udev. As a workaround I added an exception check in
halt.sh to not weed out sd?? entries from tar if they existed in the udev
Perhaps a more elegant fix would be to either revert to the old tar up
everything in /dev model, or add a sleep 5 or similar immediately after
module loading. However the sleep 5 model can't be guaranteed because
you never know if udev is going to take longer than 5 seconds to create
Steps to Reproduce:
1. build onboard controller as builtin kernel module for boot drive
2. add raid cards as addon-boot-time-modules for extra drives (you might need
several so that udev does fall behind)
3. reboot... the system will fail to boot because it cant find the devices as
udev is too slow to create them
4. adding the device nodes manually doesnt help any, because the halt.sh script
ignores them for the saved tarball as it thinks udev created them (which
it did, but way too late in the game)
the system dropped to single user mode requiring operator intervention.
either waited 5 seconds or so for all the device nodes to get created, or
tar up everything that existed in /dev irrespective of deliniating between
what udev created and what was created manually.
Portage 188.8.131.52 (default-linux/amd64/2005.0, gcc-3.4.3,
glibc-184.108.40.20641102-r1, 220.127.116.11 x86_64)
System uname: 18.104.22.168 x86_64 Intel(R) Xeon(TM) CPU 3.20GHz
Gentoo Base System version 1.6.12
Python: dev-lang/python-2.3.5 [2.3.5 (#1, Jun 25 2005, 23:54:28)]
distcc 2.18.3 x86_64-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
sys-apps/sandbox: [Not Present]
sys-devel/autoconf: 2.13, 2.59-r6
sys-devel/automake: 1.5, 1.6.3, 1.8.5-r3, 1.7.9-r1, 1.4_p6, 1.9.5
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms strict"
USE="amd64 acpi alsa apache2 berkdb bitmap-fonts bzlib chroot crypt font-server
fortran gd geoip gif gpm imagemagick imap ipv6 jp2 jpeg lcms lzw lzw-tiff mbox
mhash milter mp3 mysql mysqli ncurses nls nodroproot nptl nptlonly opengl oss
pam perl php png pthreads python readline soap ssl tcpd tiff truetype
truetype-fonts type1-fonts usb userlocales xml2 xpm xrandr zlib userland_GNU
Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS
adding calls to 'sleep' in baselayout is unacceptable
I agree, baselayout should not sleep.
mdadm should loop a few times, waiting for the nodes to show up, or just
have mdadm run as a /etc/dev.d/ script off of the udev event.
No, wait, with 060, don't use /etc/dev.d/, just use a RUN= rule in udev
to do the mdadm logic. Should be quite easy.
(In reply to comment #2)
> mdadm should loop a few times, waiting for the nodes to show up, or just
> have mdadm run as a /etc/dev.d/ script off of the udev event.
Uh no, the issue is not limited to mdadm. The issue is related to all /dev/sd*
nodes not showing up in time, so even checkfs fails because checkfs runs before
they have been mknod'd by udev.
Why not revert to the old tar up everything method when RC_DEVICE_TARBALL is
set? At least we know that worked around udev's shortcomings.
Issue is not limited to mdadm.
Then build the modules into the kernel, or use genkernel to load the modules
from your initramfs.
Udev can't really do anything about this, sorry.