Our situation is that we have a boot drive on the onboard ICH5 SATA controller and the remaining 14 disks attached to a 3ware 9500 series controller. We have to maintain the order of probing for the devices such that /dev/sda always is the ICH5 SATA controller drive, and /dev/sd[bcdefghijklmno] are on the 3ware. As a result, our 2.6.11.12 vanilla kernel has the ata_piix module builtin and the 3w-9xxx module as a boot-time-loadable via modules.autoload. Ever since the upgrade to 2005.0 from 2004.3, this is what happens on boot: a. udev loads successfully b. misc other things happen c. autoload modules load in the following order: raid10 3w-9xxx e1000 d. mdadm addon startup script attempts to run but fails e. checkfs attempts to fsck all the device partitions /dev/sd[b-o]2 and fails saying they can't be opened. f. single user mode gets entered to correct the problem g. logging in and immediately doing mdadm -As says "/dev/md0 has been started" and all of the device nodes exist as they should be. The problem turns out to be that when 3w-9xxx gets loaded, udev gets informed but udev is way too slow to add all the /dev/sd[b-o] devices, the modules obviously return and checkfs or mdadm startup happens, but since the device nodes arent there right away after the module load, everything fails. In 2004.3, everything worked as expected because even though udev may have been slow, the initial device-node tarball extraction helped to maintain all the devices on /dev as all the devices were tarred up during halt.sh. In 2005.0/1.11.12-r4 baselayout, halt.sh tars up ONLY those devices that were NOT created by udev. As a workaround I added an exception check in halt.sh to not weed out sd?? entries from tar if they existed in the udev database. Perhaps a more elegant fix would be to either revert to the old tar up everything in /dev model, or add a sleep 5 or similar immediately after module loading. However the sleep 5 model can't be guaranteed because you never know if udev is going to take longer than 5 seconds to create device nodes!! Reproducible: Always Steps to Reproduce: 1. build onboard controller as builtin kernel module for boot drive 2. add raid cards as addon-boot-time-modules for extra drives (you might need several so that udev does fall behind) 3. reboot... the system will fail to boot because it cant find the devices as udev is too slow to create them 4. adding the device nodes manually doesnt help any, because the halt.sh script ignores them for the saved tarball as it thinks udev created them (which it did, but way too late in the game) Actual Results: the system dropped to single user mode requiring operator intervention. Expected Results: either waited 5 seconds or so for all the device nodes to get created, or tar up everything that existed in /dev irrespective of deliniating between what udev created and what was created manually. Portage 2.0.51.19 (default-linux/amd64/2005.0, gcc-3.4.3, glibc-2.3.4.20041102-r1, 2.6.11.12 x86_64) ================================================================= System uname: 2.6.11.12 x86_64 Intel(R) Xeon(TM) CPU 3.20GHz Gentoo Base System version 1.6.12 Python: dev-lang/python-2.3.5 [2.3.5 (#1, Jun 25 2005, 23:54:28)] distcc 2.18.3 x86_64-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] dev-lang/python: 2.3.5 sys-apps/sandbox: [Not Present] sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.5, 1.6.3, 1.8.5-r3, 1.7.9-r1, 1.4_p6, 1.9.5 sys-devel/binutils: 2.15.92.0.2-r10 sys-devel/libtool: 1.5.16 virtual/os-headers: 2.6.8.1-r4 ACCEPT_KEYWORDS="amd64" AUTOCLEAN="yes" CFLAGS="-O2" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.ccccom.com" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="amd64 acpi alsa apache2 berkdb bitmap-fonts bzlib chroot crypt font-server fortran gd geoip gif gpm imagemagick imap ipv6 jp2 jpeg lcms lzw lzw-tiff mbox mhash milter mp3 mysql mysqli ncurses nls nodroproot nptl nptlonly opengl oss pam perl php png pthreads python readline soap ssl tcpd tiff truetype truetype-fonts type1-fonts usb userlocales xml2 xpm xrandr zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CBUILD, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS
adding calls to 'sleep' in baselayout is unacceptable
I agree, baselayout should not sleep. mdadm should loop a few times, waiting for the nodes to show up, or just have mdadm run as a /etc/dev.d/ script off of the udev event. No, wait, with 060, don't use /etc/dev.d/, just use a RUN= rule in udev to do the mdadm logic. Should be quite easy.
(In reply to comment #2) > mdadm should loop a few times, waiting for the nodes to show up, or just > have mdadm run as a /etc/dev.d/ script off of the udev event. Uh no, the issue is not limited to mdadm. The issue is related to all /dev/sd* nodes not showing up in time, so even checkfs fails because checkfs runs before they have been mknod'd by udev. Why not revert to the old tar up everything method when RC_DEVICE_TARBALL is set? At least we know that worked around udev's shortcomings.
Issue is not limited to mdadm.
Then build the modules into the kernel, or use genkernel to load the modules from your initramfs. Udev can't really do anything about this, sorry.
