Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 97685 - baselayout-1.11.12-r4 module loading/checkfs and udev device creation race condition
Summary: baselayout-1.11.12-r4 module loading/checkfs and udev device creation race co...
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] baselayout (show other bugs)
Hardware: AMD64 Linux
: High major (vote)
Assignee: Greg Kroah-Hartman (RETIRED)
Depends on:
Reported: 2005-07-01 21:04 UTC by MKI
Modified: 2005-12-07 13:16 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description MKI 2005-07-01 21:04:37 UTC
Our situation is that we have a boot drive on the onboard ICH5 SATA controller
and the remaining 14 disks attached to a 3ware 9500 series controller.  We have
to maintain the order of probing for the devices such that /dev/sda always is
the ICH5 SATA controller drive, and /dev/sd[bcdefghijklmno] are on the 3ware. 
As a result, our vanilla kernel has the ata_piix module builtin and
the 3w-9xxx module as a boot-time-loadable via modules.autoload.

Ever since the upgrade to 2005.0 from 2004.3, this is what happens on boot:

a. udev loads successfully
b. misc other things happen
c. autoload modules load in the following order:
d. mdadm addon startup script attempts to run but fails
e. checkfs attempts to fsck all the device partitions /dev/sd[b-o]2 and
   fails saying they can't be opened.
f. single user mode gets entered to correct the problem
g. logging in and immediately doing mdadm -As says "/dev/md0 has been started"
   and all of the device nodes exist as they should be.

The problem turns out to be that when 3w-9xxx gets loaded, udev gets informed
but udev is way too slow to add all the /dev/sd[b-o] devices, the modules
obviously return and checkfs or mdadm startup happens, but since the device
nodes arent there right away after the module load, everything fails.

In 2004.3, everything worked as expected because even though udev may have
been slow, the initial device-node tarball extraction helped to maintain 
all the devices on /dev as all the devices were tarred up during
In 2005.0/1.11.12-r4 baselayout, tars up ONLY those devices that
were NOT created by udev.  As a workaround I added an exception check in to not weed out sd?? entries from tar if they existed in the udev

Perhaps a more elegant fix would be to either revert to the old tar up
everything in /dev model, or add a sleep 5 or similar immediately after
module loading.  However the sleep 5 model can't be guaranteed because
you never know if udev is going to take longer than 5 seconds to create
device nodes!!

Reproducible: Always
Steps to Reproduce:
1. build onboard controller as builtin kernel module for boot drive
2. add raid cards as addon-boot-time-modules for extra drives (you might need
   several so that udev does fall behind)
3. reboot... the system will fail to boot because it cant find the devices as
   udev is too slow to create them
4. adding the device nodes manually doesnt help any, because the script
   ignores them for the saved tarball as it thinks udev created them (which
   it did, but way too late in the game)

Actual Results:  
the system dropped to single user mode requiring operator intervention.

Expected Results:  
either waited 5 seconds or so for all the device nodes to get created, or
tar up everything that existed in /dev irrespective of deliniating between
what udev created and what was created manually.

Portage (default-linux/amd64/2005.0, gcc-3.4.3,
glibc-, x86_64)
System uname: x86_64 Intel(R) Xeon(TM) CPU 3.20GHz
Gentoo Base System version 1.6.12
Python:              dev-lang/python-2.3.5 [2.3.5 (#1, Jun 25 2005, 23:54:28)]
distcc 2.18.3 x86_64-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
dev-lang/python:     2.3.5
sys-apps/sandbox:    [Not Present]
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.5, 1.6.3, 1.8.5-r3, 1.7.9-r1, 1.4_p6, 1.9.5
sys-devel/libtool:   1.5.16
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config
/usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms strict"
USE="amd64 acpi alsa apache2 berkdb bitmap-fonts bzlib chroot crypt font-server
fortran gd geoip gif gpm imagemagick imap ipv6 jp2 jpeg lcms lzw lzw-tiff mbox
mhash milter mp3 mysql mysqli ncurses nls nodroproot nptl nptlonly opengl oss
pam perl php png pthreads python readline soap ssl tcpd tiff truetype
truetype-fonts type1-fonts usb userlocales xml2 xpm xrandr zlib userland_GNU
kernel_linux elibc_glibc"
Comment 1 SpanKY gentoo-dev 2005-07-02 07:30:38 UTC
adding calls to 'sleep' in baselayout is unacceptable
Comment 2 Greg Kroah-Hartman (RETIRED) gentoo-dev 2005-07-03 00:14:18 UTC
I agree, baselayout should not sleep.

mdadm should loop a few times, waiting for the nodes to show up, or just
have mdadm run as a /etc/dev.d/ script off of the udev event.

No, wait, with 060, don't use /etc/dev.d/, just use a RUN= rule in udev
to do the mdadm logic.  Should be quite easy.
Comment 3 MKI 2005-07-03 05:28:51 UTC
(In reply to comment #2)
> mdadm should loop a few times, waiting for the nodes to show up, or just
> have mdadm run as a /etc/dev.d/ script off of the udev event.

Uh no, the issue is not limited to mdadm.  The issue is related to all /dev/sd*
nodes not showing up in time, so even checkfs fails because checkfs runs before
they have been mknod'd by udev.

Why not revert to the old tar up everything method when RC_DEVICE_TARBALL is
set?  At least we know that worked around udev's shortcomings.

Comment 4 MKI 2005-07-03 05:30:05 UTC
Issue is not limited to mdadm.
Comment 5 Greg Kroah-Hartman (RETIRED) gentoo-dev 2005-12-07 13:16:24 UTC
Then build the modules into the kernel, or use genkernel to load the modules
from your initramfs.

Udev can't really do anything about this, sorry.