Bugzilla Bug 123380

mdadm's initscripts fail on md_dX (partitioned RAID), as there's a couple
seriously invalid assumptions made about the major and minor device numbers.

The problem is in /lib/rcscripts/addons/raid-start.sh .  It hard-codes
device-major to 9, which is the kernel's major device for unpartitioned mds. 
Further, it assumes, I believe correctly for unpartitioned mds but definitely
incorrectly for partitioned mds, that the minor number will correspond to the
md number (mdX or md_dX).

The md manpage states that the major device is listed as mdp in /proc/devices. 
Here, it gets major block #254.  The minor numbers for the partitioned md
(md_dX) appear to be 64 apart, allowing up to 63 partitions on the  intervening
minors.  I believe I read that somewhere in my research before setting up,
which backs up what I see in /dev, here, but I can't recall where and don't
remember whether that was fixed or only the default.

In any case, here, md_d0 is 254,0, md_d1 is 254,64, md_d1p1 is 254,65, md_d2p2
is 254,130, etc.  Obviously, the initscript isn't going to set these up
correctly, the device nodes fail to get created (udev), and the filesystems
correspondingly fail to mount.

Current workaround:  When I setup the system, I set it up with an appropriate
kernel command line such that all the devices are created by the kernel before
control transfers to init.  This will naturally be required for rootfs on
kernel-RAID, but it's not recommended for other than the rootfs, as it limits
flexibility in terms of RAID recovery, if/when something ultimately goes bad. 
Thus, now that I'm up and stable on RAID, I'm trying to switch to initscript md
activation for other than the md containing the rootfs, and having a time of it
as they simply aren't written to accommodate partitioned RAID.

Duncan

so post a patch

(In reply to comment #1)
> so post a patch

It's actually on my list to do just that, but I didn't want to make any
promises, and the available timeframe here is several months.  If someone gets
to it before me, I just saved myself some trouble. =8^)  

The hard part might be finding documentation suitably nailing down the device
numbers for all situations, not just what happens to be the case here, as I had
difficulty finding much on partitioned RAID, during my setup research. =8^(
Perhaps I'll cheat by seeing how Mandriva (or Fedora or SuSE) handles it in
their initscripts, since I'm familiar with their style as I used to run it, tho
not on RAID.

Duncan

OK, finally getting some time to look at this.  The biggest sticking point was
validating what the official values for major and minor were.  I was able to
trace down the original mdp patches to the mainline kernel, which give the info
I need.  From the 2.6.4 long-format changelog (reformatted):

<akpm@osdl.org>
    [PATCH] md: Allow partitioning of MD devices.
    From: NeilBrown <neilb@cse.unsw.edu.au>

    With this patch, md used two major numbers for arrays.

    One Major is number 9 with name 'md' have unpartitioned
    md arrays, one per minor number.

    The other Major is allocated dynamically with name 'mdp'
    and had on array for every 64 minors, allowing for upto
    63 partitions.

    The arrays under one major are completely separate from
    the arrays under the other.

    The preferred name for devices with the new major are of
    the form:

        /dev/md/d1p3  # partion 3 of device 1 - minor 67

That confirms what I see here, so I can proceed.  I've hacked up a pseudocode
version of raidstart.sh, but it's nowhere close to being ready for posting yet.
 Having it done, tested locally, and posted by the end of the week, is possible
if nothing comes up.

Duncan

Created an attachment (id=86853) [edit]
raid-start.sh with partitioned RAID support

OK, here's the new raid-start.sh script, whole version.  It's tested and
working here -- I'm up and running on it.  I'll attach a diff/patch too, to
show what changed, but there's enough new and rewritten here that it won't be
so useful.

It's well commented, to show what it does.  Perhaps removing some of them for
script inclusion in the ebuild might be useful?

In particular:

1) The script uses the hint in the original to grab the dynamic mdp
(md-partitioned) device number from /proc/devices.  I left the hardcoded major
9 for standard unpartitioned RAID, but commenting that will cause the script to
grab that one from /proc/devices too.

2) After the initial mdp device node is created, we must also create the
appropriate partition device nodes for that mdp.

3) I decided to scan /etc/fstab for mdp partition entries and use those as the
basis for deciding what partition nodes to create.

4) It follows from 2 and 3 that any partitions not listed in fstab will not be
created.  This could be an issue where LVM or the like is layered over a RAID
partition, as the RAID partition would not normally be in fstab in that case,
because LVM would be handling it.  This can be handled with a "dummy" entry in
fstab (see point 5).

5) As a result of point 4, I'd suggest an einfo in postinst, something along
the lines of:

For partitioned RAID, the initscript scans fstab for the partition entries
needed, so it can ensure the device nodes exist.  If you have partitioned RAID
and have partitions NOT otherwise listed in fstab, as you might with LVM or the
like, consider a dummy entry for each otherwise unlisted partition as follows:

  # Dev/Part    MntPnt  Type    MntOpt            dump fsck
  /dev/md_d1p3  ignore  ignore  dummy4initscript  0    0

Mentioning it in a readme.gentoo file in the docs dir may be useful, as well.

Duncan

Created an attachment (id=86854) [edit]
patch adding partitioned RAID support to raid-start.sh

Here's the patch.

The same script is used in raid-tools as well?  It should continue to work
there as well, as I didn't touch the mdadm or raid-tools specific portions of
the script at all.

FWIW, this will be my first full new feature added project on Linux.  It's a
very good feeling to be able to point to something and say "That wouldn't be
there if it weren't for me."  =8^)  "On the shoulders of giants..."

Duncan

Just to refresh the bug... as I hit it as well ;) Many things have changed in
linux since the first posting of the bug.

To fix this it would be sufficient to a) allow mdadm to create the device nodes
for /dev/md_d[0-9]+ with the "-a yes" switch, b) tell the kernel about the
partition tables (fdisk -l $dev), c) wait a bit for udev to care of the device
nodes.

Created an attachment (id=102765) [edit]
Patch to illustrate my idea for partitioned raid devices on gentoo

(In reply to comment #6 #7)
> To fix this it would be sufficient to
> a) allow mdadm to create the device nodes for
> /dev/md_d[0-9]+ with the "-a yes" switch
> b) tell the kernel about the partition tables (fdisk -l $dev)
> c) wait a bit for udev to care of the device nodes.

I'm testing your proposal now (haven't rebooted =8^).  However, if raidtools
supports partitioned raid (does it?), your patch is going to leave it out in
the cold.  Does it have something similar to mdadm's -a switch?

One advantage of the method I proposed, expanding create_devs rather than
bypassing it, is that it's tool-choice agnostic.  Of course, even if raidtools
supports partitioned raid, Gentoo could simply state that it only supports it
with mdadm.  Still, since the idea is implemented, the patch created, might as
well use it.  Of course, if raidtools doesn't support partitioned raid, we
might as well go your route (assuming my reboot goes well =8^).

Duncan

(In reply to comment #8)
> I'm testing your proposal now (haven't rebooted =8^).

Reboot went fine.

A couple additional notes:

1) The -a yes starting mdadm -As isn't really needed, as it's the default, at
least with newer mdadms, so that line can stay as-is.

2) Unless/until the raidtools version is changed, the raid partitions creation
should be moved just before the fi closing the mdadm block (with indention
changed accordingly), since raidtools continues to use the create_devs function
(creating only non-partitioned traditional md devices) instead.  That's where I
put it, here.

3) Of course, if raidtools has a parallel to mdadm's -a, create_devs would be
unused once the raidtools side was implemented, and create_devs could be
removed entirely, thus simplifying the script.  It'd be nice. =8^)

in 2.6.3-r2.
You guys might be able to suggest some more cleanups as the raidtools stuff is
gone now anyway. Open a new bug for the cleanups.