mdadm's initscripts fail on md_dX (partitioned RAID), as there's a couple seriously invalid assumptions made about the major and minor device numbers. The problem is in /lib/rcscripts/addons/raid-start.sh . It hard-codes device-major to 9, which is the kernel's major device for unpartitioned mds. Further, it assumes, I believe correctly for unpartitioned mds but definitely incorrectly for partitioned mds, that the minor number will correspond to the md number (mdX or md_dX). The md manpage states that the major device is listed as mdp in /proc/devices. Here, it gets major block #254. The minor numbers for the partitioned md (md_dX) appear to be 64 apart, allowing up to 63 partitions on the intervening minors. I believe I read that somewhere in my research before setting up, which backs up what I see in /dev, here, but I can't recall where and don't remember whether that was fixed or only the default. In any case, here, md_d0 is 254,0, md_d1 is 254,64, md_d1p1 is 254,65, md_d2p2 is 254,130, etc. Obviously, the initscript isn't going to set these up correctly, the device nodes fail to get created (udev), and the filesystems correspondingly fail to mount. Current workaround: When I setup the system, I set it up with an appropriate kernel command line such that all the devices are created by the kernel before control transfers to init. This will naturally be required for rootfs on kernel-RAID, but it's not recommended for other than the rootfs, as it limits flexibility in terms of RAID recovery, if/when something ultimately goes bad. Thus, now that I'm up and stable on RAID, I'm trying to switch to initscript md activation for other than the md containing the rootfs, and having a time of it as they simply aren't written to accommodate partitioned RAID. Duncan
so post a patch
(In reply to comment #1) > so post a patch It's actually on my list to do just that, but I didn't want to make any promises, and the available timeframe here is several months. If someone gets to it before me, I just saved myself some trouble. =8^) The hard part might be finding documentation suitably nailing down the device numbers for all situations, not just what happens to be the case here, as I had difficulty finding much on partitioned RAID, during my setup research. =8^( Perhaps I'll cheat by seeing how Mandriva (or Fedora or SuSE) handles it in their initscripts, since I'm familiar with their style as I used to run it, tho not on RAID. Duncan
OK, finally getting some time to look at this. The biggest sticking point was validating what the official values for major and minor were. I was able to trace down the original mdp patches to the mainline kernel, which give the info I need. From the 2.6.4 long-format changelog (reformatted): <akpm@osdl.org> [PATCH] md: Allow partitioning of MD devices. From: NeilBrown <neilb@cse.unsw.edu.au> With this patch, md used two major numbers for arrays. One Major is number 9 with name 'md' have unpartitioned md arrays, one per minor number. The other Major is allocated dynamically with name 'mdp' and had on array for every 64 minors, allowing for upto 63 partitions. The arrays under one major are completely separate from the arrays under the other. The preferred name for devices with the new major are of the form: /dev/md/d1p3 # partion 3 of device 1 - minor 67 That confirms what I see here, so I can proceed. I've hacked up a pseudocode version of raidstart.sh, but it's nowhere close to being ready for posting yet. Having it done, tested locally, and posted by the end of the week, is possible if nothing comes up. Duncan
Created attachment 86853 [details] raid-start.sh with partitioned RAID support OK, here's the new raid-start.sh script, whole version. It's tested and working here -- I'm up and running on it. I'll attach a diff/patch too, to show what changed, but there's enough new and rewritten here that it won't be so useful. It's well commented, to show what it does. Perhaps removing some of them for script inclusion in the ebuild might be useful? In particular: 1) The script uses the hint in the original to grab the dynamic mdp (md-partitioned) device number from /proc/devices. I left the hardcoded major 9 for standard unpartitioned RAID, but commenting that will cause the script to grab that one from /proc/devices too. 2) After the initial mdp device node is created, we must also create the appropriate partition device nodes for that mdp. 3) I decided to scan /etc/fstab for mdp partition entries and use those as the basis for deciding what partition nodes to create. 4) It follows from 2 and 3 that any partitions not listed in fstab will not be created. This could be an issue where LVM or the like is layered over a RAID partition, as the RAID partition would not normally be in fstab in that case, because LVM would be handling it. This can be handled with a "dummy" entry in fstab (see point 5). 5) As a result of point 4, I'd suggest an einfo in postinst, something along the lines of: For partitioned RAID, the initscript scans fstab for the partition entries needed, so it can ensure the device nodes exist. If you have partitioned RAID and have partitions NOT otherwise listed in fstab, as you might with LVM or the like, consider a dummy entry for each otherwise unlisted partition as follows: # Dev/Part MntPnt Type MntOpt dump fsck /dev/md_d1p3 ignore ignore dummy4initscript 0 0 Mentioning it in a readme.gentoo file in the docs dir may be useful, as well. Duncan
Created attachment 86854 [details, diff] patch adding partitioned RAID support to raid-start.sh Here's the patch. The same script is used in raid-tools as well? It should continue to work there as well, as I didn't touch the mdadm or raid-tools specific portions of the script at all. FWIW, this will be my first full new feature added project on Linux. It's a very good feeling to be able to point to something and say "That wouldn't be there if it weren't for me." =8^) "On the shoulders of giants..." Duncan
Just to refresh the bug... as I hit it as well ;) Many things have changed in linux since the first posting of the bug. To fix this it would be sufficient to a) allow mdadm to create the device nodes for /dev/md_d[0-9]+ with the "-a yes" switch, b) tell the kernel about the partition tables (fdisk -l $dev), c) wait a bit for udev to care of the device nodes.
Created attachment 102765 [details, diff] Patch to illustrate my idea for partitioned raid devices on gentoo
(In reply to comment #6 #7) > To fix this it would be sufficient to > a) allow mdadm to create the device nodes for > /dev/md_d[0-9]+ with the "-a yes" switch > b) tell the kernel about the partition tables (fdisk -l $dev) > c) wait a bit for udev to care of the device nodes. I'm testing your proposal now (haven't rebooted =8^). However, if raidtools supports partitioned raid (does it?), your patch is going to leave it out in the cold. Does it have something similar to mdadm's -a switch? One advantage of the method I proposed, expanding create_devs rather than bypassing it, is that it's tool-choice agnostic. Of course, even if raidtools supports partitioned raid, Gentoo could simply state that it only supports it with mdadm. Still, since the idea is implemented, the patch created, might as well use it. Of course, if raidtools doesn't support partitioned raid, we might as well go your route (assuming my reboot goes well =8^). Duncan
(In reply to comment #8) > I'm testing your proposal now (haven't rebooted =8^). Reboot went fine. A couple additional notes: 1) The -a yes starting mdadm -As isn't really needed, as it's the default, at least with newer mdadms, so that line can stay as-is. 2) Unless/until the raidtools version is changed, the raid partitions creation should be moved just before the fi closing the mdadm block (with indention changed accordingly), since raidtools continues to use the create_devs function (creating only non-partitioned traditional md devices) instead. That's where I put it, here. 3) Of course, if raidtools has a parallel to mdadm's -a, create_devs would be unused once the raidtools side was implemented, and create_devs could be removed entirely, thus simplifying the script. It'd be nice. =8^)
in 2.6.3-r2. You guys might be able to suggest some more cleanups as the raidtools stuff is gone now anyway. Open a new bug for the cleanups.
