Cannot boot into md RAID with initrd produced by genkernel-3.4.6, /dev/mdX is not visible while mounting root; mdadm is 2.6.1, kernel 2.6.20. Actual code in initramfs is broken - it wants to call mdstart without /proc/mdstat (created by raidX modules which are not loaded) and without /dev/mdX nodes. Reproducible: Always Steps to Reproduce: 1.mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1 2.genkernel initrd [...] 3.try to boot with grub: root=/dev/md1 [...] Actual Results: No root device found Expected Results: I want bootable gentoo What can be done? Add raid modules: --------------------------------------- /usr/share/genkernel/x86/modules_load: --------------------------------------- MODULES_SATA="sata_promise sata_sil sata_sil24 sata_svw sata_via \ sata_nv sata_sx4 sata_sis sata_uli sata_vsc sata_qstor ahci \ ata_piix sata_mv pdc_adma raid0 raid1 raid10 raid456" MODULES_FS="ext2 ext3 reiserfs jfs nfs xfs" Add some code to autoconfigure md devices: -------------------------------------------- /usr/share/genkernel/generic/initrd.scripts: -------------------------------------------- startVolumes() { #good_msg 'Checking if volumes need to be started...' # Here, we check for /dev/device-mapper, and if it exists, we setup a # a symlink, which should hopefully fix bug #142775 and bug #147015 if [ -e /dev/device-mapper ] && [ ! -e /dev/mapper/control ] then mkdir -p /dev/mapper ln -sf /dev/device-mapper /dev/mapper/control fi if [ "${USE_MDRAID}" -eq '1' ] then for i in 0 1 10 456 ; do modprobe raid$i 1>/dev/null 2>/dev/null #/proc/mdstat will be created done for i in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ; do mknod /dev/md$i b 9 $i 1>/dev/null 2>/dev/null done # static mdadm needs to be copied by gen_initramfs.sh # cp /sbin/mdadm ${TEMP}/initramfs-base-temp/sbin/mdadm /sbin/mdadm --examine --scan > /etc/mdadm.conf /sbin/mdadm --assemble --scan 1>/dev/null 2>/dev/null # md devices will be activated fi ---- and in linuxrc CMDLINE parsing: mdraid) USE_MDRAID=1 ;;
Portage 2.1.2.2 (default-linux/x86/2006.0, gcc-4.1.2, glibc-2.5-r0, 2.6.20-gentoo-rt5 i686) ================================================================= System uname: 2.6.20-gentoo-rt5 i686 Intel(R) Celeron(R) M CPU 420 @ 1.60GHz Gentoo Base System release 1.12.9 Timestamp of tree: Sun, 25 Mar 2007 01:00:08 +0000 dev-java/java-config: 1.3.7, 2.0.31-r5 dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.61 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-Os -march=pentium3 -ftracer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="${NXDIR}/etc ${NXDIR}/home /etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/php/apache1-php5/ext-active/ /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c" CXXFLAGS="-Os -march=pentium3 -ftracer -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks metadata-transfer prelink sandbox sfperms strict" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LC_ALL="pl_PL.UTF-8" LINGUAS="pl pl_PL en" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage /usr/portage/local/layman/enlightenment /usr/portage/local/layman/nx /usr/portage/local/layman/xeffects /usr/portage/local/layman/n4g" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X aiglx alsa apache2 apm arts avahi berkdb bitmap-fonts cairo cli cracklib crypt cups dbus dlloader dri emboss encode esd foomaticdb gdbm gif gpm gstreamer gtk gtk2 hal hardened iconv imlib ipv6 isdnlog java jikes jpeg kde kdehiddenvisibility kickoff libg++ libwww mad midi mikmod mmx mono motif mp3 mp4 mpeg ncurses nls nptl ogg opengl oss pam pcre perl pic png pppd python qt3 qt3support qt4 quicktime readline reflection sdl session spell spl sqlite sqlite3 sse ssl tcpd truetype truetype-fonts type1-fonts unicode vmware vorbis x86 xcomposite xml xorg xrandr xv zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="synaptics wacom vmmouse evdev mouse keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="pl pl_PL en" USERLAND="GNU" VIDEO_CARDS="mga i740 i810 nv nvidia ati radeon r128 mach64 fglrx s3 s3virge savage sis vesa vga via vmware tdfx" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LDFLAGS, MAKEOPTS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
There's no way that I'm adding the RAID stuff to SATA. Instead, it would need its own modules_load section. Instead, try using --dmraid to genkernel and booting it and see if it works for you. I've added raid456 to that module group, which is really the one we should be using. There's no need for the mdadm stuff, either, so long as your partition is type "fd" which any boot raid should be, anyway.
> Instead, try using --dmraid to genkernel and booting it and see if it works for you. I've tried and it doesn't work. localhost ~ # fdisk -l /dev/sda Disk /dev/sda: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 12 96358+ fd Linux raid autodetect /dev/sda2 13 261 2000092+ fd Linux raid autodetect /dev/sda3 262 30401 242099550 fd Linux raid autodetect localhost ~ # fdisk -l /dev/sdb Disk /dev/sdb: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 12 96358+ fd Linux raid autodetect /dev/sdb2 13 261 2000092+ fd Linux raid autodetect /dev/sdb3 262 30401 242099550 fd Linux raid autodetect sd[ab]1 is /boot RAID1, sd[ab]3 is / intird fails, and from ash I can see that there are no /dev/md* and /dev/mapper/* devices
Do the modules not get loaded? I cannot duplicate this on a LiveCD build, so I'm trying to see what's going on here so I can get it fixed before 2007.0 is released.
---------------------- FROM INITRD SHELL ---------------------- lsmod: dm_mirror raid456 xor raid10 raid1 raid0 md_mod ls /dev/md*: /dev/md3 dmesg: md: raid0 personality registered for level 0 md: raid1 personality registered for level 1 md: raid10 personality registered for level 10 raid5: measuring checksumming speed ... md: raid4 personality registered for level 4 EXT3-fs: unable to read superblock EXT2-fs: unable to read superblock XFS: SB read failed cat /proc/mdstat: Personalities : [raid0] ... [raid4] unused devices: <none> ***************** and after running: mdadm --examine --scan > /etc/mdadm.conf mdadm --assemble --scan here is output of dmesg: md: md1 stopped. md: bind<sdb1> md: bind<sda1> raid1: raid set md1 active with 2 out of 2 mirrors md: md2 stopped. md: bind<sdb2> md: bind<sda2> md2: setting max_sectors to 128, segment boundary to 32767 raid0: looking at sda2 raid0: comparing sda2(2000000) with sda2(2000000) raid0: END raid0: ==> UNIQUE raid0: 1 zomes raid0: looking at sdb2 raid0: comparing sdb2(2000000) with sda2(2000000) raid0: EQUAL raid0: FINAL 1 zones raid0: done. ... and in /proc/mdstat I can see md2, md1 and 1 md3. System is able to boot.
strace of mdadm --assemble /dev/md1: open("/dev/md1", O_RDWR) = 4 ... ioctl(4, 0x800c0910, 0xbfd0a7a8) = 0 ... ioctl(4, 0x800c0910, 0xbfd0a568) = 0 ioctl(4, 0x80480911, 0xbfd0a644) = -1 ENODEV (No such device) ioctl(4, 0x932, 0) = 0 ... ioctl(4, 0x40480923, 0xbfd0a7b8) = 0 ioctl(4, 0x40140921, 0xbfd0a824) = 0 ioctl(4, 0x40140921, 0xbfd0a824) = 0 ioctl(4, 0x400c0930, 0) = 0 write(2, "mdadm: /dev/md1 has been started"..., 46mdadm: /dev/md1 has been started with 2 drives) = 46
----------- Assemble.c ----------- ioctl(4, 0x80480911, 0xbfd0a644) = -1 ENODEV (No such device) =ioctl(mdfd, GET_ARRAY_INFO, &info.array) ioctl(4, 0x932, 0) = 0 =ioctl(mdfd, STOP_ARRAY, NULL) ...
Some kernel autodetection removed in 2.6.20? http://lkml.org/lkml/2006/7/31/192
BusyBox v1.1.3 mdstart doesn't do anything (looking at dmesg)
(In reply to comment #2) > Instead, try using --dmraid to genkernel and booting it and see if it works for > you. I've added raid456 to that module group, which is really the one we > should be using. There's no need for the mdadm stuff, either, so long as your > partition is type "fd" which any boot raid should be, anyway. AFAIK dmraid and mdraid have nothing to do with each other. mdadm is necessary to assemble arrays if: 1) the md stuff is modular OR 2) the md superblocks are 1.0 format. Only 0.90 format is auto assembled. I would not be surprised if auto assembly is dropped sometime anyway. It makes a lot more sense to have it in userspace/initramfs.
Feel free to submit an actual unified diff patch, then. It really sucks that the kernel developers are making us add even more crap into our early userspace. Every single byte counts on some architectures. This alone might make things prohibitively sizable for certain configurations. Also, I *know* that dmraid and mdraid had (little) to do with each other, other than being in the same config section in the kernel and sharing many modules. I just saw no point in duplicating the same thing and performing another set of module loads when the end result is the same modules being loaded. Anyway, I guess since the kernel doesn't look like it will be supporting something that's been around for years and is infinitely useful to tons of administrators, we'll just have to hack around yet another brilliant move to drop support for something that simply wasn't broken to begin with... *sigh*
(In reply to comment #11) > Also, I *know* that dmraid and mdraid had (little) to do with each other, other > than being in the same config section in the kernel and sharing many modules. > I just saw no point in duplicating the same thing and performing another set of > module loads when the end result is the same modules being loaded. I prefer to see dmraid and mdraid as separate options for genkernel. I think the mdraid modules (raid0/1/456) should no longer be loaded when dmraid is requested, since dmraid does not use them. Do you agree? I'm setting up a virtual machine so I can play with changes and hopefully submit a unified diff soon.
Created attachment 114812 [details, diff] Patch to add mdraid (mdadm) support to genkernel initramfs Here's my first stab at adding mdadm support to genkernel-3.4.7_pre5. It works for me, booting a version-1 superblock RAID-1 array, with LVM on top of that, with a root filesystem on LVM. I used the genkernel --do I'd like to clean it up a bit more, but comments would be appreciated. I'm not entirely happy with calling it mdraid, as mdraid/dmraid are very confusing. Maybe mdadm is a better choice. Thoughts? Also, I compiled mdadm statically. I suppose it would be a lot better/smaller to link against klibc? I don't know anything about how that works yet. Last but not least, it is not (yet) forcing RAID modules into the kernel config or complaining if the options are not set. Comments, suggestions, testing appreciated. Thanks!
> I used the genkernel --do That should read 'genkernel --lvm2 --mdraid'
So there's no way to get around using mdadm, is there? We just managed to get away from having to bundle klibc because it was a big steaming pile of poo and caused us many headaches on many architectures. Does busybox have a mdadm work-a-like that we can use? If not, can we add mdadm as a busybox applet? This is my preferred approach, rather than forcing another static build. Your patch looks good, though. I'd just prefer try to get a "cleaner" solution that would take up less on-disk real estate.
It turns out there's a program called mdassemble in the mdadm package which is not a part of the normal ebuild. I haven't looked at it yet, so I can't say whether it's significantly smaller or how useful it is. It looks like someone submitted some mdadm patches for busybox in 2005. If it doesn't have too much bit rot maybe we can use it. The author of mdadm placed some comments on early userspace in the mdadm sources, which I didn't notice until I uploaded my patch. He has a few ideas worth incorporating into the initramfs, such as the ability to specify which array UUID to activate. It doesn't make sense to automagically activate every array in many cases. I'll consider the options and submit a new patch tomorrow.
Oops, the mdadm patches were for buildroot (uClibc), not busybox. I've looked at mdassemble. I don't think it can be made to automagically assemble arrays. It appears to need an mdadm.conf file with either a UUID, list of partitions, or similar. On my machine (amd64) the mdassemble binary is about 24% (259k) smaller than mdadm. Not being able to do automagic array detection is probably not a bad thing. The initramfs should only be concerned with activating a root filesystem. I'm about to do some tests with mdassemble to see if I can use it. Chris, why does MODULES_DMRAID include raid0, raid1, raid10?
I have a grave concern. If we only activate one array at boot time, but other arrays exist, and those arrays contain LVM data ... the LVM code in the initramfs will activate those LVs without the underlying array being active (even though it can usually tell that the PVs are duplicated), and put them in an inconsistent state. But the md subsystem will not be aware that the array is inconsistent. The LVM code has a setting to prevent this, md_component_detection. This is turned on by default in the ebuild, but the initramfs has no config file. I am adding one so that this feature can be turned on.
By the way, this is all reproducible with 2.6.19 and genkernel 3.4.7_pre5 or is this 2.6.20+ only? Since we're so close to the release date, I might just have us use the current patch here and revert it later once a better patch comes along. What do you think? My rush here is the release, so if this only affects 2.6.20+ then I can ignore it for now, get my release out, then fix it.
(In reply to comment #19) > By the way, this is all reproducible with 2.6.19 and genkernel 3.4.7_pre5 or is > this 2.6.20+ only? > > Since we're so close to the release date, I might just have us use the current > patch here and revert it later once a better patch comes along. What do you > think? My rush here is the release, so if this only affects 2.6.20+ then I can > ignore it for now, get my release out, then fix it. > I'm using 2.6.16.42 since I need Xen, and that's the latest stable kernel ported to Xen. I don't think anything here is dependent on the kernel version. I've studied the md code in newer kernels.
What's new on mdraid addition to genkernel?
I know that this has been sitting for a little while and I apologize. What I would like to see is if the new busybox coming $soon will have what we need to support mdraid without compiling our own copy of mdadm or any of its components. The upstream busybox developers are very accepting of patches that are clean and add functionality that we need. I think the ability to assemble mdraid arrays probably qualifies. ;] I've added a MODULES_MDRAID group to modules_load in genkernel SVN and want to complete the support for it by giving it a command line switch, like in the patch here. I'm adding vapier here since he's our local busybox guru and is trying to ensure we have all the support we want for genkernel in busybox. Mike, I am not very familiar with busybox internals and am definitely not up to speed on mdadm. Only recently have I even begun using it and I'm still a long way off from knowing it well enough to write code for genkernel without a little help. Does busybox already (or will in the near future) have the support we need for assembling arrays?
there is nothing in busybox world for integrating mdadm nor are there plans it'd be a pretty hard sell to integrate all of mdadm, but i think focusing on just the assemble code might be doable ... the only raid related code in busybox now is `raidautorun` and all that does is fire the ioctl(RAID_AUTORUN)
I've just bumped into this too. Any news on this ?
Chris - given that Jeff produced the patch to make mdadm be an option, it shouldn't affect those who don't need the functionality. And I doubt those who have small embedded systems need this and won't be affected by the increased size of the initramfs. Any chance that Jeff's patch would make it verbatim into genkernel anytime soon ?
Hello ??
O.k. So I've received a rather unpolite message from vslavik regarding my last message, so apologies if that's offended anyone, but Jeff has put the work in to get mdadm support into genkernel. I'm just interested in it's progress, given it's sat here for a few months now.
Please try to keep such "update" comments off this bug. It doesn't do anything to actually resolve the issue and doesn't help anyone. If there was an update, you would see it here. Asking for one reads to many people very rudely. We are, after all, volunteers, and work on what we want, when we want. You're on the CC list. You know as much as anybody else does. Have some patience. (No, I'm not going to answer when/if this will be included, as I haven't decided yet and I've now got a really bad taste in my mouth on this, so I'm going to hold off on any decisions for a while.)
No problem. I'd actually starting coding mdassemble for busybox, but I guess I'll hold off now.
(In reply to comment #15) > So there's no way to get around using mdadm, is there? We just managed to get > away from having to bundle klibc because it was a big steaming pile of poo and > caused us many headaches on many architectures. Does busybox have a mdadm > work-a-like that we can use? If not, can we add mdadm as a busybox applet? > This is my preferred approach, rather than forcing another static build. Your > patch looks good, though. I'd just prefer try to get a "cleaner" solution that > would take up less on-disk real estate. > Chris, To avoid pissing anyone off more than necessary and to show that I'm quite happy to put effort in when needed. I have a preliminary patch that adds both mdadm's --examine and --assemble functionality to busybox and it currently only increases the size of busybox by about 45K. That's probably doable to get that down furthur. So it just means some slight modification to the initrd.scripts which mimicks what Jeff originally did. That gives the flexibility for people to install their own /etc/mdadm.conf file and surpass the --examine pass, then just execute --assemble. I'll upload later today if you like.
SpanKY: would busybox upstream accept a minimal mdadm capable of doing --examine and --assemble? Alan: that would be great... I never answered your questions because #1, I'm busy as hell and #2, I thought that I had already answered it... Remember that we *need* md RAID support in our releases. That means that using Jeff's original patch is simply unmanageable for Release Engineering. Since genkernel is primarily a release-tool (in my eyes) and a user tool second, if it causes me massive headaches during release building, it simply won't go in. Anything using static binaries will cause more problems than it is worth, which is why we're working to remove all such programs from genkernel and replacing them with saner options, such as adding support to busybox itself.
Chris, I doubt the patch, in it's current form, would be accepted by the busybox folks, as it does borrow most of the code from mdadm for examine/assemble because the functions themselves are pretty well isolated in the original mdadm code. I guess if SpanKY wants to review and suggest cleanups etc then I'm quite happy to make the necessary changes to help get it upstream to busybox too. I'm just doing another test run and I'll upload the patches within the hour.
Created attachment 132494 [details, diff] Patch to genkernel 3.4.8 which adds MDADM support O.k. This is the patch to add mdadm support to genkernel. If you pass the option --mdadm to genkernel then it copies the localhosts /etc/mdadm.conf to the initramfs to use and skips the examine phase upon bootup, otherwise it will run --examine and create it's own /etc/mdadm.conf at boot time. Secondly, you must add "domdadm" to the boot kernel flags to actually make it execute the --assemble. I've just noticed that I should be checking that a /etc/mdadm.conf exists before trying to copy it, so it does need a tweak.
Created attachment 132495 [details, diff] Busybox patch for MDADM support against busybox-1.1.3+gentoo O.k. Here's the biggy. It's a busybox patch against the genkernel 3.4.8's busybox which is currently busybox-1.1.3+gentoo. The previous genkernel.patch already adds the extra CONFIG_MDADM=y line to the busy-config for X86 only by the way. I've done a bit of dead code removal, but there's still a bit more in the code which isn't needed anymore. It's down to about 35K of binary when compiled in now. Let me know what you think anyway.
You're going to want to redo that patch against current busybox (1.7.1 or so) in order to get it upstream. Also, we'll be changing the busybox that's used for genkernel to 1.7.0+ in genkernel-3.5, which is the earliest that a patch like this would probably go in, anyway.
It should be pretty trivial to fix up for any version of busybox really. Let me know where I can pull genkernel 3.5 and busybox 1.7.0+ to generate the diff though.
Genkernel 3.5 doesn't exist, yet. We're releasing 3.4.9 shortly, and then 3.5 will have major cleanups, enhancements, etc. For busybox, try upstream :P
Yes, I can get busybox 1.7.1 from upstream, but you mentioned 1.7.0+ which I assume the + means there's gentoo additions. So I'd rather generate against that first, and then let SpanKY take a look for approaching anyone upstream for busybox.
Ugh. Change "for approaching" to "before approaching"
1.7.0+ means just that...1.7.0 or higher. I have no idea what version of busybox will be around by the time we release genkernel 3.5.
O.k. So if genkernel 3.5 doesn't exist I can't generate a patch against that, so I'll stick with 3.4.8. Let's see if SpanKY and Chris can review the busybox patch first and make comments before making it work with later versions of busybox which aren't yet integrated with genkernel.
(In reply to comment #41) > O.k. So if genkernel 3.5 doesn't exist I can't generate a patch against that, > so I'll stick with 3.4.8. How about SVN, instead? http://anonsvn.gentoo.org/ Everything is in SVN under genkernel/trunk. I would actually like to try to include this in genkernel 3.4.9, so going against SVN is perfect. > Let's see if SpanKY and Chris can review the busybox patch first and make > comments before making it work with later versions of busybox which aren't yet > integrated with genkernel. We'll stick with 1.1.3+gentoo for 3.4.9, but will be using a newer busybox for 3.5, as Andrew said. I'd love to see this go into busybox upstream before we switch to a newer busybox, but I don't know if that will happen. I'll let Mike comment on the patch, since busybox is his territory.
Created attachment 132596 [details, diff] Patch for SVN version of genkernel Here you go Chris - this is for SVN genkernel.
OK. I've added this to SVN now.
Please test genkernel 3.4.9_pre4 and make sure this is fixed for you. Thanks
tested 3.4.9_pre6 and all is well. thanks for committing my updates.
