I have created an array md0 in raid 1. The array is stable when only 1 device is inside. As soon as a second device is included the system start to be unstable and reboot. Hard drives have been tested and are ok, I have even bough a brand new one and same behaviour. I have deleted the array recreate it and format it but still same behaviour. To avoid a reboot loop I have to disable one of the device to remove it from array. The unstability is not linked to one specific device as array is stable with both devices alone in array. /dev/md0: Version : 00.90.03 Creation Time : Wed Oct 22 22:25:08 2008 Raid Level : raid1 Array Size : 78148096 (74.53 GiB 80.02 GB) Used Dev Size : 78148096 (74.53 GiB 80.02 GB) Raid Devices : 2 Total Devices : 1 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Oct 25 20:00:45 2008 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 UUID : 3fff4b2a:becd3be1:e368bf24:bd0fce41 Events : 0.3018 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 1 1 active sync /dev/sda1 igor@Avalon ~ $ sudo mdadm --version mdadm - v2.6.4 - 19th October 2007 I have tried 2 files system : reiserfs and ext3 and same behaviour. I do not used /etc/mdadm.conf. Reproducible: Always Steps to Reproduce: 1.Add a device to my array (mdadm --add /dev/md0 xx 2.System crash and start reboot loop
How are both disks hooked up to your system? What type of interface do they use (PATA, SATA, SCSI)?
(In reply to comment #1) > How are both disks hooked up to your system? What type of interface do they use > (PATA, SATA, SCSI)? > Hello, I used 2 SATA for my array and 1 hdd ide for main system. Here the output for the current one in cluster: igor@Avalon ~ $ sudo hdparm -i /dev/sda /dev/sda: Model=MAXTOR STM380211AS , FwRev=3.AAE , SerialNo= 6PS2XLV1 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=2048kB, MaxMultSect=16, MultSect=?1? CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=156301488 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7 * signifies the current active mode igor@Avalon ~ $ My motherboard is asus m2n4-sli. Thank you,
Please post your `emerge --info' too. It would be useful to obtain some more output from the kernel at the time of the failure too.
(In reply to comment #3) > Please post your `emerge --info' too. It would be useful to obtain some more > output from the kernel at the time of the failure too. > Hello, here you have : igor@Avalon ~ $ sudo emerge --info Password: Portage 2.1.4.5 (default/linux/amd64/2008.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.25-gentoo-r7 x86_64) ================================================================= System uname: 2.6.25-gentoo-r7 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ Timestamp of tree: Sat, 25 Oct 2008 08:36:01 +0000 app-shells/bash: 3.2_p33 dev-java/java-config: 1.3.7, 2.1.6 dev-lang/python: 2.4.4-r13, 2.5.2-r7 dev-python/pycrypto: 2.0.1-r6 dev-util/cmake: 2.4.6-r1 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.61-r2 sys-devel/automake: 1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=athlon64 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-march=athlon64 -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://trumpetti.atm.tut.fi/gentoo/ ftp://trumpetti.atm.tut.fi/gentoo/ " LDFLAGS="-Wl,-O1" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow 3dnowext ARCH X aac aalib acl alsa amd64 avahi avi berkdb branding bzip2 bzlib cdr cdrparanoia cli cracklib crypt cups dbus divx4linux dri dv dvd dvdr dvdread encode ffmpeg flac fortran gdbm gif gnome gpm gtk gtk2 hal hardened iconv ipv6 isdnlog jpeg lame mad mdnsresponder-compat midi mmx mmxext mp3 msn mudflap multilib ncurses nls nptl nptlonly nvidia ogg opengl openmp pam pcre perl png pppd python qt qt3support quicktime readline reflection samba sdl session spell spl sse sse2 ssl svg sysfs tcpd tiff tk truetype unicode usb vorbis xorg xv xvid xvmc zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="nvidia nv" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY igor@Avalon ~ $ However I cannot send output of kernel during failure as my system reboot pretty quickly ...
Hi, I can provide remote access to the machine if needed.
What size is your power supply? If, instead of adding the second disk to the array, you just format it and run long bonnie tests on both disks at the same time, do you still get the instability?
(In reply to comment #6) > What size is your power supply? > If, instead of adding the second disk to the array, you just format it and run > long bonnie tests on both disks at the same time, do you still get the > instability? > Hello, I have standard 500W ATX power supply. Could you please let me know what test/command do you suggest me to do ? I can try. Igor
just anything that causes load on the disk. even this would probably work fine: (assuming the second disk is sdb) dd if=/dev/sdb of=/dev/null bs=1M & dd if=/dev/sda of=/dev/null bs=1M & See if the system crashes at all. That should run both disks heavily. You can try to run something that causes lots of cpuload at the same time. Specifically, you are trying to use as much power as possible, to see if your power supply is faulty or too small for your setup (specifically the 5V/12V disk rails might be underpowered) - I suspect it is since you bought a new disk and still have the same problem. Alternatively as another test, if you have another power supply handy, try running just one of the disks off it, and see if the problem persists at that.
(In reply to comment #8) > just anything that causes load on the disk. > even this would probably work fine: > (assuming the second disk is sdb) > dd if=/dev/sdb of=/dev/null bs=1M & > dd if=/dev/sda of=/dev/null bs=1M & > > See if the system crashes at all. That should run both disks heavily. You can > try to run something that causes lots of cpuload at the same time. > > Specifically, you are trying to use as much power as possible, to see if your > power supply is faulty or too small for your setup (specifically the 5V/12V > disk rails might be underpowered) - I suspect it is since you bought a new disk > and still have the same problem. Alternatively as another test, if you have > another power supply handy, try running just one of the disks off it, and see > if the problem persists at that. > Hello, I have tried as requested : dd if=/dev/sdb of=/dev/null bs=1M count=30000 dd if=/dev/sda of=/dev/null bs=1M count=30000 The results are 31 GB copied 23.0 MB/s and 22.1 MB/s for both drives. At the same time I started a kernel compilation to load cpu but system is stable. No problem when there is no raid array. Avalon / # fdisk -l Disk /dev/hdb: 251.0 GB, 251000193024 bytes 255 heads, 63 sectors/track, 30515 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x34101895 Device Boot Start End Blocks Id System /dev/hdb1 30143 30515 2996122+ 82 Linux swap / Solaris /dev/hdb2 3649 30142 212813024+ f W95 Ext'd (LBA) /dev/hdb3 * 1 3648 29302528+ 83 Linux Partition table entries are not in disk order Disk /dev/sda: 80.0 GB, 80026361856 bytes 255 heads, 63 sectors/track, 9729 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 1 9729 78148161 83 Linux Disk /dev/sdb: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 9729 78148161 83 Linux /dev/sdb2 9730 19457 78140160 83 Linux Avalon / # sda1 and sdb1 were moved from raid to ext file system. I think the issue is really related to raid and not due to hw.
The partition table order shouldn't matter at all. I don't know of any problems with the RAID code that cause stability issues, it's been rock solid everywhere that I've used it. I didn't notice before that you had 3 drives. Can you please use the dd command, making sure the '&' is on the end for backgrounding, on all 3 drives at the same time? Also, can you hook up a serial console and record any kernel output that happens with the stability issues and reboot? Does memtest pass on the machine ok?
(In reply to comment #10) > The partition table order shouldn't matter at all. > I don't know of any problems with the RAID code that cause stability issues, > it's been rock solid everywhere that I've used it. > I didn't notice before that you had 3 drives. > Can you please use the dd command, making sure the '&' is on the end for > backgrounding, on all 3 drives at the same time? > > Also, can you hook up a serial console and record any kernel output that > happens with the stability issues and reboot? Does memtest pass on the machine > ok? > (In reply to comment #10) > The partition table order shouldn't matter at all. > I don't know of any problems with the RAID code that cause stability issues, > it's been rock solid everywhere that I've used it. > I didn't notice before that you had 3 drives. > Can you please use the dd command, making sure the '&' is on the end for > backgrounding, on all 3 drives at the same time? > > Also, can you hook up a serial console and record any kernel output that > happens with the stability issues and reboot? Does memtest pass on the machine > ok? > Hello, sorry for the delay I have some trouble for the serial, I ll need an adapter but I hope that I could manage next week. I have tried again : dd if=/dev/sdb of=/dev/null bs=1M count=30000 & dd if=/dev/sda of=/dev/null bs=1M count=30000 & dd if=/dev/hdb of=/dev/null bs=1M count=30000 & No stability issue when the array is only with 1 drive. I have installed memtest86+, I will test it.
Created attachment 172936 [details] Reboot loop after issue from Serial connection
Hello, I have readded the second partition in my array. Every thing was fine during one day until that I need to reboot. Then it never boot correctly and always reboot at the end of the boot. I have attached the output from the serial as you have suggested. Then the reboot loop stop and in the other attachement you can see that my other drive has disappeared and my array is only with 1 partition. And my system is stable again. Igor
Created attachment 172938 [details] End of the loop
(In reply to comment #13) > Hello, Hi Igor, > I have readded the second partition in my array. Every thing was fine during > one day until that I need to reboot. > Then it never boot correctly and always reboot at the end of the boot. > I have attached the output from the serial as you have suggested. > > Then the reboot loop stop and in the other attachement you can see that my > other drive has disappeared and my array is only with 1 partition. And my > system is stable again. I noticed what's look like some HD drivers related errors. Please, try to reproduce the bug with the lastest stable 2.6.26-r3 kernel.
Created attachment 173070 [details] Issue with las kernel
Created attachment 173074 [details] After remove hdd of array from bios. Removing one hdd from array and back to stable system
(In reply to comment #17) > Created an attachment (id=173074) [edit] > After remove hdd of array from bios. > > Removing one hdd from array and back to stable system Thank you. Looking to your last attachments, please try adding the 'irqpoll' option at the boot command line. If the bug still occurs, you should install the sys-apps/smartmontools tools and try a long test on your IDE hard drive. Also, you may want to try with the 2.6.27-r4 kernel.
Created attachment 173781 [details] boot failure with irqpoll and all hard drives
Important notice stability is restored and boot is normal when any of the 2 drives in the array is removed.
Could you post your .config file?
Created attachment 174188 [details] .config file
How about your lspci output? Also, does your motherboard have an nforce pro chipset?
So irqpoll didn't make any difference? It still rebooted during startup? A few more ideas: Try booting with acpi=off - does that help anything? Please post output of "cat /proc/interrupts" from any successfully booted kernel Are there any BIOS upgrades available?
(In reply to comment #23) > How about your lspci output? Also, does your motherboard have an nforce pro > chipset? > Hello, Thank you for your support, here the lspci output (my motherboard is a m2n4-sli) igor@Avalon ~ $ sudo lspci Password: 00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3) 00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev f3) 00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2) 00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2) 00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3) 00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2) 00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2) 00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3) 00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev f2) 00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev f3) 00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) 00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) 00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) 00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:06.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10) 05:00.0 VGA compatible controller: nVidia Corporation G72 [GeForce 7300 LE] (rev a1) igor@Avalon ~ $
(In reply to comment #24) > So irqpoll didn't make any difference? It still rebooted during startup? > > > A few more ideas: > > Try booting with acpi=off - does that help anything? > > Please post output of "cat /proc/interrupts" from any successfully booted > kernel > > Are there any BIOS upgrades available? > I will try this on friday evening and post result. Thank you.
Created attachment 174547 [details] With acpi=off. Boot is not completed
Created attachment 174549 [details] cat /proc/interrupts. With only one hdd in the RAID 1 array
OK, so irqpoll didn't help at all? Are there any BIOS updates available? Also, you posted the boot logs from an irqpoll-enabled boot in comment #19. However, the logs themselves show that irqpoll was not enabled. Please could you retry and post the correct logs? Thanks.
Created attachment 174557 [details] new boot with irqpoll Boot is completed but seems that sda drive is not really recognize. fdisk -l take more than 5 min. And mdadm cannot find the partition to add in array (probably timeout).
Are there any BIOS updates available?
looking at igor's lspci output, it would seem i have very similar hardware... I also have the same problem. My 2 disk raid1 seems to crash during resync, causing a crapload of errors in dmesg. cromartie ~ # lspci 00:00.0 RAM memory: nVidia Corporation MCP61 Memory Controller (rev a1) 00:01.0 ISA bridge: nVidia Corporation MCP61 LPC Bridge (rev a2) 00:01.1 SMBus: nVidia Corporation MCP61 SMBus (rev a2) 00:01.2 RAM memory: nVidia Corporation MCP61 Memory Controller (rev a2) 00:02.0 USB Controller: nVidia Corporation MCP61 USB Controller (rev a3) 00:02.1 USB Controller: nVidia Corporation MCP61 USB Controller (rev a3) 00:04.0 PCI bridge: nVidia Corporation MCP61 PCI bridge (rev a1) 00:05.0 Audio device: nVidia Corporation MCP61 High Definition Audio (rev a2) 00:06.0 IDE interface: nVidia Corporation MCP61 IDE (rev a2) 00:07.0 Bridge: nVidia Corporation MCP61 Ethernet (rev a2) 00:08.0 IDE interface: nVidia Corporation MCP61 SATA Controller (rev a2) 00:08.1 IDE interface: nVidia Corporation MCP61 SATA Controller (rev a2) 00:09.0 PCI bridge: nVidia Corporation MCP61 PCI Express bridge (rev a2) 00:0b.0 PCI bridge: nVidia Corporation MCP61 PCI Express bridge (rev a2) 00:0d.0 VGA compatible controller: nVidia Corporation GeForce 6150SE nForce 430 (rev a2) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:05.0 FireWire (IEEE 1394): Agere Systems FW323 (rev 70) 01:0a.0 Network controller: RaLink RT2561/RT61 802.11g PCI cromartie ~ # hdparm -i /dev/sd{b,c} /dev/sdb: Model=ST31500341AS, FwRev=CC3G, SerialNo=6VS04QRA Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?16? CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=2930277168 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled Drive conforms to: unknown: ATA/ATAPI-4,5,6,7 * signifies the current active mode /dev/sdc: Model=ST31500341AS, FwRev=SD1B, SerialNo=9VS0757F Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?16? CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=2930277168 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled Drive conforms to: unknown: ATA/ATAPI-4,5,6,7 * signifies the current active mode ########################## # dmesg output attached below ##########################
Created attachment 187596 [details] dmesg output
After updateing to 2.6.29-r1 the problem appears to be solved.
Ok, closing then. Thanks for discovering that it went away with newer kernels, there were a lot of PCI changes, so I guess it's somewhere in that.
