Created attachment 77435 [details]
complete dmesg output of 2.6.11-hardened-r15
Created attachment 77436 [details]
complete dmesg output of 2.6.14-hardened-r3
Created attachment 77437 [details]
complete /proc/scsi/aic79xx/0 of 2.6.11-hardened-r15
Created attachment 77438 [details]
complete /proc/scsi/aic79xx/1 of 2.6.11-hardened-r15
Created attachment 77439 [details]
complete /proc/scsi/aic79xx/0 of 2.6.14-hardened-r3
Created attachment 77440 [details]
complete /proc/scsi/aic79xx/1 of 2.6.14-hardened-r3
Created attachment 77441 [details]
kernel-config-x86-2.6.11-hardened-r15
Created attachment 77442 [details]
kernel-config-x86-2.6.14-hardened-r3
Do you rely on any features of the hardened-sources kernel? We really need to get this reported upstream, but that would involve you testing the latest 'vanilla' development kernel first (currently 2.6.16-rc1). (In reply to comment #9) > Do you rely on any features of the hardened-sources kernel? Not really. > We really need to > get this reported upstream, but that would involve you testing the latest > 'vanilla' development kernel first (currently 2.6.16-rc1). OK, I'll test that kernel. But since it's the main server of our workgroup and the colleagues already grumble about me rebooting all the time I'll wait with the reboot until Wednesday 11:30 CET (our time slot for planned reboots). After checking /proc/scsc/aic79xx/0 under kernel 2.6.16-rc1 would you advice me to leave that kernel running (What's about it's stability and secureness?) or boot back to 2.6.11-hardened-r15? (In reply to comment #10) > After checking /proc/scsc/aic79xx/0 under kernel 2.6.16-rc1 would you advice me > to leave that kernel running (What's about it's stability and secureness?) or > boot back to 2.6.11-hardened-r15? This question certainly isn't relevant if the vanilla kernel has the problem too. (In reply to comment #9) > but that would involve you testing the latest > 'vanilla' development kernel first (currently 2.6.16-rc1). I did the test (vanilla-sources-2.6.16_rc1) and the problem remains (I'll attach the logs and kernel-config). So it's an upstream problem. Or could anything of this be the cause: 1) The change from 2.4.28-hardened-r4 to 2.6.11-hardened-r1 kernel I did with this command: genkernel --kernel-config=/etc/kernels/kernel-config-x86-2.4.28-hardened-r4 --menuconfig all But this kernel and 2.6.11-hardened-r13, 2.6.11-hardened-r15 didn't show the problem. 2) The first kernel showing the problem was the first one built after gcc upgrade (3.3.6 -> 3.4.4-r1). Created attachment 78068 [details]
complete dmesg output of vanilla 2.6.16-rc1
Created attachment 78069 [details]
complete /proc/scsi/aic79xx/0 of vanilla 2.6.16-rc1
The differences in the first line of the
"Target 1 Negotiation Settings" in /proc/scsi/aic79xx/0 for
- 2.6.16-rc1:
Target 1 Negotiation Settings
User: 160.000MB/s transfers (80.000MHz RDSTRM|DT|IU|QAS, 16bit)
- 2.6.14-hardened-r3:
Target 1 Negotiation Settings
User: 320.000MB/s transfers (160.000MHz RDSTRM|DT|IU|QAS, 16bit)
- 2.6.11-hardened-r15:
Target 1 Negotiation Settings
User: 160.000MB/s transfers (80.000MHz DT|IU|QAS, 16bit)
are caused by me changing the default for target 1 in the Adaptec BIOS from 320 to 160 (didn't help for 2.6.14-hardened-r3 and 2.6.16-rc1 and 2.6.11-hardened-r15 works with both) and activating CONFIG_AIC79XX_ENABLE_RD_STRM in the kernel.
Created attachment 78070 [details]
kernel-config-x86-2.6.16-rc1
Ok, thanks for testing that. The next step is to report the problem to the Linux SCSI developers. Please file a bug for this against 2.6.16-rc1 at http://bugzilla.kernel.org and post the new bug URL here. (In reply to comment #16) > Please file a bug for this against 2.6.16-rc1 at > http://bugzilla.kernel.org and post the new bug URL here. > Done: http://bugzilla.kernel.org/show_bug.cgi?id=5955 Just a note on my comment #4 in http://bugzilla.kernel.org/show_bug.cgi?id=5955 of today: I also tried 2.6.14-hardened-r8 with the two changes in the Adaptec BIOS but had no success. But that's just for the record since I'm happy now running vanilla kernel 2.6.17-rc5. Fixed, see upstream bug |
Server: MB Intel SE7501WV2, Dual Xeon 3.066 GHz, hyperthreading, 4096 Mbyte RAM ext. RAID: Stor^3 VISMASTER IDE-SCSI 960GB, raid5 with one spare disk Model: IFT-7200 system disk: SEAGATE Model: ST3146807LC emerge --info Portage 2.0.53 (default-linux/x86/2005.0, gcc-3.4.4, glibc-2.3.5-r2, 2.6.11-hardened-r15 i686) ================================================================= System uname: 2.6.11-hardened-r15 i686 Intel(R) Xeon(TM) CPU 3.06GHz Gentoo Base System version 1.6.13 ccache version 2.3 [disabled] dev-lang/python: 2.4.2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O3 -march=pentium4 -funroll-loops -fprefetch-loop-arrays -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/share/config /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/ghostscript/7.07/lib/gs_init.ps /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/web2c/texmf.cnf /usr/share/texmf/xdvi/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O3 -march=pentium4 -funroll-loops -fprefetch-loop-arrays -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://hanf.informatik.uni-stuttgart.de/gentoo http://mirrors.sec.informatik.tu-darmstadt.de/gentoo http://ftp.uni-erlangen.de/pub/mirrors/gentoo http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/ http://www.mirror.ac.uk/sites/www.ibiblio.org/gentoo/" LANG="en_US" LINGUAS="en-US de fr ru" MAKEOPTS="-j6" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="x86 X aalib alsa apache2 apm arts audiofile avi berkdb bitmap-fonts bonobo bzip2 cdr crypt cups curl eds emboss encode esd ethereal exif expat fam flac font-server foomaticdb gcj gd gdbm gif gimpprint glut gmp gnome gpm gstreamer gtk gtk2 gtkhtml idn imagemagick imap imlib java jpeg junit kde lcms ldap libcaca libg++ libwww mad mbox mikmod mng motif mozilla mp3 mpeg mysql ncurses nls ogg oggvorbis opengl oss pam pcre pdflib perl png ppds python qt quicktime readline recode samba sdl slang spell ssl svga tcltk tcpd tetex tiff truetype truetype-fonts type1-fonts udev userlocales vorbis wmf xine xml xml2 xmms xv zlib linguas_en-US linguas_de linguas_fr linguas_ru userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LC_ALL, LDFLAGS Shortly after the last kernel upgrade (from 2.6.11-hardened-r15 to 2.6.14-hardened-r3) we noticed that disk io with our RAID was really slow (the speed of the system disk was ok). According to "hdparm -t" (10 times average) the slow down is from about 42MB/sec under 2.6.11-hardened-r15 to about 2.2MB/sec under 2.6.14-hardened-r3 and under heavy io the system even hangs for 30 seconds to 1 minute. The reason for this slow io is a failed negotiation between scsi driver aic79xx and RAID: (from dmesg) Linux version 2.6.11-hardened-r15 (root@kresse) (gcc version 3.3.5-20050130 (Gentoo Linux 3.3.5.20050130-r1, ssp-3.3.5.20050130-1, pie-8.7.7.1)) #1 SMP Wed Jul 27 10:33:06 CEST 2005 ... scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.11 <Adaptec AIC7902 Ultra320 SCSI adapter> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs (scsi0:A:1): 160.000MB/s transfers (80.000MHz DT, 16bit) Vendor: IFT Model: IFT-7200 Rev: 231T Type: Direct-Access ANSI SCSI revision: 04 scsi0:A:1:0: Tagged Queuing enabled. Depth 48 scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.11 <Adaptec AIC7902 Ultra320 SCSI adapter> aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs (scsi1:A:1): 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit) Vendor: SEAGATE Model: ST3146807LC Rev: 0007 Type: Direct-Access ANSI SCSI revision: 03 scsi1:A:1:0: Tagged Queuing enabled. Depth 48 Vendor: ESG-SHV Model: SCA HSBP M18 Rev: 0.07 Type: Processor ANSI SCSI revision: 02 SCSI device sda: 1444319232 512-byte hdwr sectors (739491 MB) SCSI device sda: drive cache: write through SCSI device sda: 1444319232 512-byte hdwr sectors (739491 MB) SCSI device sda: drive cache: write through /dev/scsi/host0/bus0/target1/lun0: p1 p2 p3 Attached scsi disk sda at scsi0, channel 0, id 1, lun 0 SCSI device sdb: 286749488 512-byte hdwr sectors (146816 MB) SCSI device sdb: drive cache: write back SCSI device sdb: 286749488 512-byte hdwr sectors (146816 MB) SCSI device sdb: drive cache: write back /dev/scsi/host1/bus0/target1/lun0: p1 p2 p3 Attached scsi disk sdb at scsi1, channel 0, id 1, lun 0 ... -------------------------------------------------------- Linux version 2.6.14-hardened-r3 (root@kresse) (gcc version 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)) #1 SMP PREEMPT Tue Jan 17 15:16:47 CET 2006 ... scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.11 <Adaptec AIC7902 Ultra320 SCSI adapter> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs Vendor: IFT Model: IFT-7200 Rev: 231T Type: Direct-Access ANSI SCSI revision: 04 target0:0:1: asynchronous. scsi0:A:1:0: Tagged Queuing enabled. Depth 48 target0:0:1: Beginning Domain Validation target0:0:1: wide asynchronous. target0:0:1: Domain Validation skipping write tests target0:0:1: Ending Domain Validation ACPI: PCI Interrupt 0000:04:07.1[B] -> GSI 49 (level, low) -> IRQ 19 scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.11 <Adaptec AIC7902 Ultra320 SCSI adapter> aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs Vendor: SEAGATE Model: ST3146807LC Rev: 0007 Type: Direct-Access ANSI SCSI revision: 03 target1:0:1: asynchronous. scsi1:A:1:0: Tagged Queuing enabled. Depth 48 target1:0:1: Beginning Domain Validation target1:0:1: wide asynchronous. target1:0:1: FAST-160 WIDE SCSI 320.0 MB/s DT IU QAS RDSTRM WRFLOW PCOMP (6.25 ns, offset 63) target1:0:1: Ending Domain Validation Vendor: ESG-SHV Model: SCA HSBP M18 Rev: 0.07 Type: Processor ANSI SCSI revision: 02 target1:0:6: asynchronous. target1:0:6: Beginning Domain Validation target1:0:6: Ending Domain Validation SCSI device sda: 1444319232 512-byte hdwr sectors (739491 MB) SCSI device sda: drive cache: write through SCSI device sda: 1444319232 512-byte hdwr sectors (739491 MB) SCSI device sda: drive cache: write through sda: sda1 sda2 sda3 Attached scsi disk sda at scsi0, channel 0, id 1, lun 0 SCSI device sdb: 286749488 512-byte hdwr sectors (146816 MB) SCSI device sdb: drive cache: write back SCSI device sdb: 286749488 512-byte hdwr sectors (146816 MB) SCSI device sdb: drive cache: write back sdb: sdb1 sdb2 sdb3 Attached scsi disk sdb at scsi1, channel 0, id 1, lun 0 -------------------------------------------------------- (kernel 2.6.14-hardened-r3, from /proc/scsi/aic79xx/0, the RAID) ... Target 1 Negotiation Settings User: 320.000MB/s transfers (160.000MHz RDSTRM|DT|IU|QAS, 16bit) Goal: 3.300MB/s transfers Curr: 3.300MB/s transfers Channel A Target 1 Lun 0 Settings Commands Queued 46461 Commands Active 0 Command Openings 48 Max Tagged Openings 48 Device Queue Frozen Count 0 ... -------------------------------------------------------- (kernel 2.6.14-hardened-r3, from /proc/scsi/aic79xx/1, the system disk) ... Target 1 Negotiation Settings User: 320.000MB/s transfers (160.000MHz RDSTRM|DT|IU|QAS, 16bit) Goal: 320.000MB/s transfers (160.000MHz RDSTRM|DT|IU|QAS, 16bit) Curr: 320.000MB/s transfers (160.000MHz RDSTRM|DT|IU|QAS, 16bit) Channel A Target 1 Lun 0 Settings Commands Queued 31169 Commands Active 0 Command Openings 48 Max Tagged Openings 48 Device Queue Frozen Count 0 ... -------------------------------------------------------- (Kernel 2.6.11-hardened-r15, from /proc/scsi/aic79xx/0, the RAID) ... Target 1 Negotiation Settings User: 160.000MB/s transfers (80.000MHz DT|IU|QAS, 16bit) Goal: 160.000MB/s transfers (80.000MHz DT, 16bit) Curr: 160.000MB/s transfers (80.000MHz DT, 16bit) Transmission Errors 0 Channel A Target 1 Lun 0 Settings Commands Queued 45033 Commands Active 0 Command Openings 48 Max Tagged Openings 48 Device Queue Frozen Count 0 ... -------------------------------------------------------- (Kernel 2.6.11-hardened-r15, from /proc/scsi/aic79xx/1, the system disk) ... Target 1 Negotiation Settings User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit) Goal: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit) Curr: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit) Transmission Errors 0 Channel A Target 1 Lun 0 Settings Commands Queued 23848 Commands Active 0 Command Openings 48 Max Tagged Openings 48 Device Queue Frozen Count 0 ... --------------------------------------------------------