I'm using xen vm, paravirtualized kernel. After upgrade from hardened-sources-2.6.36-r9 to 2.6.37-hardened-r5 kernel startet throws messages in log: WARNING: at block/blk-core.c:260 blk_remove_plug+0x1d/0x41() . I tried vanilla-sources-2.6.37.2 and problem still exists. [ 71.650055] WARNING: at block/blk-core.c:260 blk_remove_plug+0x1d/0x41() [ 71.650058] Modules linked in: iptable_mangle iptable_nat nf_nat iptable_filter ipt_addrtype xt_iprange tun sit tunnel4 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state ip6t_rt ip6table_filter xt_TCPMSS ip6table_mangle xt_owner xt_NFQUEUE xt_multiport xt_mark xt_hashlimit xt_connmark ip6_tables fuse usbcore [ 71.650103] Pid: 0, comm: swapper Tainted: G W 2.6.37.2 #1 [ 71.650105] Call Trace: [ 71.650110] [<c1023690>] ? warn_slowpath_common+0x6a/0x7d [ 71.650115] [<c10e98a2>] ? blk_remove_plug+0x1d/0x41 [ 71.650119] [<c10236b0>] ? warn_slowpath_null+0xd/0x10 [ 71.650124] [<c10e98a2>] ? blk_remove_plug+0x1d/0x41 [ 71.650129] [<c10e99de>] ? blk_stop_queue+0x8/0x12 [ 71.650134] [<c115bec0>] ? do_blkif_request+0x2b0/0x308 [ 71.650138] [<c115c104>] ? blkif_interrupt+0x1c5/0x1dc [ 71.650143] [<c1046890>] ? handle_IRQ_event+0x16/0x92 [ 71.650148] [<c1047d00>] ? handle_fasteoi_irq+0x5c/0x82 [ 71.650153] [<c113004b>] ? __xen_evtchn_do_upcall+0x9d/0xf7 [ 71.650157] [<c113101f>] ? xen_evtchn_do_upcall+0x16/0x23 [ 71.650162] [<c1007017>] ? xen_do_upcall+0x7/0xc [ 71.650167] [<c10013a7>] ? hypercall_page+0x3a7/0x1000 [ 71.650171] [<c1004fb4>] ? xen_safe_halt+0xf/0x19 [ 71.650176] [<c100b78a>] ? default_idle+0x29/0x47 [ 71.650180] [<c100598c>] ? cpu_idle+0x77/0x91 [ 71.650185] [<c131c61c>] ? start_kernel+0x294/0x29a [ 71.650190] [<c131d369>] ? xen_start_kernel+0x5b6/0x5be [ 71.650193] ---[ end trace 37c8426f18e2ec30 ]--- Reproducible: Always emerge --info FEATURES variable contains unknown value(s): Xkeepwork, Xnoclean, Xnostrip, Xtest Portage 2.1.9.25 (hardened/linux/x86, gcc-4.5.2, glibc-2.11.3-r0, 2.6.37.2 i686) ================================================================= System uname: Linux-2.6.37.2-i686-Pentium-R-_Dual-Core_CPU_E6300_@_2.80GHz-with-gentoo-1.12.14 Timestamp of tree: Sat, 12 Mar 2011 09:45:01 +0000 distcc 3.1 i686-pc-linux-gnu [disabled] ccache version 3.1.4 [enabled] app-shells/bash: 4.1_p9 dev-lang/python: 2.6.6-r2, 3.1.3-r1 dev-util/ccache: 3.1.4 dev-util/cmake: 2.8.1-r2 sys-apps/baselayout: 1.12.14-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.65-r1 sys-devel/automake: 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5, 4.5.2 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.81-r2 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=native -mfpmath=sse -pipe -fprefetch-loop-arrays -fomit-frame-pointer -fpeel-loops -frename-registers -fpredictive-commoning -fweb -ftree-vectorize -floop-interchange -ftree-loop-distribution -fivopts -floop-block -floop-strip-mine -s -fira-loop-pressure" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -march=native -mfpmath=sse -pipe -fprefetch-loop-arrays -fomit-frame-pointer -fpeel-loops -frename-registers -fpredictive-commoning -fweb -ftree-vectorize -floop-interchange -ftree-loop-distribution -fivopts -floop-block -floop-strip-mine -s -fira-loop-pressure" DISTDIR="/usr/portage/distfiles" FEATURES="Xkeepwork Xnoclean Xnostrip Xtest assume-digests binpkg-logs ccache collision-protect distlocks fail-clean fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" GENTOO_MIRRORS="http://trumpetti.atm.tut.fi/gentoo/" LANG="pl" LC_ALL="pl_PL.utf-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="pl" MAKEOPTS="-j1" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="-6 -O" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local/layman/sunrise /usr/portage/local/layman/science /usr/portage/local/layman/sping /usr/portage/local/layman/steev /usr/portage/local/layman/gnustep /usr/portage/local/layman/mgorny /usr/portage/local /usr/local/portage/miro/staging /usr/local/portage/miro/portage" SYNC="rsync://trumpetti.atm.tut.fi/gentoo-portage/" USE="acl acpi adns aio apache2 ares bash-completion bcmath bzip2 caps chroot clamav clamdtop cli cracklib crypt curl custom-cflags cxx dkim dri dsn enscript exiscan exiscan-acl fastcgi force-cgi-redirect glibc-omitfp graphite hardened iconv idn imap iproute2 ipv6 logrotate maildir mmap mmx modules mudflap ncurses network-cron nls nptl nptlonly openmp openssl pam pcre pic pppd readline server session slang spell srs sse sse2 sse3 ssl ssse3 subversion suhosin sysfs syslog threads threadsafe tools unicode urandom vhosts vim vim-pager vim-syntax x86 xattr xml xorg zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon auth_digest authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user cache cgid dav dav_fs dav_lock dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif status unique_id usertrack vhost_alias" APACHE2_MPMS="prefork" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="pl" NGINX_MODULES_HTTP="access auth_basic browser charset gzip limit_req limit_zone map proxy referer rewrite stub_status upstream_ip_hash userid" PHP_TARGETS="php5-3" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="geoip ipset psd tarpit" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
Created attachment 265633 [details] kernel config
Created attachment 265635 [details] dmesg
Please test with vanilla-sources-2.6.38
$ uname -r 2.6.38 Still the same problem. [ 1.523385] udevd (465): /proc/465/oom_adj is deprecated, please use /proc/465/oom_score_adj instead. [ 2.853472] EXT4-fs (xvda5): re-mounted. Opts: acl [ 4.723519] blkfront: xvda: empty write barrier op failed [ 4.723524] blkfront: xvda: barriers disabled [ 4.984114] ------------[ cut here ]------------ [ 4.984123] WARNING: at block/blk-core.c:352 blk_start_queue+0x1d/0x2f() [ 4.984126] Modules linked in: [ 4.984132] Pid: 0, comm: swapper Not tainted 2.6.38 #1 [ 4.984135] Call Trace: [ 4.984141] [<c102316b>] ? warn_slowpath_common+0x6a/0x7d [ 4.984145] [<c10eb1c4>] ? blk_start_queue+0x1d/0x2f [ 4.984150] [<c102318b>] ? warn_slowpath_null+0xd/0x10 [ 4.984155] [<c10eb1c4>] ? blk_start_queue+0x1d/0x2f [ 4.984160] [<c115eee2>] ? kick_pending_request_queues+0x19/0x27 [ 4.984165] [<c115f0b5>] ? blkif_interrupt+0x1c5/0x1dc [ 4.984171] [<c1046508>] ? handle_IRQ_event+0x16/0x92 [ 4.984176] [<c1047970>] ? handle_fasteoi_irq+0x5c/0x82 [ 4.984182] [<c1132899>] ? __xen_evtchn_do_upcall+0x9b/0xf5 [ 4.984187] [<c1133b17>] ? xen_evtchn_do_upcall+0x16/0x23 [ 4.984193] [<c1007317>] ? xen_do_upcall+0x7/0xc [ 4.984198] [<c10013a7>] ? hypercall_page+0x3a7/0x1000 [ 4.984202] [<c1004a24>] ? xen_safe_halt+0xf/0x19 [ 4.984207] [<c100bbce>] ? default_idle+0x29/0x47 [ 4.984212] [<c1005c8c>] ? cpu_idle+0x77/0x91 [ 4.984217] [<c132462e>] ? start_kernel+0x2a2/0x2a8 [ 4.984222] [<c1325406>] ? xen_start_kernel+0x5bd/0x5c5 [ 4.984225] ---[ end trace df33656ffc6bc32d ]--- [ 4.984228] ------------[ cut here ]------------ [ 4.984232] WARNING: at block/blk-core.c:260 blk_remove_plug+0x1d/0x41() [ 4.984234] Modules linked in: [ 4.984239] Pid: 0, comm: swapper Tainted: G W 2.6.38 #1 [ 4.984242] Call Trace: [ 4.984246] [<c102316b>] ? warn_slowpath_common+0x6a/0x7d [ 4.984251] [<c10eb091>] ? blk_remove_plug+0x1d/0x41 [ 4.984255] [<c102318b>] ? warn_slowpath_null+0xd/0x10 [ 4.984260] [<c10eb091>] ? blk_remove_plug+0x1d/0x41 [ 4.984264] [<c10eb0c0>] ? __blk_run_queue+0xb/0x61 [ 4.984269] [<c115eee2>] ? kick_pending_request_queues+0x19/0x27 [ 4.984273] [<c115f0b5>] ? blkif_interrupt+0x1c5/0x1dc [ 4.984279] [<c1046508>] ? handle_IRQ_event+0x16/0x92 [ 4.984283] [<c1047970>] ? handle_fasteoi_irq+0x5c/0x82 [ 4.984288] [<c1132899>] ? __xen_evtchn_do_upcall+0x9b/0xf5 [ 4.984293] [<c1133b17>] ? xen_evtchn_do_upcall+0x16/0x23 [ 4.984298] [<c1007317>] ? xen_do_upcall+0x7/0xc [ 4.984303] [<c10013a7>] ? hypercall_page+0x3a7/0x1000 [ 4.984307] [<c1004a24>] ? xen_safe_halt+0xf/0x19 [ 4.984312] [<c100bbce>] ? default_idle+0x29/0x47 [ 4.984317] [<c1005c8c>] ? cpu_idle+0x77/0x91 [ 4.984321] [<c132462e>] ? start_kernel+0x2a2/0x2a8 [ 4.984326] [<c1325406>] ? xen_start_kernel+0x5bd/0x5c5 [ 4.984329] ---[ end trace df33656ffc6bc32e ]--- [ 5.426529] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null) [ 5.437676] ------------[ cut here ]------------ [ 5.437684] WARNING: at block/blk-core.c:352 blk_start_queue+0x1d/0x2f() [ 5.437687] Modules linked in: [ 5.437692] Pid: 0, comm: swapper Tainted: G W 2.6.38 #1 [ 5.437695] Call Trace: [ 5.437702] [<c102316b>] ? warn_slowpath_common+0x6a/0x7d [ 5.437706] [<c10eb1c4>] ? blk_start_queue+0x1d/0x2f [ 5.437711] [<c102318b>] ? warn_slowpath_null+0xd/0x10 [ 5.437716] [<c10eb1c4>] ? blk_start_queue+0x1d/0x2f [ 5.437721] [<c115eee2>] ? kick_pending_request_queues+0x19/0x27 [ 5.437726] [<c115f0b5>] ? blkif_interrupt+0x1c5/0x1dc [ 5.437732] [<c1046508>] ? handle_IRQ_event+0x16/0x92 [ 5.437737] [<c1047970>] ? handle_fasteoi_irq+0x5c/0x82 [ 5.437743] [<c1132899>] ? __xen_evtchn_do_upcall+0x9b/0xf5 [ 5.437748] [<c1133b17>] ? xen_evtchn_do_upcall+0x16/0x23 [ 5.437754] [<c1007317>] ? xen_do_upcall+0x7/0xc [ 5.437758] [<c10013a7>] ? hypercall_page+0x3a7/0x1000 [ 5.437763] [<c1004a24>] ? xen_safe_halt+0xf/0x19 [ 5.437769] [<c100bbce>] ? default_idle+0x29/0x47 [ 5.437773] [<c1005c8c>] ? cpu_idle+0x77/0x91 [ 5.437778] [<c132462e>] ? start_kernel+0x2a2/0x2a8 [ 5.437783] [<c1325406>] ? xen_start_kernel+0x5bd/0x5c5 [ 5.437786] ---[ end trace df33656ffc6bc32f ]--- [ 5.437789] ------------[ cut here ]------------ [ 5.437793] WARNING: at block/blk-core.c:260 blk_remove_plug+0x1d/0x41() [ 5.437795] Modules linked in: [ 5.437800] Pid: 0, comm: swapper Tainted: G W 2.6.38 #1 [ 5.437803] Call Trace: [ 5.437807] [<c102316b>] ? warn_slowpath_common+0x6a/0x7d [ 5.437812] [<c10eb091>] ? blk_remove_plug+0x1d/0x41 [ 5.437816] [<c102318b>] ? warn_slowpath_null+0xd/0x10 [ 5.437821] [<c10eb091>] ? blk_remove_plug+0x1d/0x41 [ 5.437825] [<c10eb0c0>] ? __blk_run_queue+0xb/0x61 [ 5.437830] [<c115eee2>] ? kick_pending_request_queues+0x19/0x27 [ 5.437834] [<c115f0b5>] ? blkif_interrupt+0x1c5/0x1dc [ 5.437840] [<c1046508>] ? handle_IRQ_event+0x16/0x92 [ 5.437844] [<c1047970>] ? handle_fasteoi_irq+0x5c/0x82 [ 5.437849] [<c1132899>] ? __xen_evtchn_do_upcall+0x9b/0xf5 [ 5.437854] [<c1133b17>] ? xen_evtchn_do_upcall+0x16/0x23 [ 5.437859] [<c1007317>] ? xen_do_upcall+0x7/0xc [ 5.437864] [<c10013a7>] ? hypercall_page+0x3a7/0x1000 [ 5.437868] [<c1004a24>] ? xen_safe_halt+0xf/0x19 [ 5.437873] [<c100bbce>] ? default_idle+0x29/0x47 [ 5.437878] [<c1005c8c>] ? cpu_idle+0x77/0x91 [ 5.437882] [<c132462e>] ? start_kernel+0x2a2/0x2a8 [ 5.437887] [<c1325406>] ? xen_start_kernel+0x5bd/0x5c5 [ 5.437890] ---[ end trace df33656ffc6bc330 ]--- [ 5.658496] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null) [ 5.733584] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5.823870] EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null)
Should I see xen_start_kernel in a vanilla kernel oops?
Disregard that comment. I can't see to find anything useful on your issue, can you please take to upstream at bugzilla.kernel.org
Unfortunally, i can't try git-sources (with compilled debug) now. I'll try to do it as soon as it be possible.
I found another report of this bug in a wiki, but without further information. You can try git-sources, just in case, but I think this bug should be reported upstream (if you do so, make sure that kernel@gentoo.org is added to the CC list, so that the bug status can be tracked). If you'd like, I could report it to b.k.o The warning is caused by "WARN_ON(!irqs_disabled)", in 2 functions inside blk-core.c, which means that for some reason the Interrupt Flag is probably not set properly, although this code runs 'inside' a spin_lock_irqsave/spin_lock_irqrestore block.
I'm not reporting upstream because i don't have vnc access to xen VM. If VM hangs i need a couple hours to get assistance from admin. I can't do bisect etc, and i can't do quick test if new patch will be available :(
Ok, # uname -a Linux hermes 2.6.39-rc5-git7 and i got: 71.870518] ------------[ cut here ]------------ [ 71.870529] WARNING: at block/blk-core.c:239 blk_start_queue+0x20/0x32() [ 71.870532] Modules linked in: iptable_filter xt_iprange tun sit tunnel4 xt_TCPMSS ip6table_mangle nf_conntrack_ipv6 nf_defrag_ipv6 xt_state ip6t_rt ip6table_filter xt_owner xt_NFQUEUE xt_multiport xt_mark xt_hashlimit xt_connmark ip6_tables fuse usbcore [ 71.870578] Pid: 0, comm: swapper Tainted: G W 2.6.39-rc5-git7 #1 [ 71.870581] Call Trace: [ 71.870588] [<c1025763>] warn_slowpath_common+0x65/0x7a [ 71.870593] [<c10f2879>] ? blk_start_queue+0x20/0x32 [ 71.870598] [<c1025787>] warn_slowpath_null+0xf/0x13 [ 71.870602] [<c10f2879>] blk_start_queue+0x20/0x32 [ 71.870608] [<c116b4ba>] kick_pending_request_queues+0x1c/0x2a [ 71.870613] [<c116b68d>] blkif_interrupt+0x1c5/0x1db [ 71.870619] [<c104a6ec>] handle_irq_event_percpu+0x28/0x114 [ 71.870624] [<c104a7f4>] handle_irq_event+0x1c/0x28 [ 71.870628] [<c104bc39>] handle_fasteoi_irq+0x59/0x6f [ 71.870633] [<c113dc93>] __xen_evtchn_do_upcall+0x10f/0x198 [ 71.870638] [<c113ea11>] xen_evtchn_do_upcall+0x19/0x27 [ 71.870644] [<c1237197>] xen_do_upcall+0x7/0xc [ 71.870650] [<c104007b>] ? posix_clock_release+0x23/0x35 [ 71.870655] [<c10013a7>] ? hypercall_page+0x3a7/0x1000 [ 71.870660] [<c1004cd7>] ? xen_safe_halt+0x12/0x1d [ 71.870665] [<c100b77d>] default_idle+0x2c/0x4c [ 71.870670] [<c1005eae>] cpu_idle+0x74/0x8e [ 71.870676] [<c122edba>] rest_init+0x52/0x54 [ 71.870680] [<c133c668>] start_kernel+0x29d/0x2a4 [ 71.870685] [<c133c076>] i386_start_kernel+0x65/0x67 [ 71.870690] [<c133d42e>] xen_start_kernel+0x568/0x570 [ 71.870693] ---[ end trace 5c490816ac9331f5 ]--- problem still exists.
I just found a discussion in xen-devel about the same bug, but in a 2.6.32 kernel. [1] It seems that they didn't fix it, but they proposed a solution: ********************************************************** Either: Can you add a bunch of similar WARN_ONs along that path? Or: This lock is quite coarse-grained. The lock only matters for queue access, and we know irqs are reenabled, so no need for flags. In fact we only need to spin_lock_irq around the __blk_end_ calls and kick_pending_. ********************************************************** I can provide a patch for the second solution, if you want, to silence the warning, or you can try to debug it (with WARN_ONs). Of course, you can also reply to the thread at the xen-devel mailing list, to inform upstream that the bug isn't solved. And, you can also file a bug at the Kernel Bugzilla. [1] http://lists.xensource.com/archives/html/xen-devel/2010-09/msg00165.html
Continue to work this on the upstream mailing list. Any fix identified we will try to back port
Created attachment 278407 [details] message.log from boot, kernel 3.0.0-rc4
