I have for bug hunting purposes enabled various debugging output in the kernel config related to spinlocks and mutexes. Now today I took a look in the dmesg for unrelated reasons, and found the output below. I have seen this with hardened-sources-2.6.36-r[78] and replicated it with vanilla-sources-2.6.36.2. This comes every time, and the system seems to work fine even with it, but as I have had various issues with the IO-subsystem hanging, most often related to load I thought maybe I should report this somewhere. [ 29.601576] ======================================================= [ 29.601590] [ INFO: possible circular locking dependency detected ] [ 29.601596] 2.6.36.2 #1 [ 29.601599] ------------------------------------------------------- [ 29.601604] kworker/0:1/11 is trying to acquire lock: [ 29.601609] (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811915d6>] revalidate_disk+0x56/0x90 [ 29.601623] [ 29.601623] but task is already holding lock: [ 29.601629] (&ap->scsi_scan_mutex){+.+.+.}, at: [<ffffffff81511350>] ata_scsi_dev_rescan+0x30/0x130 [ 29.601642] [ 29.601643] which lock already depends on the new lock. [ 29.601643] [ 29.601651] [ 29.601651] the existing dependency chain (in reverse order) is: [ 29.601658] [ 29.601658] -> #3 (&ap->scsi_scan_mutex){+.+.+.}: [ 29.601668] [<ffffffff810bfea3>] lock_acquire+0x93/0x180 [ 29.601676] [<ffffffff817fdedb>] mutex_lock_nested+0x4b/0x380 [ 29.601684] [<ffffffff81511350>] ata_scsi_dev_rescan+0x30/0x130 [ 29.601692] [<ffffffff810a25f8>] process_one_work+0x188/0x510 [ 29.601701] [<ffffffff810a44a9>] worker_thread+0x169/0x340 [ 29.601708] [<ffffffff810a9216>] kthread+0x96/0xa0 [ 29.601716] [<ffffffff81038ad4>] kernel_thread_helper+0x4/0x10 [ 29.601724] [ 29.601724] -> #2 ((&ap->scsi_rescan_task)){+.+.+.}: [ 29.601733] [<ffffffff810bfea3>] lock_acquire+0x93/0x180 [ 29.601740] [<ffffffff810a25e2>] process_one_work+0x172/0x510 [ 29.601748] [<ffffffff810a44a9>] worker_thread+0x169/0x340 [ 29.601755] [<ffffffff810a9216>] kthread+0x96/0xa0 [ 29.602313] [<ffffffff81038ad4>] kernel_thread_helper+0x4/0x10 [ 29.602321] [ 29.602322] -> #1 (events){+.+.+.}: [ 29.602330] [<ffffffff810bfea3>] lock_acquire+0x93/0x180 [ 29.602337] [<ffffffff810a3247>] flush_work+0x127/0x1c0 [ 29.602344] [<ffffffff810a50a3>] schedule_on_each_cpu+0xe3/0x120 [ 29.602352] [<ffffffff8111dff0>] lru_add_drain_all+0x10/0x20 [ 29.602359] [<ffffffff8118ace8>] invalidate_bdev+0x28/0x40 [ 29.602366] [<ffffffff81190d6a>] __invalidate_device+0x4a/0x60 [ 29.602374] [<ffffffff813771b1>] invalidate_partition+0x31/0x50 [ 29.602382] [<ffffffff811bcede>] rescan_partitions+0x5e/0x550 [ 29.602390] [<ffffffff81192196>] __blkdev_get+0x1a6/0x3c0 [ 29.602397] [<ffffffff811923bb>] blkdev_get+0xb/0x10 [ 29.602404] [<ffffffff811bce65>] register_disk+0x155/0x170 [ 29.602412] [<ffffffff813773e1>] add_disk+0xa1/0x160 [ 29.602418] [<ffffffff814fe3c1>] sd_probe_async+0x111/0x1c0 [ 29.602426] [<ffffffff810b177d>] async_run_entry_fn+0x7d/0x180 [ 29.602436] [<ffffffff810a25f8>] process_one_work+0x188/0x510 [ 29.602443] [<ffffffff810a44a9>] worker_thread+0x169/0x340 [ 29.602450] [<ffffffff810a9216>] kthread+0x96/0xa0 [ 29.602457] [<ffffffff81038ad4>] kernel_thread_helper+0x4/0x10 [ 29.602465] [ 29.602465] -> #0 (&bdev->bd_mutex){+.+.+.}: [ 29.602474] [<ffffffff810bf429>] __lock_acquire+0x17f9/0x1c60 [ 29.602483] [<ffffffff810bfea3>] lock_acquire+0x93/0x180 [ 29.602493] [<ffffffff817fdedb>] mutex_lock_nested+0x4b/0x380 [ 29.602504] [<ffffffff811915d6>] revalidate_disk+0x56/0x90 [ 29.602516] [<ffffffff814fb892>] sd_rescan+0x22/0x40 [ 29.602526] [<ffffffff814f3906>] scsi_rescan_device+0xb6/0x140 [ 29.602536] [<ffffffff815113ea>] ata_scsi_dev_rescan+0xca/0x130 [ 29.602546] [<ffffffff810a25f8>] process_one_work+0x188/0x510 [ 29.602558] [<ffffffff810a44a9>] worker_thread+0x169/0x340 [ 29.602569] [<ffffffff810a9216>] kthread+0x96/0xa0 [ 29.602579] [<ffffffff81038ad4>] kernel_thread_helper+0x4/0x10 [ 29.602591] [ 29.602591] other info that might help us debug this: [ 29.602591] [ 29.603876] 3 locks held by kworker/0:1/11: [ 29.604286] #0: (events){+.+.+.}, at: [<ffffffff810a258a>] process_one_work+0x11a/0x510 [ 29.604701] #1: ((&ap->scsi_rescan_task)){+.+.+.}, at: [<ffffffff810a258a>] process_one_work+0x11a/0x510 [ 29.605129] #2: (&ap->scsi_scan_mutex){+.+.+.}, at: [<ffffffff81511350>] ata_scsi_dev_rescan+0x30/0x130 [ 29.605555] [ 29.605555] stack backtrace: [ 29.606389] Pid: 11, comm: kworker/0:1 Tainted: G W 2.6.36.2 #1 [ 29.606805] Call Trace: [ 29.607217] [<ffffffff810bc6c7>] print_circular_bug+0xe7/0xf0 [ 29.607624] [<ffffffff810bf429>] __lock_acquire+0x17f9/0x1c60 [ 29.608036] [<ffffffff810be010>] ? __lock_acquire+0x3e0/0x1c60 [ 29.608452] [<ffffffff810bfea3>] lock_acquire+0x93/0x180 [ 29.608870] [<ffffffff811915d6>] ? revalidate_disk+0x56/0x90 [ 29.609284] [<ffffffff817ffe40>] ? _raw_spin_unlock+0x30/0x60 [ 29.609698] [<ffffffff811908f0>] ? bdev_test+0x0/0x20 [ 29.610105] [<ffffffff8107a431>] ? get_parent_ip+0x11/0x50 [ 29.610521] [<ffffffff811915d6>] ? revalidate_disk+0x56/0x90 [ 29.610928] [<ffffffff817fdedb>] mutex_lock_nested+0x4b/0x380 [ 29.611339] [<ffffffff811915d6>] ? revalidate_disk+0x56/0x90 [ 29.611750] [<ffffffff81384917>] ? kobject_put+0x27/0x60 [ 29.612156] [<ffffffff81376fa8>] ? disk_get_part+0x18/0xd0 [ 29.612564] [<ffffffff811915d6>] revalidate_disk+0x56/0x90 [ 29.612970] [<ffffffff814fb892>] sd_rescan+0x22/0x40 [ 29.613374] [<ffffffff814f3906>] scsi_rescan_device+0xb6/0x140 [ 29.613774] [<ffffffff815113ea>] ata_scsi_dev_rescan+0xca/0x130 [ 29.614176] [<ffffffff810a25f8>] process_one_work+0x188/0x510 [ 29.614579] [<ffffffff810a258a>] ? process_one_work+0x11a/0x510 [ 29.614980] [<ffffffff81511320>] ? ata_scsi_dev_rescan+0x0/0x130 [ 29.615386] [<ffffffff810a44a9>] worker_thread+0x169/0x340 [ 29.615785] [<ffffffff810a4340>] ? worker_thread+0x0/0x340 [ 29.616186] [<ffffffff810a9216>] kthread+0x96/0xa0 [ 29.616587] [<ffffffff81038ad4>] kernel_thread_helper+0x4/0x10 [ 29.616984] [<ffffffff81075269>] ? finish_task_switch+0x69/0xf0 [ 29.617383] [<ffffffff81800480>] ? restore_args+0x0/0x30 [ 29.617783] [<ffffffff810a9180>] ? kthread+0x0/0xa0 [ 29.618178] [<ffffffff81038ad0>] ? kernel_thread_helper+0x0/0x10
Created attachment 259607 [details] .config
Created attachment 259609 [details] dmesg
$ emerge --info Portage 2.2.0_alpha14 (hardened/linux/amd64, gcc-4.5.2, glibc-2.12.1-r3, 2.6.36.2 x86_64) ================================================================= System uname: Linux-2.6.36.2-x86_64-Intel-R-_Core-TM-_i7_CPU_920_@_2.67GHz-with-gentoo-2.0.1 Timestamp of tree: Tue, 11 Jan 2011 07:30:14 +0000 distcc 3.1 x86_64-pc-linux-gnu [disabled] ccache version 3.1.4 [disabled] app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.1::Mine, 3.1.3 dev-util/ccache: 3.1.4 dev-util/cmake: 2.8.1-r2 sys-apps/baselayout: 2.0.1-r1 sys-apps/openrc: 0.6.8 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13, 2.68 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.21 sys-devel/gcc: 4.5.2 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) Repositories: gentoo gamerlay-stable x11 mozilla Mine ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -ggdb -mtune=native" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe -ggdb -mtune=native" DISTDIR="/var/portage/distfiles" FEATURES="assume-digests binpkg-logs buildpkg distlocks fixlafiles fixpackages metadata-transfer news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict test unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox" GENTOO_MIRRORS="ftp://ftp.sunet.se/pub/os/Linux/distributions/gentoo" INSTALL_MASK="*.la" LANG="sv_SE.UTF-8" LC_ALL="C" LDFLAGS="-Wl,--as-needed -Wl,-O1 -Wl,--sort-common -Wl,--warn-once,--hash-style=gnu" LINGUAS="sv en" MAKEOPTS="-j16 -l15" PKGDIR="/var/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/var/portage" PORTDIR_OVERLAY="/var/overlays/layman/gamerlay /var/overlays/layman/x11 /var/overlays/layman/mozilla /var/overlays/mine" SYNC="rsync://liten.csbnet.se/gentoo-portage" USE="X a52 aac accessibility acl acpi alsa amd64 amr amrnb amrwb applet archive asyncns autoipd avahi bash-completion bluetooth branding bzip2 cairo caps ccache cdaudio cdda cdr cleartype cli clutter connection-sharing consolekit coverart cracklib crypt cups cxx dbus device-mapper devicekit devkit dhcpcd digitalradio djvu dri dts dvd dvdr dvi eds enca encode eselect evo exif faac faad fat fbcondecor ffmpeg fftw flac fluidsynth fontconfig fuse gdbm gdm gdu gif gimp glib gmp gnome gnome-keyring gphoto2 gpm grammar graphite gsf gsm gstreamer gtk gudev hardened hpn ical iconv iconvacl icq icu id3tag idn ieee1394 iptc ipv6 jabber jack java6 jingle jpeg jpeg2k justify kate kvm lcms libffi libnotify libsamplerate logrotate lvm lvm2 lzma mad maps math matroska md mdadm midi mms mmx mmxext mng moonlight mp2 mp3 mpeg mpi msn mtp mudflap multilib musepack musicbrainz natspec nautilus ncurses network-cron networkmanager nfs nls nntp nptl nptlonly ntfs ntp nut offensive ogg openal opencore-amr opengl openmp openntpd ots pam pango parted pcre pdf perl pic pidgin playlist png policykit pppd pulseaudio python qt3support quicktime raw readline rrdcgi rtmp samba schroedinger seed sensord session smp sms speex spell sse sse2 sse3 ssl ssse3 startup-notification subversion svg sysfs test tex theora thesaurus threads tiff totem truetype udev unicode upnp urandom usb userlocales v4l2 vaapi vhook videos vim-syntax vorbis webkit wmf x264 xattr xcb xcomposite xml xmp xmpp xorg xrandr xscreensaver xulrunner xv xvid xvmc zeroconf zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="sv en" PHP_TARGETS="php5-3" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
just confirmed that I got the same with vanilla-sources-2.6.37
Err, if this is happening on vanilla-sources shouldn't you just report it upstream? I resolve this as UPSTREAM. If you still think that it has to do with us, you're free to reopen it. (You can read the REPORTING-BUGS file inside every kernel sources, to see how to report kernel bugs). Thank you for your report.
