After upgrading to sys-kernel/gentoo-sources-3.10.7 I sometimes get a kernel panic (see screenshots attached). x11-base/xorg-server-1.14.2 was built with the following: USE="ipv6 kdrive nptl suid udev xorg -dmx -doc -minimal (-selinux) -static-libs -tslib -xnest -xvfb" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,lazy" x11-drivers/xf86-video-nouveau-1.0.8 was built with the following: USE="" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,lazy $ lspci 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04) 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04) 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4) 00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 2 (rev b4) 00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b4) 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b4) 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation QM67 Express Chipset Family LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller (rev 04) 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation GF119M [Quadro NVS 4200M] (rev ff) 03:00.0 Network controller: Intel Corporation Centrino Wireless-N 1000 [Condor Peak] 0d:00.0 System peripheral: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 08) 0d:00.3 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 PCIe IEEE 1394 Controller (rev 04) Reproducible: Sometimes Portage 2.1.12.2 (default/linux/amd64/13.0/desktop/gnome, gcc-4.7.3, glibc-2.15-r3, 3.9.11-gentoo-r1 x86_64) ================================================================= System uname: Linux-3.9.11-gentoo-r1-x86_64-Intel-R-_Core-TM-_i7-2720QM_CPU_@_2.20GHz-with-gentoo-2.2 KiB Mem: 8055748 total, 5807464 free KiB Swap: 0 total, 0 free Timestamp of tree: Tue, 27 Aug 2013 08:45:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-java/java-config: 2.1.12-r1 dev-lang/python: 2.7.5-r2, 3.2.5-r2 dev-util/cmake: 2.8.10.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.11.6, 1.12.6 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3, 4.7.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.7 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo sunrise java steam-overlay systemd-love x-portage hibiscus-gentoo-overlay ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE freedist as-is" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=corei7-avx -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.0/conf /usr/share/maven-bin-3.1/conf /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=corei7-avx -O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/" LANG="de_DE.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j8" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/sunrise /var/lib/layman/java /var/lib/layman/steam /var/lib/layman/systemd-love /usr/local/portage /home/fabian/workspace/Privat/gentoo/hibiscus-ebuild/hibiscus-gentoo-overlay.git" SYNC="rsync://rsync.de.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa amd64 avx bash-completion berkdb bluetooth branding bzip2 cairo cdda cdr cli colord consolekit cracklib crypt cups cxx dbus dri dts dvb dvd dvdr eds emboss encode evo exif fam firefox flac fortran gdbm gif gnome gnome-keyring gnome-online-accounts gpm gstreamer gtk hdaps iconv idn introspection ipv6 jabber jpeg lastfm lcms libnotify libsecret mad mmx mng modemmanager modules mp3 mp4 mpeg mudflap multilib nautilus ncurses networkmanager nfsv4 nls nptl ogg opengl openmp opus pam pango pcre pcsc-lite pdf png policykit ppds pulseaudio qt3support qt4 readline rss sdl session socialweb spell sse sse2 sse3 sse4 sse4_1 sse4_2 ssl ssse3 startup-notification svg systemd tcpd theora tiff truetype udev udisks unicode upower usb v4l v4l2 vaapi vorbis webm wxwidgets x264 xcb xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="active braindump flow karbon krita stage tables words sheets" CAMERAS="canon" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DVB_CARDS="usb-dib0700" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="de" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" QEMU_SOFTMMU_TARGETS="arm i386 x86_64" QEMU_USER_TARGETS="arm i386 x86_64" RUBY_TARGETS="ruby19" SANE_BACKENDS="genesys plustek" USERLAND="GNU" VIDEO_CARDS="intel nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USE_PYTHON="2.7 3.2" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 357176 [details] screenshot 1
Created attachment 357178 [details] screenshot 2
I will try =sys-kernel/gentoo-sources-3.10.9 and see if the problem is present there.
(In reply to Fabian Köster from comment #3) > I will try =sys-kernel/gentoo-sources-3.10.9 and see if the problem is > present there. Though the changelogs for .8 and .9 do not seem to include a fix for this.
Thank you for your report, will try to look at this soon.
Also happens with sys-kernel/gentoo-sources-3.10.9. I now switch back to sys-kernel/gentoo-sources-3.9.11-r1 and see if the problem is really not present there (and this bug a regression).
(In reply to Fabian Köster from comment #6) > Also happens with sys-kernel/gentoo-sources-3.10.9. > > I now switch back to sys-kernel/gentoo-sources-3.9.11-r1 and see if the > problem is really not present there (and this bug a regression). The problem also occurs with kernel sys-kernel/gentoo-sources-3.9.11-r1 but if I disable nvidia graphics (just internal intel graphics) the issue seems to be gone.
TL;DR: What was the last working kernel version? Let's see what calls those debug functions... drivers/gpu/drm/nouveau/core/engine/graph/nvc0.c: if (!nv_wait(priv, 0x409800, 0x80000000, 0x80000000)) { nv_error(priv, "HUB_INIT timed out\n"); nvc0_graph_ctxctl_debug(priv); return -EBUSY; } Ah, some kind of HUB_INIT timed out. Let's see, 0x409800 is PGRAPH.CTXCTL.CC_SCRATCH[0] for NVC0. https://github.com/envytools/envytools/blob/master/rnndb/nvc0_pgraph.xml#L693 That doesn't say much on its own, let's see what CC_SCRATCH is used for. http://cgit.freedesktop.org/nouveau/linux-2.6/plain/drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpc.fuc http://cgit.freedesktop.org/nouveau/linux-2.6/plain/drivers/gpu/drm/nouveau/core/engine/graph/fuc/hup.fuc GPC stands for Graphics Processing Cluster, FUC I don't know what that stands for but it appears to be the microcode language (as you can deduce my the extension) that gets loaded into the GPC. Actually, if we go back to nvc0.c we see comments before wait that read: /* load HUB microcode */ /* load GPC microcode */ /* start HUB ucode running, it'll init the GPCs */ So, the HUB is something that will init the GPCs; so it is a hub of GPCs. Since the code around there wasn't really touched in 2012 the problem does not lie with the loading code; so, the problem more likely lies with the microcode. # git blame drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpcnvc0.fuc | grep 2013 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpcnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 90) .b8 0xd7 0 0 0 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpcnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 91) .b16 #nvd9_gpc_mmio_head 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpcnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 92) .b16 #nvd9_gpc_mmio_tail 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpcnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 93) .b16 #nvd9_tpc_mmio_head 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/gpcnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 94) .b16 #nvd9_tpc_mmio_tail # git blame drivers/gpu/drm/nouveau/core/engine/graph/fuc/hubnvc0.fuc | grep 2013 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/hubnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 65) .b8 0xd7 0 0 0 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/hubnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 66) .b16 #nvd9_hub_mmio_head 3f196a04 drivers/gpu/drm/nouveau/core/engine/graph/fuc/hubnvc0.fuc (Ben Skeggs 2013-03-30 21:56:26 +1000 67) .b16 #nvd9_hub_mmio_tail Oh, look, upstream those files were changed on 30 March 2013; commit log: commit 3f196a045e2f7e0b7c5302d359a9772c1567d55b Author: Ben Skeggs <bskeggs@redhat.com> Date: Sat Mar 30 21:56:26 2013 +1000 drm/nve0: magic up some support for GF117 Seen in the wild, don't have the hardware but this hacks things up to treat it the same as GF119 for now. Should be relatively safe, I'd be very surprised if anything major changed outside of PGRAPH. PGRAPH (3D etc) is disabled by default however until it's confirmed working. Signed-off-by: Ben Skeggs <bskeggs@redhat.com> Now let's see which stable versions this commit was introduced in; 3.10-rc1. # git tag --contains 3f196a045e2f7e0b7c5302d359a9772c1567d55b | tr '\n' ' ' v3.10 v3.10-rc1 v3.10-rc2 v3.10-rc3 v3.10-rc4 v3.10-rc5 v3.10-rc6 v3.10-rc7 v3.10.1 v3.10.10 v3.10.2 v3.10.3 v3.10.4 v3.10.5 v3.10.6 v3.10.7 v3.10.8 v3.10.9 v3.11 v3.11-rc1 v3.11-rc2 v3.11-rc3 v3.11-rc4 v3.11-rc5 v3.11-rc6 v3.11-rc7 You have a GF119 and if you look that up in drivers/gpu/drm/nouveau/core/engine/device/nvc0.c you see it is 0xd9 whereas the above microcode seems to deal with 0xd7; so, we're not there yet, let's look at the commit whether it broke functionality for you. http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=3f196a045e2f7e0b7c5302d359a9772c1567d55b Bummer, no change in logic for that! That's not what we are looking for... :( So, back to `git log drivers/gpu/drm/nouveau/core/engine/graph/nvc0.c`; when I look there, this commit jumps to my eye because it mentions GF119 in graph FUC: commit 902530693ef38f3bb007efae594e54443d84fa56 Author: Ben Skeggs <bskeggs@redhat.com> Date: Thu Dec 20 12:50:52 2012 +1000 drm/nvc0/graph: fix fuc, and enable acceleration on GF119 Signed-off-by: Ben Skeggs <bskeggs@redhat.com> # git tag --contains 902530693ef38f3bb007efae594e54443d84fa56 | tr '\n' ' ' v3.10 v3.10-rc1 v3.10-rc2 v3.10-rc3 v3.10-rc4 v3.10-rc5 v3.10-rc6 v3.10-rc7 v3.10.1 v3.10.10 v3.10.2 v3.10.3 v3.10.4 v3.10.5 v3.10.6 v3.10.7 v3.10.8 v3.10.9 v3.11 v3.11-rc1 v3.11-rc2 v3.11-rc3 v3.11-rc4 v3.11-rc5 v3.11-rc6 v3.11-rc7 v3.8 v3.8-rc2 v3.8-rc3 v3.8-rc4 v3.8-rc5 v3.8-rc6 v3.8-rc7 v3.8.1 v3.8.10 v3.8.11 v3.8.12 v3.8.13 v3.8.2 v3.8.3 v3.8.4 v3.8.5 v3.8.6 v3.8.7 v3.8.8 v3.8.9 v3.9 v3.9-rc1 v3.9-rc2 v3.9-rc3 v3.9-rc4 v3.9-rc5 v3.9-rc6 v3.9-rc7 v3.9-rc8 v3.9.1 v3.9.10 v3.9.11 v3.9.2 v3.9.3 v3.9.4 v3.9.5 v3.9.6 v3.9.7 v3.9.8 v3.9.9 Ugh, that's long, this appears to have been introduced in v3.8-rc2; so, now I wonder if your last working kernel was before or after v3.8. So, could you tell us what your last working kernel was so we can restrict the commit log and know more exactly in which range of time to look?
(In reply to Tom Wijsman (TomWij) from comment #8) Wow, thanks for your in-depth analysis! > Ugh, that's long, this appears to have been introduced in v3.8-rc2; so, now > I wonder if your last working kernel was before or after v3.8. > > So, could you tell us what your last working kernel was so we can restrict > the commit log and know more exactly in which range of time to look? The problem did at least not occur when I was using Kernel 3.8 but I assume it was just not triggered before my switch to GNOME 3 (Previously I was using Xfce and just stable Xorg packages). When I am back at home on Friday I can do more testing of this issue, e.g. reverting commit 902530693ef38f3bb007efae594e54443d84fa56 and see if this fixes my problem. Or do you maybe have other ideas?
(In reply to Fabian Köster from comment #9) > (In reply to Tom Wijsman (TomWij) from comment #8) > > When I am back at home on Friday I can do more testing of this issue, e.g. > reverting commit 902530693ef38f3bb007efae594e54443d84fa56 and see if this > fixes my problem. Or do you maybe have other ideas? I built a Kernel from commit 5ddf4d4a543dd3303b20d7e9a4b3549589c5f095 (the one before the commit mentioned above) and I will test it now. I am not sure though, if this bug could really be hidden for such a long time...
(In reply to Fabian Köster from comment #10) > I built a Kernel from commit 5ddf4d4a543dd3303b20d7e9a4b3549589c5f095 (the > one before the commit mentioned above) and I will test it now. The bug now also happened with commit 5ddf4d4a543dd3303b20d7e9a4b3549589c5f095 so commit 902530693ef38f3bb007efae594e54443d84fa56 seems not to be the root of the problem. Any other ideas?
Created attachment 358104 [details] dmesg output I noticed I have still access to the machine when this issue happens so I attach the relevant output of dmesg.
I just found two bug reports in Red Hat's tracker: https://bugzilla.redhat.com/show_bug.cgi?id=917202 https://bugzilla.redhat.com/show_bug.cgi?id=994291 Maybe they are related?
(In reply to Fabian Köster from comment #12) > Created attachment 358104 [details] > dmesg output > > I noticed I have still access to the machine when this issue happens so I > attach the relevant output of dmesg. Hmm, it reads this: > [ 5417.069919] Code: Bad RIP value. > [ 5417.069947] RIP [< (null)>] (null) > [ 5417.069980] RSP <ffff88022d825b60> > [ 5417.070001] CR2: 0000000000000000 > [ 5417.070029] ---[ end trace 13059c79dd277520 ]--- > [ 5417.070056] Fixing recursive fault but reboot is needed! RIP is null; well, that is kind of odd and means we don't have an instruction to refer to at which point it went wrong. We do have a stack trace however; but because RIP is null, I wonder if the integrity of that stack trace is still alright. Given that it then says that this is a recursive fault I kind of have a doubt about that. Some questions: 1. Is this something that floods the dmesg? 2. Could you provide me the earliest BUG / OOPS / TRACE output it gives? 3. If you reproduce this, do the function names in the trace stay the same? Thank you in advance. As for more ideas, I'll look at what more we can do soon...
(In reply to Tom Wijsman (TomWij) from comment #14) > > Some questions: > > 1. Is this something that floods the dmesg? No, of what I have seen so far it is just a single message. > 2. Could you provide me the earliest BUG / OOPS / TRACE output it gives? If I reproduce it again, I will check if there is something. > 3. If you reproduce this, do the function names in the trace stay the same? No, I am pretty sure I have already seen different function names in trace. > RIP is null; well, that is kind of odd and means we don't have an > instruction to refer to at which point it went wrong. We do have a stack > trace however; but because RIP is null, I wonder if the integrity of that > stack trace is still alright. Given that it then says that this is a > recursive fault I kind of have a doubt about that. Because it is so odd and also not many people seem to have this issue I have the fear that this is maybe caused by a hardware defect in my discrete graphics unit. Is this possible?
This bug seems to be gone probably after upgrading Kernel to current 3.12 release candidates. This new major Kernel version includes many changes for Optimus hardware.
