Hi, Upgraded from 2.6.27-hardened-r8 to both 2.6.28-hardened-r2 and r3, unfortunately both the 2.6.28 based kernels have a kernel panic when the rt61pci wireless lan (compiled as a module, using wpa_supplicant) connects to my router. One other user in the forums is experiencing the same with rt73usb. Does not specify if hardened-sources are used. I've tried a plain 2.6.28 kernel from kernel.org and it's the same - 2.6.29-rc8 works OK though so it does not appear to be a wireless configuration issue. I note from the changelogs of 2.6.29rc1 thru rc4 that there were a lot of updates to the rt2x00 driver, but searching in the 2.6.28 series changelogs shows nothing (apart from the update to rt2x00 v2.2.0 iirc in original 2.6.28). I have dmesg outputs for both 2.6.27 & .28 kernels, also a mobile cam shot of the kernel oops. The last entry in my /var/log/messages when it fails is - Mar 14 20:03:04 foucault wpa_cli: interface wlan0 CONNECTED (this was when the original r2 release failed, hence the date). Pretty sure this is an upstream kernel bug (or maybe some problem with modules). Referring hardened-sources as I use that and gentoo-sources/vanilla-sources are still 2.6.27 based on x86. Any additional info available on request. Reproducible: Always emerge --info Portage 2.1.6.7 (hardened/linux/x86, gcc-3.4.6, glibc-2.6.1-r0, 2.6.27-hardened-r8 i686) ================================================================= System uname: Linux-2.6.27-hardened-r8-i686-Pentium_III_-Coppermine-with-glibc2.3.2 Timestamp of tree: Tue, 17 Mar 2009 23:45:02 +0000 app-shells/bash: 3.2_p39 dev-lang/python: 2.5.2-r7 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.63 sys-devel/automake: 1.10.2 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.27-r2 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium3 -fomit-frame-pointer -fforce-addr -ftracer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-O2 -march=pentium3 -fomit-frame-pointer -fforce-addr -ftracer -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps y" FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://ftp.heanet.ie/pub/gentoo/" LC_ALL="en_GB.UTF-8" LDFLAGS="-Wl,-O1" LINGUAS="en en_GB" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.de.gentoo.org/gentoo-portage" USE="acpi berkdb cli cracklib crypt dri glibc-omitfp gpm hardened iconv isdnloglogrotate midi mmx mudflap ncurses nptl nptlonly openmp pam pcre perl pic pppd python readline reflection samba session spl sse ssl tcpd unicode urandom x86 xorg zlib" ALSA_CARDS="intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindexcache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_GB" USERLAND="GNU" VIDEO_CARDS="i810" Unset: CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LANG, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Reassigning to kernel@ since it is a vanilla kernel issue. On their behalf: please post kernel config, oops shot (with GRKERNSEC_HIDESYM disabled) and lspci -vv output.
Created attachment 185431 [details] Screen shot of kernel panic
Created attachment 185433 [details] kernel .config Grsecurity level changed to Medium instead of Hardened Gentoo (Server) in order to disable Grsec HIDESYM option.
Created attachment 185434 [details] Output of lspci -vv
Since you said 2.6.29-rc8 works ok, I took a look @ the changes since 2.6.28. Here's the list of patches that have gone in drivers/net/wireless/rt2x00 since 2.6.28 was released: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=history;f=drivers/net/wireless/rt2x00;h=276eb11e9bf14305b1b5d27ac182199fe102974f;hb=HEAD Seeing as there are only 11 and 3 of them are comment changes or device additions you may be able to track down a fix with a couple patch+compile+test cycles. Looking over the patches briefly - this patch is probably the most interesting to me and probably where I'd start: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f3d340c1d536fd3e5a104c99ac9c3f8694270d72
(In reply to comment #5) > Since you said 2.6.29-rc8 works ok, I took a look @ the changes since 2.6.28. > Here's the list of patches that have gone in drivers/net/wireless/rt2x00 since > 2.6.28 was released: > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=history;f=drivers/net/wireless/rt2x00;h=276eb11e9bf14305b1b5d27ac182199fe102974f;hb=HEAD > > Seeing as there are only 11 and 3 of them are comment changes or device > additions you may be able to track down a fix with a couple patch+compile+test > cycles. > > Looking over the patches briefly - this patch is probably the most interesting > to me and probably where I'd start: > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f3d340c1d536fd3e5a104c99ac9c3f8694270d72 > I'll take a look at these then. I don't think it'll be the specific patch you linked to as it seems to relate to usb/tkip - my card is pci and I use CCMP (realise I did not mention this).
The first thing I notice is that the version of the rt2x00 driver (from reading rt2x00.h) is v2.2.1, which according to the git page was committed on 2008-08-29. So there are a LOT more than 11 patches (I count 60) from then until the latest one. The earliest patch of the 11 referred to is this http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d1b29405bd3590bc97c4d3ff2c9139ca55e56ccd And if I have a look at rt2x00leds.c in my sources, it bears hardly any resemblance to the one on that git link. That patch simply would not apply to my kernel sources.
OK I've spent several hours blindly applying patches and have got nowhere, and now my source tree is broken and won't compile. It appears that any commits from git made on 2008-10-31 or later have not made it into 2.6.28; as well as one commit on 2008-10-28 (Johannes Berg - net: convert mac to %pM). I've basically been trying the patches specific to the rt2x00 portion but there are so many patches to the wireless net subsystem that presumably involve changes to rt2x00 as well that I cannot pin down one specific git commit. Rebuilt and rebooted about 2 dozen times. I count 45 commits in the git tree from Gordon's post #5. I can't work out if there is any one patch that would fix this - it may be a combination, and I don't like the look of those odds. It might be easier to look at what changed from linux-2.6.27-hardened-r8 to linux-2.6.28-hardened-r3 There must be a better way to debug this.
The hardened team doesn't really have the time/bandwidth to figure out mainline kernel bugs unless they impede on the project itself. We try to help when we can and I am happy to include patches that fix problems. As for how to fix the issue - reverting problem patches or backporting fixes, either way is fine (though backporting fixes is usually preferable). Instead of blindly applying patches you could try using git bisect. Do some google searching on it, there's way more about it on the net than I can or am willing to type here.
(In reply to comment #9) I don't expect the hardened team to fix this as it isn't a hardened problem, as you say, it's a vanilla-kernel issue. The guy on the forum reporting the same problem w/ rt73usb is using a tuxonice kernel btw. I only filed the bug under hardened because that's the kernel I use on the box in question - is there a security reason to bump hardened-sources to being 2.6.28 based? Maybe backporting the security fixes to 2.6.27 would help, but in the meantime I'm going to mask 2.6.28 kernels and have another look at this when I have some time. I'll take a look at git bisect but it's 3am here and some people are apparently showing up @ 8am to rip the ceiling out of my living room.
(In reply to comment #10) > (In reply to comment #9) > > I don't expect the hardened team to fix this as it isn't a hardened problem, as > you say, it's a vanilla-kernel issue. The guy on the forum reporting the same > problem w/ rt73usb is using a tuxonice kernel btw. I only filed the bug under > hardened because that's the kernel I use on the box in question - is there a > security reason to bump hardened-sources to being 2.6.28 based? Maybe > backporting the security fixes to 2.6.27 would help, but in the meantime I'm > going to mask 2.6.28 kernels and have another look at this when I have some > time. > > I'll take a look at git bisect but it's 3am here and some people are apparently > showing up @ 8am to rip the ceiling out of my living room. > Do we have any news regarding your issues, Jonathan? Did you try git bisect?
Nothing, sorry. I'm living in what feels like a building site at the moment and just haven't had the time to look into this. Just masked hardened 2.6.28 kernels, and copied the last working hardened 2.6.27 into local overlay (seeing as how it's been removed from the tree, for some reason), or I'd have been downgraded to 2.6.26. I hope to try vanilla 2.6.29.1 to verify that this has been fixed in the kernel upstream, but right at this time, 2.6.28 is still broken. If I get some time I'll look at git bisect, but I've also got a broken nuclear power station to try and help fix so don't hold your breath.
Alright, I sent a mail to linux-wireless and rt2x00-devel regarding this bug. You can find the discussion thread over here: http://marc.info/?l=linux-wireless&m=123965712807866&w=2
Jonathan, could you try out the patch that Ivo van Doorn suggested (http://marc.info/?l=linux-wireless&m=123965896810244&w=2)?
(In reply to comment #14) > Jonathan, could you try out the patch that Ivo van Doorn suggested > (http://marc.info/?l=linux-wireless&m=123965896810244&w=2)? > Tried it with hardened-sources-2.6.28-r7, mod is not applicable to rt2500usb.c, manually edited rt61pci.c and rt73pci.c Now my problem is the network simply doesn't come up - I'm no wlan expert, but does this mean I can't use encryption? Or enable it in software? So I can't tell if that patch works either way.
> Tried it with hardened-sources-2.6.28-r7, mod is not applicable to rt2500usb.c, > manually edited rt61pci.c and rt73pci.c I was afraid of that, but rt2500usb doesn't really matter in this case. ;) > Now my problem is the network simply doesn't come up - I'm no wlan expert, but > does this mean I can't use encryption? Or enable it in software? No the patch only disables crypto in the hardware, this means it will automatically fallback to the behavior of 2.6.27 and before where everything was handled in software. If the network doesn't get up, do you mean the interface or is the association attempt failing?
(In reply to comment #16) > > No the patch only disables crypto in the hardware, this means it will > automatically fallback to the behavior of 2.6.27 and before where everything > was handled in software. > > If the network doesn't get up, do you mean the interface or is the association > attempt failing? > The network comes up and 'backgrounds' the task of association to wpa_cli - I had the timeout set to 30s which is fine with 2.6.27 (and the 2.6.29rc kernel I tried), increased it to 60s, but it just times out without associating. I can't seem to find anything in the logs to suggest what is actually going wrong. I'll look into how to try associating manually and increasing the output verbosity.
OK tried manually using - wpa_supplicant -C/etc/wpa_supplicant/wpa_supplicant.conf -Dwext -iwlan- -dd It was definitely seeing my router (and my neighbour's), but not connecting. I can post the o/p, but something in it made me realise that I'd disabled the security option "Broadcast Network Name" in the router's config. If I enable that, 2.6.28 can connect with the patch Ivo posted. So is there another patch to get it working like it does in 2.6.27/29 when my router isn't broadcasting itself? Or a kernel config issue I'm missing?
> So is there another patch to get it working like it does in 2.6.27/29 when my > router isn't broadcasting itself? Or a kernel config issue I'm missing? That might be a bug in mac80211 itself. rt2x00 doesn't care what essid is provided or not by the AP, but mac80211 does.
FYI, hidden SSID is really a misfeature. It does not add any security if there is at least 1 network user. It is not part of 802.11, it is just implemented by the overcompetitive router/AP manufacturers. It also breaks various types of network setups beyond a single AP. http://www.wi-fiplanet.com/tutorials/article.php/3576541 This does sound like a kernel bug which we should attack as normal, but I just felt compelled to let you know about the nonsecurity that you are working with ;)
(In reply to comment #18) > OK tried manually using - > > wpa_supplicant -C/etc/wpa_supplicant/wpa_supplicant.conf -Dwext -iwlan- -dd > > It was definitely seeing my router (and my neighbour's), but not connecting. I > can post the o/p, but something in it made me realise that I'd disabled the > security option "Broadcast Network Name" in the router's config. > > If I enable that, 2.6.28 can connect with the patch Ivo posted. > > So is there another patch to get it working like it does in 2.6.27/29 when my > router isn't broadcasting itself? Or a kernel config issue I'm missing? > A bit of a shot in the dark, but I'd like you to try out the following patch against a 2.6.28 kernel tainted with Ivo's patch ;) http://kerneltrap.org/index.php?q=mailarchive/git-commits-head/2008/5/21/1892654 That patch was introduced in 2.6.29 (which means that if the patch works, the reason that 2.6.27 functions correctly is a bit of a mystery), but it seems like a possible fix if your AP fits the 'emptry string as SSID when broadcasting is disabled' category. Thanks :)
(In reply to comment #20) > FYI, hidden SSID is really a misfeature. It does not add any security if there > is at least 1 network user. It is not part of 802.11, it is just implemented by > the overcompetitive router/AP manufacturers. It also breaks various types of > network setups beyond a single AP. > http://www.wi-fiplanet.com/tutorials/article.php/3576541 > > This does sound like a kernel bug which we should attack as normal, but I just > felt compelled to let you know about the nonsecurity that you are working with > ;) > Yeah, that's why I'm not that bothered about enabling it. I disabled it when I first set up the router, only found out it was essentially worthless afterwards, and never got around to re-enabling it until now.
(In reply to comment #21) > A bit of a shot in the dark, but I'd like you to try out the following > patch against a 2.6.28 kernel tainted with Ivo's patch ;) > http://kerneltrap.org/index.php?q=mailarchive/git-commits-head/2008/5/21/1892654 > > That patch was introduced in 2.6.29 (which means that if the patch > works, the reason that 2.6.27 functions correctly is a bit of a > mystery), but it seems like a possible fix if your AP fits the > 'emptry string as SSID when broadcasting is disabled' category. > > Thanks :) > This is (I think) the relevant section of mlmc.c I have here - static int ieee80211_sta_config_auth(struct ieee80211_sub_if_data *sdata, struct ieee80211_if_sta *ifsta) { struct ieee80211_local *local = sdata->local; struct ieee80211_bss *bss, *selected = NULL; int top_rssi = 0, freq; spin_lock_bh(&local->bss_lock); freq = local->oper_channel->center_freq; list_for_each_entry(bss, &local->bss_list, list) { if (!(bss->capability & WLAN_CAPABILITY_ESS)) continue; if ((ifsta->flags & (IEEE80211_STA_AUTO_SSID_SEL | IEEE80211_STA_AUTO_BSSID_SEL | IEEE80211_STA_AUTO_CHANNEL_SEL)) && (!!(bss->capability & WLAN_CAPABILITY_PRIVACY) ^ !!sdata->default_key)) continue; if (!(ifsta->flags & IEEE80211_STA_AUTO_CHANNEL_SEL) && bss->freq != freq) continue; So it looks as if that patch is already applied to the kernel source I have (and the only change I made myself was Ivo's single digit patch).
(In reply to comment #23) > (In reply to comment #21) > > > A bit of a shot in the dark, but I'd like you to try out the following > > patch against a 2.6.28 kernel tainted with Ivo's patch ;) > > http://kerneltrap.org/index.php?q=mailarchive/git-commits-head/2008/5/21/1892654 > > > > That patch was introduced in 2.6.29 (which means that if the patch > > works, the reason that 2.6.27 functions correctly is a bit of a > > mystery), but it seems like a possible fix if your AP fits the > > 'emptry string as SSID when broadcasting is disabled' category. > > > > Thanks :) > > > > This is (I think) the relevant section of mlmc.c I have here - > > static int ieee80211_sta_config_auth(struct ieee80211_sub_if_data *sdata, > struct ieee80211_if_sta *ifsta) > { > struct ieee80211_local *local = sdata->local; > struct ieee80211_bss *bss, *selected = NULL; > int top_rssi = 0, freq; > > spin_lock_bh(&local->bss_lock); > freq = local->oper_channel->center_freq; > list_for_each_entry(bss, &local->bss_list, list) { > if (!(bss->capability & WLAN_CAPABILITY_ESS)) > continue; > > if ((ifsta->flags & (IEEE80211_STA_AUTO_SSID_SEL | > IEEE80211_STA_AUTO_BSSID_SEL | > IEEE80211_STA_AUTO_CHANNEL_SEL)) && > (!!(bss->capability & WLAN_CAPABILITY_PRIVACY) ^ > !!sdata->default_key)) > continue; > > if (!(ifsta->flags & IEEE80211_STA_AUTO_CHANNEL_SEL) && > bss->freq != freq) > continue; > > So it looks as if that patch is already applied to the kernel source I have > (and the only change I made myself was Ivo's single digit patch). > Excuse me, wrong clipboard entry. I meant this patch: http://markmail.org/message/f4xwfshiz4mvd7jt Like I said it's a shot in the dark and I'm gonna push the bug upstream if that doesn't help either.
(In reply to comment #24) > > Excuse me, wrong clipboard entry. > I meant this patch: http://markmail.org/message/f4xwfshiz4mvd7jt > > Like I said it's a shot in the dark and I'm gonna push the bug upstream if that > doesn't help either. > That patch didn't work either, unfortunately.
Alright, regarding this one: -We should first open a new bug for the mac80211 bug that spawned. Let's keep things clean :) -Ivo, as far as the kernel panic is concerned, your patch is just a quick and dirty workaround, right? It just showed us that the panic is caused by the hardware crypto functions. How should we move from here?
(In reply to comment #26) > Alright, regarding this one: > > -We should first open a new bug for the mac80211 bug that spawned. Let's keep > things clean :) > -Ivo, as far as the kernel panic is concerned, your patch is just a quick and > dirty workaround, right? It just showed us that the panic is caused by the > hardware crypto functions. How should we move from here? > Yes mac80211 is a different bug. Hopefully Ivo can shift the rt61/73 crypto patch for the kernel panic upstream, to be fixed in the next 2.6.28 point release. That's what it would take to close this.
I created a new bugzilla entry specifically for the mac80211 issue. You can find it here: http://bugs.gentoo.org/show_bug.cgi?id=267368
> -Ivo, as far as the kernel panic is concerned, your patch is just a quick and > dirty workaround, right? It just showed us that the panic is caused by the > hardware crypto functions. How should we move from here? Well the patch is a workaround, but as the initial bugreport said the bug is fixed in 2.6.29. That means that I think it is best to have the workaround as the fix and not spend much time in finding out what the true fix would be. Also as far as I know, with the 2.6.29 release, haven't the stable releases for 2.6.28 been stopped and moved on to 2.6.29? If there is still somebody collecting patches for a stable 2.6.28 then my workaround would be good enough.
(In reply to comment #29) > > -Ivo, as far as the kernel panic is concerned, your patch is just a quick and > > dirty workaround, right? It just showed us that the panic is caused by the > > hardware crypto functions. How should we move from here? > > Well the patch is a workaround, but as the initial bugreport said the bug is > fixed in 2.6.29. That means that I think it is best to have the workaround as > the fix and not spend much time in finding out what the true fix would be. > > Also as far as I know, with the 2.6.29 release, haven't the stable releases for > 2.6.28 been stopped and moved on to 2.6.29? If there is still somebody > collecting patches for a stable 2.6.28 then my workaround would be good enough. > I agree Ivo. I'll also try pinging the rt73usb user on the Gentoo Forum and see if he can replicate the fix. And also double-check that 2.6.29 definitely fixes it - I only tried 1 release candidate kernel.
I removed rt2500usb references off Ivo's patch (since they wouldn't apply anyway) and added the patch in the genpatches trunk. It should come with the next 2.6.28 gentoo-sources release :)
Released in gentoo-sources-2.6.28-r6.
