When mounting unclean xfs filesystem I'd got oops. On that filesystem (before xfs_repair) every time it crashed (on 2.6.23-r6, and kernel from grml 1.1 also). Unfortunalty I haven't log from xfs_repair. Reproducible: Didn't try Portage 2.2_rc1 (default-linux/x86/2007.0, gcc-4.2.3, glibc-2.7-r1, 2.6.25-hardened-r2 i686) ================================================================= System uname: Linux-2.6.25-hardened-r2-i686-Intel-R-_Xeon-TM-_CPU_3.20GHz-with-glibc2.0 Timestamp of tree: Tue, 22 Jul 2008 02:03:01 +0000 distcc 2.18.3 i486-slackware-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] app-shells/bash: 3.2_p39 dev-lang/python: 2.4.4-r5, 2.5.1-r5 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 2.0.0 sys-apps/openrc: 0.2.5 sys-apps/sandbox: 1.2.18.1-r3 sys-devel/autoconf: 2.13, 2.62-r1 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.17-r2 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.25-r4 ACCEPT_KEYWORDS="x86 ~x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=i686 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /var/log" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-I/home/staff/marmarek/.usr/include" DISTDIR="/usr/portage/distfiles" FEATURES="buildpkg distlocks parallel-fetch preserve-libs sandbox sfperms strict test unmerge-orphans userfetch" GENTOO_MIRRORS="ftp://bizon.staszic.waw.pl/portage ftp://sunsite.icm.edu.pl/pub/linux/dist/gentoo gentoo.inode.at http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LDFLAGS="-L/home/staff/marmarek/.usr/lib" LINGUAS="en pl es de" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/home/others/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://bizon.staszic.waw.pl/gentoo-portage/" USE="aalib acct acl acpi amavis apache2 bash-completion berkdb bzip2 caps cddb clamav cli clisp cpio cracklib crypt cups curl curlwrappers cvs diskio dovecot-sasl exif expat flac fortran ftp gd gdbm gif gnutls gocr gpgme gpm gs hash iconv idn imagemagick imap iproute2 ipv6 jabber jpeg latex ldap ldap-sasl libcaca libssh2 libwww lm_sensors lzo maildir midi ming mmx mmxext mnogosearch mp3 mplayer mudflap mysql ncurses nethack nfs nls nntp nptl nptlonly objc objc++ objc-gc ocaml openmp pam pcntl pcre pdf pdo perl php png pnm pop pop3d posix postfix postgres python quotas rar readline reflection rpc rss ruby sasl server session sftp sieve skey slang smp smtp snmp socks5 sox spell spl sqlite sqlite3 sse sse2 ssl syslog tcpd tcpwrapper tesseract tetex tidy tiff tools tos unicode unzip userlocales utempter vhosts vim vim-pager vim-syntax win32codecs wmf x264 x86 xattr xfs xlockrc xml xmlreader xmlrpc xmlwriter xorg xpm xvid zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias asis auth_digest ident proxy" APACHE2_MPMS="prefork" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en pl es de" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 i810 imstt mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa vga via vmware voodoo" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, MAKEOPTS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Kernel log from grml 1.1 kernel (I haven't from hardened kernel - it oops on mounting /, but looks similar...): XFS mounting filesystem md2 Starting XFS recovery on filesystem: md2 (logdev: internal) BUG: unable to handle kernel paging request at virtual address 68d28d8b printing eip: c027fac6 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP Modules linked in: raid456 async_xor async_memcpy async_tx xor ipv6 video output sbs container dock battery ac fuse button iTCO_wdt i2c_i801 e752x_edac shpchp pci_hotplug edac_core i2c_core tsdev evdev rtc pcspkr loop aufs usb_storage aic79xx tg3 uhci_hcd usbcore thermal processor fan squashfs sqlzma unlzma CPU: 1 EIP: 0060:[<c027fac6>] Not tainted VLI EFLAGS: 00010206 (2.6.23-grml #1) EIP is at xlog_recover_process_data+0x55/0x175 eax: 6f6c6f67 ebx: 09726d6e ecx: f9661e1c edx: 09726d6e esi: 68d28d83 edi: f646bb8c ebp: 00000082 esp: f646bab0 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068 Process mount (pid: 5431, ti=f646a000 task=f6479170 task.ti=f646a000) Stack: 00002c78 000000df f73e1000 f646bb60 f6429e00 f9666000 f9661e1c 00000e00 f9661200 00000000 f6429e00 c0280264 f9661200 00000001 000007a0 00000000 00000000 00000001 00000020 00000000 00000000 00000000 00000001 f646bb30 Call Trace: [<c0280264>] xlog_do_recovery_pass+0x67e/0x834 [<c028045e>] xlog_do_log_recovery+0x44/0x92 [<c02804c9>] xlog_do_recover+0x1d/0x112 [<c028064c>] xlog_recover+0x8e/0x9a [<c027a258>] xfs_log_mount+0xab/0xee [<c028309c>] xfs_mountfs+0x8f1/0xcc5 [<c0268f2f>] xfs_fs_vcmn_err+0x67/0x8d [<c028362f>] xfs_mru_cache_create+0xe1/0x10e [<c0289ebb>] xfs_mount+0x2f5/0x36c [<c023ad6a>] xfs_qm_parseargs+0x1c5/0x1cf [<c0289bc6>] xfs_mount+0x0/0x36c [<c029a347>] vfs_mount+0x17/0x1a [<c029a220>] xfs_fs_fill_super+0x6c/0x17c [<c02b8aa6>] snprintf+0x1f/0x22 [<c01a13cf>] disk_name+0x30/0x83 [<c016f5de>] get_sb_bdev+0xcc/0x10a [<c02995fd>] xfs_fs_get_sb+0x20/0x25 [<c029a1b4>] xfs_fs_fill_super+0x0/0x17c [<c016f1e0>] vfs_kern_mount+0x83/0xfe [<c016f2a5>] do_kern_mount+0x35/0xbb [<c0181137>] do_mount+0x5e5/0x647 [<c01594d5>] __do_fault+0x32f/0x35e [<c0175c60>] link_path_walk+0xa9/0xb3 [<c015b80b>] handle_mm_fault+0x38c/0x76a [<c01165cf>] apic_wait_icr_idle+0xe/0x15 [<c015309a>] __alloc_pages+0x4d/0x27d [<c015d1db>] unmap_region+0xfb/0x119 [<c017fb47>] copy_mount_options+0x26/0x109 [<c0181210>] sys_mount+0x77/0xb3 [<c0104032>] syscall_call+0x7/0xb ======================= Code: 89 ca 8b 69 28 0f cd e8 b0 d2 ff ff ba 05 00 00 00 85 c0 0f 85 28 01 00 00 e9 13 01 00 00 c7 44 24 04 a1 7a 4d c0 e9 c7 00 00 00 <8a> 46 08 3c 69 74 04 3c aa 75 e8 8b 7c 24 0c 8d 4e 0c 8b 1e 0f EIP: [<c027fac6>] xlog_recover_process_data+0x55/0x175 SS:ESP 0068:f646bab0
Created attachment 161779 [details] kernel configuration
Un-CCing hardened@g.o, doesn't appear related to our patches. Also, should probably check your XFS partitions for corruption.
(In reply to comment #3) > Un-CCing hardened@g.o, doesn't appear related to our patches. Also, should > probably check your XFS partitions for corruption. Yes, xfs_repair fix problem, but kernel shouldn't oops on unclean filesystem.
(In reply to comment #4) > (In reply to comment #3) > > Un-CCing hardened@g.o, doesn't appear related to our patches. Also, should > > probably check your XFS partitions for corruption. > > Yes, xfs_repair fix problem, but kernel shouldn't oops on unclean filesystem. > Very true - was not suggesting otherwise, simply a suggestion from your oops. Reading again I see you probably already ran it, sorry. :)
Do you still have the kernel that the crash occurred on? If so, could you run the following command, please: > gdb /usr/src/<path>/fs/xfs/xfs_log_recover.o Then, at the prompt, enter this command and paste the output here: (gdb) l *xlog_recover_process_data+0x55 I don't suppose you still have a copy of the corrupted image? That would greatly help testing.
(In reply to comment #6) > Do you still have the kernel that the crash occurred on? If so, could you run > the following command, please: > > gdb /usr/src/<path>/fs/xfs/xfs_log_recover.o > > Then, at the prompt, enter this command and paste the output here: > (gdb) l *xlog_recover_process_data+0x55 Actualy I don't have compiled sources of that kernel (from which backtrace is). It was from grml-1.1. Downloading sources and compiling it with same config will generate same binary (even gcc version differ)? Similar crash was on 2.6.25-hardened-r2, but I haven't exact backtrace... Reading symbols from /usr/src/linux-2.6.25-hardened-r2/fs/xfs/xfs_log_recover.o...(no debugging symbols found)...done. I can compile it with debugging symbols, but I dont know if addresses will be the same. It will help? > I don't suppose you still have a copy of the corrupted image? That would > greatly help testing. I know, but it was on production machine...
Sorry, there's not much we can do here: we don't have the original oops, 2.6.23 is too old to be supported, we have no way of reproducing or getting debugging info. (rebuilding will almost certainly break those offsets)
