Mar 4 03:55:27 svr-media smartd[5474]: Device: /dev/hda, SMART Usage Attribute: 194 Temperature_Celsius changed from 82 to 83 Mar 4 04:16:34 svr-media hdc: status error: status=0x37 { DeviceFault SeekComplete CorrectedError Index Error } Mar 4 04:16:34 svr-media hdc: status error: error=0x00 { } Mar 4 04:16:34 svr-media ide: failed opcode was: unknown Mar 4 04:16:34 svr-media hdc: DMA disabled Mar 4 04:16:34 svr-media hdc: drive not ready for command Mar 4 04:17:09 svr-media ide1: reset timed-out, status=0xb7 Mar 4 04:17:09 svr-media hdc: status timeout: status=0xb7 { Busy } Mar 4 04:17:09 svr-media ide: failed opcode was: unknown Mar 4 04:17:09 svr-media hdc: drive not ready for command Mar 4 04:17:39 svr-media ide1: reset timed-out, status=0xb7 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 18432775 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849407 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849415 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849423 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849431 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849439 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849447 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849455 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849463 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849471 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849479 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849487 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849495 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849503 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849511 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849519 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849527 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849407 Mar 4 04:17:39 svr-media raid5: Disk failure on hdc1, disabling device. Operation continuing on 4 device s Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849415 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849423 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849431 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849439 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849447 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849455 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849463 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849471 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849479 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849487 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849495 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849503 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849511 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849519 Mar 4 04:17:39 svr-media end_request: I/O error, dev hdc, sector 145849527 Mar 4 04:17:39 svr-media RAID5 conf printout: Mar 4 04:17:39 svr-media --- rd:5 wd:4 fd:1 Mar 4 04:17:39 svr-media disk 0, o:0, dev:hdc1 Mar 4 04:17:39 svr-media disk 1, o:1, dev:hde1 Mar 4 04:17:39 svr-media disk 2, o:1, dev:hdg1 Mar 4 04:17:39 svr-media disk 3, o:1, dev:hdi1 Mar 4 04:17:39 svr-media disk 4, o:1, dev:hdk1 Mar 4 04:17:39 svr-media RAID5 conf printout: Mar 4 04:17:39 svr-media --- rd:5 wd:4 fd:1 Mar 4 04:17:39 svr-media disk 1, o:1, dev:hde1 Mar 4 04:17:39 svr-media disk 2, o:1, dev:hdg1 Mar 4 04:17:39 svr-media disk 3, o:1, dev:hdi1 Mar 4 04:17:39 svr-media disk 4, o:1, dev:hdk1 Mar 4 04:17:39 svr-media 0x0: 01 9a 01 09 72 00 06 0b ec 91 0a 39 08 45 0b 0e Mar 4 04:17:39 svr-media Filesystem "md0": XFS internal error xfs_da_do_buf(2) at line 2084 of file fs/x fs/xfs_da_btree.c. Caller 0xc021a6ab Mar 4 04:17:39 svr-media [] xfs_corruption_error+0xbb/0xc6 Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] __down_failed+0x7/0xc Mar 4 04:17:39 svr-media [] .text.lock.xfs_buf+0x23/0x5c Mar 4 04:17:39 svr-media [] xfs_buf_read_flags+0x3c/0x6c Mar 4 04:17:39 svr-media [] kmem_zone_alloc+0x55/0x9a Mar 4 04:17:39 svr-media [] xfs_da_do_buf+0x5e4/0x6bf Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_dir2_leafn_lookup_int+0x31e/0x452 Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup_int+0x3e/0x17e Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup_int+0x3e/0x17e Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup+0x18/0x8f Mar 4 04:17:39 svr-media [] xfs_dir_lookup+0xac/0xfb Mar 4 04:17:39 svr-media [] tcp_v4_send_check+0x7e/0xc3 Mar 4 04:17:39 svr-media [] xfs_dir_lookup_int+0x29/0xa7 Mar 4 04:17:39 svr-media [] xfs_lookup+0x44/0x6d Mar 4 04:17:39 svr-media [] xfs_access+0x2b/0x33 Mar 4 04:17:39 svr-media [] xfs_vn_lookup+0x30/0x68 Mar 4 04:17:39 svr-media [] do_lookup+0xa6/0x13d Mar 4 04:17:39 svr-media [] __link_path_walk+0x72b/0xb7e Mar 4 04:17:39 svr-media [] link_path_walk+0x49/0xbb Mar 4 04:17:39 svr-media [] autoremove_wake_function+0x0/0x3a Mar 4 04:17:39 svr-media [] do_path_lookup+0x195/0x1b0 Mar 4 04:17:39 svr-media [] __user_walk_fd+0x29/0x41 Mar 4 04:17:39 svr-media [] vfs_stat_fd+0x1a/0x45 Mar 4 04:17:39 svr-media [] autoremove_wake_function+0x0/0x3a Mar 4 04:17:39 svr-media [] vfs_stat+0xf/0x13 Mar 4 04:17:39 svr-media [] sys_stat64+0x10/0x26 Mar 4 04:17:39 svr-media [] sys_read+0x3b/0x64 Mar 4 04:17:39 svr-media [] sysenter_past_esp+0x56/0x79 Mar 4 04:17:39 svr-media 0x0: 01 9a 01 09 72 00 06 0b ec 91 0a 39 08 45 0b 0e Mar 4 04:17:39 svr-media Filesystem "md0": XFS internal error xfs_da_do_buf(2) at line 2084 of file fs/x fs/xfs_da_btree.c. Caller 0xc021a6ab Mar 4 04:17:39 svr-media [] xfs_corruption_error+0xbb/0xc6 Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_buf_read_flags+0x1f/0x6c Mar 4 04:17:39 svr-media [] kmem_zone_alloc+0x55/0x9a Mar 4 04:17:39 svr-media [] xfs_da_do_buf+0x5e4/0x6bf Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_dir2_leafn_lookup_int+0x31e/0x452 Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup_int+0x3e/0x17e Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup_int+0x3e/0x17e Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup+0x18/0x8f Mar 4 04:17:39 svr-media [] xfs_dir_lookup+0xac/0xfb Mar 4 04:17:39 svr-media [] tcp_v4_send_check+0x7e/0xc3 Mar 4 04:17:39 svr-media [] xfs_dir_lookup_int+0x29/0xa7 Mar 4 04:17:39 svr-media [] xfs_lookup+0x44/0x6d Mar 4 04:17:39 svr-media [] xfs_access+0x2b/0x33 Mar 4 04:17:39 svr-media [] xfs_vn_lookup+0x30/0x68 Mar 4 04:17:39 svr-media [] do_lookup+0xa6/0x13d Mar 4 04:17:39 svr-media [] __link_path_walk+0x72b/0xb7e Mar 4 04:17:39 svr-media [] link_path_walk+0x49/0xbb Mar 4 04:17:39 svr-media [] autoremove_wake_function+0x0/0x3a Mar 4 04:17:39 svr-media [] do_path_lookup+0x195/0x1b0 Mar 4 04:17:39 svr-media [] __user_walk_fd+0x29/0x41 Mar 4 04:17:39 svr-media [] vfs_lstat_fd+0x17/0x42 Mar 4 04:17:39 svr-media [] autoremove_wake_function+0x0/0x3a Mar 4 04:17:39 svr-media [] vfs_lstat+0xf/0x13 Mar 4 04:17:39 svr-media [] sys_lstat64+0x10/0x26 Mar 4 04:17:39 svr-media [] sys_read+0x3b/0x64 Mar 4 04:17:39 svr-media [] sysenter_past_esp+0x56/0x79 Mar 4 04:17:39 svr-media 0x0: 01 9a 01 09 72 00 06 0b ec 91 0a 39 08 45 0b 0e Mar 4 04:17:39 svr-media Filesystem "md0": XFS internal error xfs_da_do_buf(2) at line 2084 of file fs/x fs/xfs_da_btree.c. Caller 0xc021a6ab Mar 4 04:17:39 svr-media [] xfs_corruption_error+0xbb/0xc6 Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_buf_read_flags+0x1f/0x6c Mar 4 04:17:39 svr-media [] kmem_zone_alloc+0x55/0x9a Mar 4 04:17:39 svr-media [] xfs_da_do_buf+0x5e4/0x6bf Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_da_read_buf+0x23/0x27 Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup_int+0x3e/0x17e Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup_int+0x3e/0x17e Mar 4 04:17:39 svr-media [] xfs_dir2_block_lookup+0x18/0x8f Mar 4 04:17:39 svr-media [] xfs_dir_lookup+0xac/0xfb Mar 4 04:17:39 svr-media [] tcp_v4_send_check+0x7e/0xc3 Mar 4 04:17:39 svr-media [] xfs_dir_lookup_int+0x29/0xa7 Mar 4 04:17:39 svr-media [] xfs_lookup+0x44/0x6d Mar 4 04:17:39 svr-media [] xfs_vn_lookup+0x30/0x68 Mar 4 04:17:39 svr-media [] do_lookup+0xa6/0x13d Mar 4 04:17:39 svr-media [] __link_path_walk+0x72b/0xb7e Mar 4 04:17:39 svr-media [] link_path_walk+0x49/0xbb Mar 4 04:17:39 svr-media [] filemap_nopage+0x98/0x2c8 Mar 4 04:17:39 svr-media [] filemap_nopage+0x163/0x2c8 Mar 4 04:17:39 svr-media [] do_path_lookup+0x195/0x1b0 Mar 4 04:17:39 svr-media [] __user_walk_fd+0x29/0x41 Mar 4 04:17:39 svr-media [] vfs_stat_fd+0x1a/0x45 Mar 4 04:17:39 svr-media [] filemap_nopage+0x98/0x2c8 Mar 4 04:17:39 svr-media [] filemap_nopage+0x163/0x2c8 Mar 4 04:17:39 svr-media [] vfs_stat+0xf/0x13 Mar 4 04:17:39 svr-media [] sys_stat64+0x10/0x26 Mar 4 04:17:39 svr-media [] do_page_fault+0x18a/0x469 Mar 4 04:17:39 svr-media [] do_page_fault+0x1f4/0x469 Mar 4 04:17:39 svr-media [] do_page_fault+0x0/0x469 Mar 4 04:17:39 svr-media [] sysenter_past_esp+0x56/0x79 Mar 4 04:17:39 svr-media 0x0: 01 9a 01 09 72 00 06 0b ec 91 0a 39 08 45 0b 0e Mar 4 04:17:39 svr-media Filesystem "md0": XFS internal error xfs_da_do_buf(2) at line 2084 of file fs/x fs/xfs_da_btree.c. Caller 0xc021a6ab ... (another gigabyte or so of this).