Capturing live packets on a C8000 with a 64-bit kernel causes the kernel to apparently exhaust its memory resources. This variably ends up with an HPMC, eating a SPARC or a plain kernel panic. Sometimes libpcap reports something along these lines: Error while capturing packets: corrupted frame on kernel ring mac offset 9562 + caplen -1163995555 > frame len 262144 so that's a negative caplen. Usually the kernel chokes at that point. With a 32-bit kernel (on a C3650) I don't see this. karsten login: [633252.873341] device eth0 entered promiscuous mode <Cpu0> e800009800e00000 000000004049ea90 CC_ERR_CHECK_HPMC ^M<Cpu1> e800009801e00000 00000000fa1f4d8f CC_ERR_CHECK_HPMC ^M<Cpu0> 37000f7300e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu1> 37000f7301e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu0> f600105e00e00000 fffffff0f0c00000 CC_MC_HPMC_MONARCH_SELECTED ^M<Cpu1> 5600100b01e00000 0000000000000194 CC_MC_OS_HPMC_LEN_ERR ^M<Cpu0> 140003b200e00000 000000000000000b CC_ERR_HPMC_STATE_ENTRY ^M<Cpu1> 5600106401e00000 fffffff0f0436fc0 CC_MC_BR_TO_OS_HPMC_FAILED ^M<Cpu1> 160012cf01e00000 010001001e00000d CC_MPS_CPU_WAITING ^M<Cpu0> 5600100b00e00000 0000000000000194 CC_MC_OS_HPMC_LEN_ERR ^M<Cpu0> 5600106400e00000 fffffff0f0435110 CC_MC_BR_TO_OS_HPMC_FAILED ^M<Cpu0> e800009800e00000 000000004049ea90 CC_ERR_CHECK_HPMC ^M<Cpu0> 37000f7300e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu0> 4000109f00e00000 0000000000000000 CC_MC_HPMC_INITIATED ^M<Cpu0> 4000101900e00000 0000000000000000 CC_MC_MULTIPLE_HPMCS ^M<Cpu0> 030010d500e00000 0000000000000000 CC_CPU_STOP ^M<Cpu1> 160012d001e00000 0100010000000001 CC_MPS_CPU_WAIT_OVER ^M<Cpu1> 2800035d01e00000 00000000fa1f4d8f CC_ERR_CHECK_TOC ^M<Cpu1> 37000f7301e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu1> 4300107401e00000 0000000000000000 CC_MC_OS_TOC_MISSING ^M<Cpu1> 170003db01e00000 ffffffffffffffff CC_ERR_TOC_NO_OS_TOC ^M<Cpu1> 160012cf01e00000 01000100b400000d CC_MPS_CPU_WAITING ^M<Cpu1> 160012d001e00000 0100010000000001 CC_MPS_CPU_WAIT_OVER ^M<Cpu0> 160012be00e00000 0000000000000213 CC_BOOT_INIT_CHASSIS_CODES
This is karsten.jer-c2.orkz.net (Linux parisc64 3.17.7-gentoo-JeR) 00:37:55 karsten login: [49579.498375] device eth0 entered promiscuous mode <Cpu0> e800009800e00000 00000000fa0992d3 CC_ERR_CHECK_HPMC ^M<Cpu1> e800009801e00000 000000004049a450 CC_ERR_CHECK_HPMC ^M<Cpu0> 37000f7300e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu1> 37000f7301e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu0> f600105e00e00000 fffffff0f0c00000 CC_MC_HPMC_MONARCH_SELECTED ^M<Cpu1> 5600100b01e00000 0000000000000194 CC_MC_OS_HPMC_LEN_ERR ^M<Cpu0> 140003b200e00000 000000000000000b CC_ERR_HPMC_STATE_ENTRY ^M<Cpu1> 5600106401e00000 fffffff0f0436fc0 CC_MC_BR_TO_OS_HPMC_FAILED ^M<Cpu1> 160012cf01e00000 010001001e00000d CC_MPS_CPU_WAITING ^M<Cpu0> 5600100b00e00000 0000000000000194 CC_MC_OS_HPMC_LEN_ERR ^M<Cpu0> 5600106400e00000 fffffff0f0435110 CC_MC_BR_TO_OS_HPMC_FAILED ^M[49579.891708] timer_interrupt(CPU 0): delayed! cycles 6B827B94 rem 22E3EC next/now 2E5128EE9AC5/2E5128CBB6D9 [49582.015041] Unaligned handler failed, ret = -2 [49582.068375] _______________________________ [49582.068375] < Your System ate a SPARC! Gah! > [49582.068375] ------------------------------- [49582.068375] \ ^__^ [49582.068375] (__)\ )\/\ [49582.068375] U ||----w | [49582.068375] || || [49582.428375] dumpcap (pid 22599): Unaligned data reference (code 28) [49582.505041] CPU: 0 PID: 22599 Comm: dumpcap Not tainted 3.17.7-gentoo-JeR #1 [49582.588375] task: 00000000438156a0 ti: 00000000513e0000 task.ti: 00000000513e0000 [49582.678375] [49582.698375] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [49582.755041] PSW: 00001000000000010000000000001000 Not tainted [49582.821708] r00-03 0000000008010008 fffffff0f0e37000 fffffff0f0d75858 0000000000000000 [49582.918375] r04-07 fffffff0f0402970 fffffff0f0402570 fffffffffffffff9 0000000000000000 [49583.015041] r08-11 0000000000000001 8400000000800000 fffffff0f040b880 8400000000800000 [49583.111708] r12-15 0000000000000000 000000004101306c 000000004127718c 0000000000000000 [49583.205041] r16-19 00000000412771e8 00000000412771e4 0000000041277020 0000000000000000 [49583.301708] r20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [49583.398374] r24-27 fffffff0f0402e1c 0000000000000000 fffffff0f0e93508 fffffff0f0400000 [49583.495041] r28-31 fffffff0f0435110 fffffff0f0402970 fffffff0f0402a70 636e746c696e6974 [49583.591708] sr00-03 0000000004fb4800 0000000000000000 0000000000000000 0000000004fb4800 [49583.688374] sr04-07 0000000004fb4800 0000000004fb4800 0000000004fb4800 0000000004fb4800 [49583.785041] [49583.801708] IASQ: 000000003ffffff0 000000003ffffff0 IAOQ: fffffff0f0d597d4 fffffff0f0d597d8 [49583.901708] IIR: 52730240 ISR: 000000003ffff800 IOR: c00007f0f0402f3c [49583.985041] CPU: 0 CR30: 00000000513e0000 CR31: ffffffffffffffff [49584.068374] ORIG_R28: 0000000000000000 [49584.115041] IAOQ[0]: 0xfffffff0f0d597d4 [49584.161708] IAOQ[1]: 0xfffffff0f0d597d8 [49584.208374] RP(r2): 0xfffffff0f0d75858 [49584.255041] Backtrace: [49584.281708] [49584.298374] CPU: 0 PID: 22599 Comm: dumpcap Not tainted 3.17.7-gentoo-JeR #1 [49584.385041] Backtrace: [49584.411708] [<000000004017bf94>] show_stack+0x2c/0x40 [49584.475041] [<000000004048f688>] dump_stack+0xe0/0x150 [49584.538374] [<000000004017c258>] die_if_kernel+0x290/0x3c8 [49584.605041] [<0000000040187b10>] handle_unaligned+0x8f8/0x988 [49584.675041] [49584.691708] ---[ end trace e5e7cb8386160465 ]--- [49584.755041] [sched_delayed] sched: RT throttling activated [49608.778372] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [dumpcap:22599] [49608.778372] Modules linked in: nfnetlink_queue nfnetlink_log auth_rpcgss oid_registry nfsv4 dns_resolver nfsv3 nfs lockd sunrpc nf_conntrack_netlink nf_conntrack nfnetlink radeon hwmon drm_kms_helper ttm drm agpgart sr_mod cdrom ohci_pci radeonfb cfbfillrect ohci_hcd ehci_pci cfbimgblt ehci_hcd cfbcopyarea usbcore i2c_algo_bit usb_common fb_ddc pata_sil680 nls_base fb libata [49608.778372] CPU: 0 PID: 22599 Comm: dumpcap Tainted: G D 3.17.7-gentoo-JeR #1 [49608.778372] task: 00000000438156a0 ti: 00000000513e0000 task.ti: 00000000513e0000 [49608.778372] [49608.778372] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [49608.778372] PSW: 00001000000011100000000000001111 Tainted: G D [49608.778372] r00-03 00000000080e000f 00000000513e0db0 00000000402304d4 00000000513e0ce0 [49608.778372] r04-07 0000000040898080 00000000513e0ce8 0000000000000001 0000000000000001 [49608.778372] r08-11 0000000040871220 0000000000000000 00000000513e0ce8 00000000409542b8 [49608.778372] r12-15 00000000409541d0 00000000f767f000 00000000409541d0 0000000000000c08 [49608.778372] r16-19 0000000208905a00 00000000118c0b03 00000000513f6000 fffffffffe781000 [49608.778372] r20-23 0000000000000001 0000000000000020 0000000000000004 0000000000000004 [49608.778372] r24-27 00000000513e0ce8 000000000800000f 00000000427f9228 0000000040898080 [49608.778372] r28-31 0000000000000003 00000000513e0e10 00000000513e0db0 0000000000000001 [49608.778372] sr00-03 0000000004fa1800 0000000004fb4800 0000000000000000 0000000004fa1800 [49608.778372] sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [49608.778372] [49608.778372] IASQ: 0000000000000000 0000000000000000 IAOQ: 00000000402304f4 00000000402304e8 [49608.778372] IIR: 44bc0030 ISR: 00000000513e0e60 IOR: 0000000040898080 [49608.778372] CPU: 0 CR30: 00000000513e0000 CR31: ffffffffffffffff [49608.778372] ORIG_R28: 0000000040898080 [49608.778372] IAOQ[0]: generic_exec_single+0x1dc/0x210 [49608.778372] IAOQ[1]: generic_exec_single+0x1d0/0x210 [49608.778372] RP(r2): generic_exec_single+0x1bc/0x210 [49608.778372] Backtrace: [49608.778372] [<00000000402305e4>] smp_call_function_single+0xbc/0xd0 [49608.778372] [<0000000040230994>] smp_call_function_many+0x1ec/0x440 [49608.778372] [<0000000040230c40>] smp_call_function+0x58/0x70 [49608.778372] [<0000000040230c9c>] on_each_cpu+0x44/0x90 [49608.778372] [<00000000401790c4>] flush_tlb_all+0x13c/0x200 [49608.778372] [<00000000402a57c0>] tlb_flush_mmu_tlbonly+0x58/0x70 [49608.778372] [<00000000402a7080>] unmap_single_vma+0x838/0x8d8 [49608.778372] [<00000000402a87dc>] unmap_vmas+0x74/0xa8 [49608.778372] [<00000000402b2398>] exit_mmap+0xf0/0x268 [49608.778372] [<00000000401a20e0>] mmput+0xc8/0x218 [49608.778372] [<00000000401a8734>] do_exit+0x76c/0x1070 [49608.778372] [49608.781705] timer_interrupt(CPU 0): delayed! cycles A57BF970 rem 18F110 next/now 2E5770A5EC05/2E57708CFAF5 <Cpu1> 160012d001e00000 0100010000000001 CC_MPS_CPU_WAIT_OVER ^M<Cpu1> 2800035d01e00000 000000004049a450 CC_ERR_CHECK_TOC ^M<Cpu1> 37000f7301e00000 8400000000800000 CC_ERR_CPU_CHECK_SUMMARY ^M<Cpu1> 4300107401e00000 0000000000000000 CC_MC_OS_TOC_MISSING ^M<Cpu1> 170003db01e00000 ffffffffffffffff CC_ERR_TOC_NO_OS_TOC ^M<Cpu1> 160012cf01e00000 01000100b400000d CC_MPS_CPU_WAITING ^M[49636.778369] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [dumpcap:22599] [49636.778369] Modules linked in: nfnetlink_queue nfnetlink_log auth_rpcgss oid_registry nfsv4 dns_resolver nfsv3 nfs lockd sunrpc nf_conntrack_netlink nf_conntrack nfnetlink radeon hwmon drm_kms_helper ttm drm agpgart sr_mod cdrom ohci_pci radeonfb cfbfillrect ohci_hcd ehci_pci cfbimgblt ehci_hcd cfbcopyarea usbcore i2c_algo_bit usb_common fb_ddc pata_sil680 nls_base fb libata [49636.778369] CPU: 0 PID: 22599 Comm: dumpcap Tainted: G D L 3.17.7-gentoo-JeR #1 [49636.778369] task: 00000000438156a0 ti: 00000000513e0000 task.ti: 00000000513e0000 [49636.778369] [49636.778369] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [49636.778369] PSW: 00001000000011100000000000001111 Tainted: G D L [49636.778369] r00-03 00000000080e000f 00000000513e0db0 00000000402304d4 00000000513e0ce0 [49636.778369] r04-07 0000000040898080 00000000513e0ce8 0000000000000001 0000000000000001 [49636.778369] r08-11 0000000040871220 0000000000000000 00000000513e0ce8 00000000409542b8 [49636.778369] r12-15 00000000409541d0 00000000f767f000 00000000409541d0 0000000000000c08 [49636.778369] r16-19 0000000208905a00 00000000118c0b03 00000000513f6000 fffffffffe781000 [49636.778369] r20-23 0000000000000001 0000000000000020 0000000000000004 0000000000000004 [49636.778369] r24-27 00000000513e0ce8 000000000800000f 00000000427f9228 0000000040898080 [49636.778369] r28-31 0000000000000003 00000000513e0e10 00000000513e0db0 0000000000000001 [49636.778369] sr00-03 0000000004fa1800 0000000004fb4800 0000000000000000 0000000004fa1800 [49636.778369] sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [49636.778369] [49636.778369] IASQ: 0000000000000000 0000000000000000 IAOQ: 00000000402304f4 00000000402304e8 [49636.778369] IIR: 44bc0030 ISR: 00000000513e0e60 IOR: 0000000040898080 [49636.778369] CPU: 0 CR30: 00000000513e0000 CR31: ffffffffffffffff [49636.778369] ORIG_R28: 0000000040898080 [49636.778369] IAOQ[0]: generic_exec_single+0x1dc/0x210 [49636.778369] IAOQ[1]: generic_exec_single+0x1d0/0x210 [49636.778369] RP(r2): generic_exec_single+0x1bc/0x210 [49636.778369] Backtrace: [49636.778369] [<00000000402305e4>] smp_call_function_single+0xbc/0xd0 [49636.778369] [<0000000040230994>] smp_call_function_many+0x1ec/0x440 [49636.778369] [<0000000040230c40>] smp_call_function+0x58/0x70 [49636.778369] [<0000000040230c9c>] on_each_cpu+0x44/0x90 [49636.778369] [<00000000401790c4>] flush_tlb_all+0x13c/0x200 [49636.778369] [<00000000402a57c0>] tlb_flush_mmu_tlbonly+0x58/0x70 [49636.778369] [<00000000402a7080>] unmap_single_vma+0x838/0x8d8 [49636.778369] [<00000000402a87dc>] unmap_vmas+0x74/0xa8 [49636.778369] [<00000000402b2398>] exit_mmap+0xf0/0x268 [49636.778369] [<00000000401a20e0>] mmput+0xc8/0x218 [49636.778369] [<00000000401a8734>] do_exit+0x76c/0x1070 [49636.778369] [49636.781702] timer_interrupt(CPU 0): delayed! cycles A57CF051 rem 17FA2F next/now 2E5D4EAF6805/2E5D4E976DD6 [49639.985036] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=18028 jiffies, g=816488, c=816487, q=33) [49639.985036] Task dump for CPU 1: [49639.985036] top R running task 0 3563 3561 0x00000014 [49639.985036] Backtrace: [49639.985036] [<00000000401711d0>] __schedule+0x4c0/0x780 [49639.985036] [<000000004017154c>] schedule+0xbc/0xd0 [49639.985036] [<00000000402fc220>] __d_lookup+0xb0/0x1d8 [49639.985036] [49664.781700] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [dumpcap:22599] [49664.781700] Modules linked in: nfnetlink_queue nfnetlink_log auth_rpcgss oid_registry nfsv4 dns_resolver nfsv3 nfs lockd sunrpc nf_conntrack_netlink nf_conntrack nfnetlink radeon hwmon drm_kms_helper ttm drm agpgart sr_mod cdrom ohci_pci radeonfb cfbfillrect ohci_hcd ehci_pci cfbimgblt ehci_hcd cfbcopyarea usbcore i2c_algo_bit usb_common fb_ddc pata_sil680 nls_base fb libata [49664.781700] CPU: 0 PID: 22599 Comm: dumpcap Tainted: G D L 3.17.7-gentoo-JeR #1 [49664.781700] task: 00000000438156a0 ti: 00000000513e0000 task.ti: 00000000513e0000 [49664.781700] [49664.781700] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [49664.781700] PSW: 00001000000011100000000000001111 Tainted: G D L [49664.781700] r00-03 00000000080e000f 00000000513e0db0 00000000402304d4 00000000513e0ce0 [49664.781700] r04-07 0000000040898080 00000000513e0ce8 0000000000000001 0000000000000001 [49664.781700] r08-11 0000000040871220 0000000000000000 00000000513e0ce8 00000000409542b8 [49664.781700] r12-15 00000000409541d0 00000000f767f000 00000000409541d0 0000000000000c08 [49664.781700] r16-19 0000000208905a00 00000000118c0b03 00000000513f6000 fffffffffe781000 [49664.781700] r20-23 0000000000000001 0000000000000020 0000000000000004 0000000000000004 [49664.781700] r24-27 00000000513e0ce8 000000000800000f 00000000427f9228 0000000040898080 [49664.781700] r28-31 0000000000000003 00000000513e0e10 00000000513e0db0 0000000000000001 [49664.781700] sr00-03 0000000004fa1800 0000000004fb4800 0000000000000000 0000000004fa1800 [49664.781700] sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [49664.781700] [49664.781700] IASQ: 0000000000000000 0000000000000000 IAOQ: 00000000402304f4 00000000402304e8 [49664.781700] IIR: 44bc0030 ISR: 00000000513e0e60 IOR: 0000000040898080 [49664.781700] CPU: 0 CR30: 00000000513e0000 CR31: ffffffffffffffff [49664.781700] ORIG_R28: 0000000040898080 [49664.781700] IAOQ[0]: generic_exec_single+0x1dc/0x210 [49664.781700] IAOQ[1]: generic_exec_single+0x1d0/0x210 [49664.781700] RP(r2): generic_exec_single+0x1bc/0x210 [49664.781700] Backtrace: [49664.781700] [<00000000402305e4>] smp_call_function_single+0xbc/0xd0 [49664.781700] [<0000000040230994>] smp_call_function_many+0x1ec/0x440 [49664.781700] [<0000000040230c40>] smp_call_function+0x58/0x70 [49664.781700] [<0000000040230c9c>] on_each_cpu+0x44/0x90 [49664.781700] [<00000000401790c4>] flush_tlb_all+0x13c/0x200 [49664.781700] [<00000000402a57c0>] tlb_flush_mmu_tlbonly+0x58/0x70 [49664.781700] [<00000000402a7080>] unmap_single_vma+0x838/0x8d8 [49664.781700] [<00000000402a87dc>] unmap_vmas+0x74/0xa8 [49664.781700] [<00000000402b2398>] exit_mmap+0xf0/0x268 [49664.781700] [<00000000401a20e0>] mmput+0xc8/0x218 [49664.781700] [<00000000401a8734>] do_exit+0x76c/0x1070 [49664.781700] [49664.785033] timer_interrupt(CPU 0): delayed! cycles A57C88EA rem 186196 next/now 2E632CE6AAC5/2E632CCE492F and so on.
https://github.com/the-tcpdump-group/libpcap/pull/404 was apparently the wrong approach to the problem but has some interesting feedback.
I can't reproduce this with a 64-bit kernel compiled with GCC 4.8.3, so now I must assume the compiler causes this. With 4.8.3, the kernel simply stops passing captured packets to the capturer after a while: (strace of tcpdump:) _newselect(4, [3], NULL, NULL, {0, 250000}) = 0 (Timeout) gettimeofday({1425631202, 196101}, NULL) = 0 _newselect(4, [3], NULL, NULL, {0, 250000}) = 0 (Timeout) gettimeofday({1425631202, 452767}, NULL) = 0 _newselect(4, [3], NULL, NULL, {0, 250000}) = 0 (Timeout) gettimeofday({1425631202, 709433}, NULL) = 0 _newselect(4, [3], NULL, NULL, {0, 250000}) = 0 (Timeout) gettimeofday({1425631202, 966099}, NULL) = 0 _newselect(4, [3], NULL, NULL, {0, 250000}) = 0 (Timeout) gettimeofday({1425631203, 222764}, NULL) = 0 _newselect(4, [3], NULL, NULL, {0, 250000}) = 0 (Timeout) gettimeofday({1425631203, 479430}, NULL) = 0 ad inf. dumpcap behaves the same. No warning or error messages from the kernel, though. I'm seeing a possibly related HPMC when playing back sound on a Soundblaster Audigy, so I'm going to be testing that too with a 4.8.3 compiled kernel.
(In reply to Jeroen Roovers from comment #3) > (strace of tcpdump:) No, that was dumpcap, in fact.
Created attachment 398568 [details] lib-memcpy-4.8.3.s
Created attachment 398570 [details] lib-memcpy-4.9.2.s
https://github.com/the-tcpdump-group/libpcap/commit/2743ab9ee055c1500c5e41127dadf2e1be59e812
Long-fixed in tree.
