Another RCU stall of the same boot

rcu-4.txt (text/plain), 9.96 KB, created by satmd on 2023-06-24 08:20:12 UTC

Jun 23 19:58:47 naomi kernel: igb 0000:01:00.0 eno1: Reset adapter
Jun 23 19:58:49 naomi kernel: igb 0000:01:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jun 23 19:58:50 naomi kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Jun 23 19:58:50 naomi kernel: rcu: \x096-....: (1 GPs behind) idle=5ffc/1/0x4000000000000004 softirq=3096/3096 fqs=815
Jun 23 19:58:50 naomi kernel: rcu: \x09         hardirqs   softirqs   csw/system
Jun 23 19:58:50 naomi kernel: rcu: \x09 number:       10          0            0
Jun 23 19:58:50 naomi kernel: rcu: \x09cputime:        0      10480            0   ==> 10480(ms)
Jun 23 19:58:50 naomi kernel: rcu: \x09(t=2101 jiffies g=1705 q=2398 ncpus=12)
Jun 23 19:58:50 naomi kernel: CPU: 6 PID: 0 Comm: swapper/6 Tainted: G        W       T  6.3.9-gentoo #1
Jun 23 19:58:50 naomi kernel: Hardware name: Dell Inc. PowerEdge R620/0VV3F2, BIOS 2.9.0 12/06/2019
Jun 23 19:58:50 naomi kernel: RIP: 0010:wg_packet_receive+0x776/0x860 [wireguard]
Jun 23 19:58:50 naomi kernel: Code: 63 c1 48 83 f8 3f 0f 87 e4 00 00 00 48 89 f2 b8 40 00 00 00 48 d3 e2 48 f7 da 48 23 15 63 b4 88 ca 74 05 f3 48 0f bc c2 31 d2 <f7> 35 34 b6 88 ca 89 d0 49 89 c4 48 0f a3 05 47 b4 88 ca 0f 82 9d
Jun 23 19:58:50 naomi kernel: RSP: 0018:ffffa50240364b48 EFLAGS: 00000246
Jun 23 19:58:50 naomi kernel: RAX: 0000000000000040 RBX: ffff9176cd39c700 RCX: 0000000000000011
Jun 23 19:58:50 naomi kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9176cabfcea0
Jun 23 19:58:50 naomi kernel: RBP: ffff9176c6046900 R08: 000000000000000e R09: 0000000000000008
Jun 23 19:58:50 naomi kernel: R10: ffffffff8b76e880 R11: 0000000000006d33 R12: 0000000000000010
Jun 23 19:58:50 naomi kernel: R13: ffff9176cb91f000 R14: ffff9176c42b10ce R15: ffffa50240364cc8
Jun 23 19:58:50 naomi kernel: FS:  0000000000000000(0000) GS:ffff9181bf380000(0000) knlGS:0000000000000000
Jun 23 19:58:50 naomi kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 23 19:58:50 naomi kernel: CR2: 00007fa4675f49a0 CR3: 0000000105ad8005 CR4: 00000000001706e0
Jun 23 19:58:50 naomi kernel: Call Trace:
Jun 23 19:58:50 naomi kernel:  <IRQ>
Jun 23 19:58:50 naomi kernel:  ? rcu_dump_cpu_stacks+0xea/0x170
Jun 23 19:58:50 naomi kernel:  ? rcu_sched_clock_irq+0x500/0x1010
Jun 23 19:58:50 naomi kernel:  ? ep_poll_callback+0x224/0x270
Jun 23 19:58:50 naomi kernel:  ? perf_event_task_tick+0x42/0x70
Jun 23 19:58:50 naomi kernel:  ? update_process_times+0x57/0x90
Jun 23 19:58:50 naomi kernel:  ? tick_sched_timer+0xc8/0xf0
Jun 23 19:58:50 naomi kernel:  ? __pfx_tick_sched_timer+0x10/0x10
Jun 23 19:58:50 naomi kernel:  ? __hrtimer_run_queues+0x12b/0x2d0
Jun 23 19:58:50 naomi kernel:  ? hrtimer_interrupt+0xf4/0x230
Jun 23 19:58:50 naomi kernel:  ? __sysvec_apic_timer_interrupt+0x62/0x140
Jun 23 19:58:50 naomi kernel:  ? sysvec_apic_timer_interrupt+0x35/0x90
Jun 23 19:58:50 naomi kernel:  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Jun 23 19:58:50 naomi kernel:  ? wg_packet_receive+0x776/0x860 [wireguard]
Jun 23 19:58:50 naomi kernel:  ? wg_packet_receive+0x40d/0x860 [wireguard]
Jun 23 19:58:50 naomi kernel:  ? __udp4_lib_rcv+0x435/0xaf0
Jun 23 19:58:50 naomi kernel:  ? __pfx_wg_receive+0x10/0x10 [wireguard]
Jun 23 19:58:50 naomi kernel:  wg_receive+0x25/0x40 [wireguard]
Jun 23 19:58:50 naomi kernel:  udp_queue_rcv_one_skb+0x3be/0x5a0
Jun 23 19:58:50 naomi kernel:  udp_unicast_rcv_skb+0x75/0x90
Jun 23 19:58:50 naomi kernel:  ip_protocol_deliver_rcu+0x120/0x2f0
Jun 23 19:58:50 naomi kernel:  ip_local_deliver_finish+0x7a/0xa0
Jun 23 19:58:50 naomi kernel:  ip_sublist_rcv_finish+0x64/0x70
Jun 23 19:58:50 naomi kernel:  ip_sublist_rcv+0x160/0x1a0
Jun 23 19:58:50 naomi kernel:  ? bond_handle_frame+0x58/0x250 [bonding]
Jun 23 19:58:50 naomi kernel:  ? __pfx_ip_rcv_finish+0x10/0x10
Jun 23 19:58:50 naomi kernel:  ip_list_rcv+0xff/0x130
Jun 23 19:58:50 naomi kernel:  __netif_receive_skb_list_core+0x228/0x250
Jun 23 19:58:50 naomi kernel:  netif_receive_skb_list_internal+0x1c9/0x2f0
Jun 23 19:58:50 naomi kernel:  napi_complete_done+0x6e/0x180
Jun 23 19:58:50 naomi kernel:  igb_poll+0x8d2/0x1450 [igb]
Jun 23 19:58:50 naomi kernel:  __napi_poll+0x2b/0x1c0
Jun 23 19:58:50 naomi kernel:  net_rx_action+0x134/0x260
Jun 23 19:58:50 naomi kernel:  ? __napi_schedule+0x69/0x80
Jun 23 19:58:50 naomi kernel:  __do_softirq+0xd8/0x2c9
Jun 23 19:58:50 naomi kernel:  common_interrupt+0x82/0xa0
Jun 23 19:58:50 naomi kernel:  </IRQ>
Jun 23 19:58:50 naomi kernel:  asm_common_interrupt+0x22/0x40
Jun 23 19:58:50 naomi kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x440
Jun 23 19:58:50 naomi kernel: Code: ff ff 8b 53 04 49 89 c5 8b 05 e8 f4 bd 00 85 c0 0f 8f e8 02 00 00 31 ff e8 31 4f 53 ff 45 84 ff 0f 85 f8 01 00 00 fb 45 85 f6 <0f> 88 2f 01 00 00 48 8b 04 24 49 63 ce 4c 89 ea 48 6b f1 68 48 29
Jun 23 19:58:50 naomi kernel: RSP: 0018:ffffa502400dbea0 EFLAGS: 00000202
Jun 23 19:58:50 naomi kernel: RAX: ffff9181bf3aaa00 RBX: ffffc5023f380108 RCX: 0000000000000000
Jun 23 19:58:50 naomi kernel: RDX: 0000000000000006 RSI: 0000000000000002 RDI: 0000000000000000
Jun 23 19:58:50 naomi kernel: RBP: 0000000000000001 R08: fffffd5266cb75ed R09: 000000003cf3d124
Jun 23 19:58:50 naomi kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff8af9ee60
Jun 23 19:58:50 naomi kernel: R13: 0000000c588beca6 R14: 0000000000000001 R15: 0000000000000000
Jun 23 19:58:50 naomi kernel:  cpuidle_enter+0x2f/0x40
Jun 23 19:58:50 naomi kernel:  do_idle+0x165/0x190
Jun 23 19:58:50 naomi kernel:  cpu_startup_entry+0x19/0x20
Jun 23 19:58:50 naomi kernel:  start_secondary+0xe5/0xf0
Jun 23 19:58:50 naomi kernel:  secondary_startup_64_no_verify+0xe0/0xeb
Jun 23 19:58:50 naomi kernel:  </TASK>
Jun 23 19:58:50 naomi kernel: Sending NMI from CPU 6 to CPUs 9:
Jun 23 19:58:50 naomi kernel: NMI backtrace for cpu 9
Jun 23 19:58:50 naomi kernel: CPU: 9 PID: 99 Comm: kworker/9:1 Tainted: G        W       T  6.3.9-gentoo #1
Jun 23 19:58:50 naomi kernel: Hardware name: Dell Inc. PowerEdge R620/0VV3F2, BIOS 2.9.0 12/06/2019
Jun 23 19:58:50 naomi kernel: Workqueue: wg-crypt-wg0 wg_packet_decrypt_worker [wireguard]
Jun 23 19:58:50 naomi kernel: RIP: 0010:wg_packet_send_staged_packets+0x5e5/0x6d0 [wireguard]
Jun 23 19:58:50 naomi kernel: Code: 17 c9 e9 ff fc ff ff b8 40 00 00 00 e9 51 fe ff ff 89 c2 8d 4a 01 48 63 c1 48 83 f8 3f 0f 87 e1 00 00 00 48 c7 c2 ff ff ff ff <b8> 40 00 00 00 48 d3 e2 48 23 15 0c cf 88 ca 74 07 f3 48 0f bc d2
Jun 23 19:58:50 naomi kernel: RSP: 0018:ffffa502404007f0 EFLAGS: 00000297
Jun 23 19:58:50 naomi kernel: RAX: 0000000000000011 RBX: ffff9176c6046900 RCX: 0000000000000011
Jun 23 19:58:50 naomi kernel: RDX: ffffffffffffffff RSI: ffff9176c9c65d00 RDI: ffff9176cabfce70
Jun 23 19:58:50 naomi kernel: RBP: ffffa50240400870 R08: 0000000000000002 R09: 0000000000000098
Jun 23 19:58:50 naomi kernel: R10: ffff9176c7151298 R11: ffff9176cbb0cc28 R12: ffff9176c9c65d00
Jun 23 19:58:50 naomi kernel: R13: 0000000000000010 R14: ffff9176cb91f000 R15: ffffa50240400820
Jun 23 19:58:50 naomi kernel: FS:  0000000000000000(0000) GS:ffff9181bf440000(0000) knlGS:0000000000000000
Jun 23 19:58:50 naomi kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 23 19:58:50 naomi kernel: CR2: 00007f3c9ed9b6dc CR3: 00000007b1e2a001 CR4: 00000000001706e0
Jun 23 19:58:50 naomi kernel: Call Trace:
Jun 23 19:58:50 naomi kernel:  <NMI>
Jun 23 19:58:50 naomi kernel:  ? nmi_cpu_backtrace+0x84/0xf0
Jun 23 19:58:50 naomi kernel:  ? nmi_cpu_backtrace_handler+0xd/0x20
Jun 23 19:58:50 naomi kernel:  ? nmi_handle+0x5d/0x160
Jun 23 19:58:50 naomi kernel:  ? default_do_nmi+0x40/0x100
Jun 23 19:58:50 naomi kernel:  ? exc_nmi+0xf7/0x120
Jun 23 19:58:50 naomi kernel:  ? end_repeat_nmi+0x16/0x67
Jun 23 19:58:50 naomi kernel:  ? wg_packet_send_staged_packets+0x5e5/0x6d0 [wireguard]
Jun 23 19:58:50 naomi kernel:  ? wg_packet_send_staged_packets+0x5e5/0x6d0 [wireguard]
Jun 23 19:58:50 naomi kernel:  ? wg_packet_send_staged_packets+0x5e5/0x6d0 [wireguard]
Jun 23 19:58:50 naomi kernel:  </NMI>
Jun 23 19:58:50 naomi kernel:  <IRQ>
Jun 23 19:58:50 naomi kernel:  wg_xmit+0x2c5/0x380 [wireguard]
Jun 23 19:58:50 naomi kernel:  dev_hard_start_xmit+0xd7/0x220
Jun 23 19:58:50 naomi kernel:  __dev_queue_xmit+0x274/0xce0
Jun 23 19:58:50 naomi kernel:  ? nf_nat_ipv6_fn+0x65/0xf0 [nf_nat]
Jun 23 19:58:50 naomi kernel:  ip6_finish_output2+0x1cc/0x6d0
Jun 23 19:58:50 naomi kernel:  ? ip6_output+0xe9/0x160
Jun 23 19:58:50 naomi kernel:  ? ip6_mtu+0x3e/0x70
Jun 23 19:58:50 naomi kernel:  ip6_send_skb+0x1e/0x70
Jun 23 19:58:50 naomi kernel:  icmpv6_echo_reply+0x5f3/0x610
Jun 23 19:58:50 naomi kernel:  ? icmpv6_rcv+0x607/0x6c0
Jun 23 19:58:50 naomi kernel:  icmpv6_rcv+0x607/0x6c0
Jun 23 19:58:50 naomi kernel:  ip6_protocol_deliver_rcu+0x3e2/0x5c0
Jun 23 19:58:50 naomi kernel:  ip6_input+0xdd/0x100
Jun 23 19:58:50 naomi kernel:  ? __pfx_ip6_input_finish+0x10/0x10
Jun 23 19:58:50 naomi kernel:  ip6_sublist_rcv_finish+0x45/0x70
Jun 23 19:58:50 naomi kernel:  ip6_sublist_rcv+0x20e/0x260
Jun 23 19:58:50 naomi kernel:  ? __pfx_ip6_rcv_finish+0x10/0x10
Jun 23 19:58:50 naomi kernel:  ipv6_list_rcv+0x102/0x130
Jun 23 19:58:50 naomi kernel:  __netif_receive_skb_list_core+0x1a3/0x250
Jun 23 19:58:50 naomi kernel:  netif_receive_skb_list_internal+0x1c9/0x2f0
Jun 23 19:58:50 naomi kernel:  ? __mod_timer+0x11f/0x310
Jun 23 19:58:50 naomi kernel:  napi_complete_done+0x6e/0x180
Jun 23 19:58:50 naomi kernel:  wg_packet_rx_poll+0x3b1/0x730 [wireguard]
Jun 23 19:58:50 naomi kernel:  __napi_poll+0x2b/0x1c0
Jun 23 19:58:50 naomi kernel:  net_rx_action+0x134/0x260
Jun 23 19:58:50 naomi kernel:  __do_softirq+0xd8/0x2c9
Jun 23 19:58:50 naomi kernel:  do_softirq+0x63/0x80
Jun 23 19:58:50 naomi kernel:  </IRQ>
Jun 23 19:58:50 naomi kernel:  <TASK>
Jun 23 19:58:50 naomi kernel:  __local_bh_enable_ip+0x64/0x70
Jun 23 19:58:50 naomi kernel:  process_one_work+0x1f2/0x440
Jun 23 19:58:50 naomi kernel:  worker_thread+0x4e/0x3c0
Jun 23 19:58:50 naomi kernel:  ? __pfx_worker_thread+0x10/0x10
Jun 23 19:58:50 naomi kernel:  kthread+0xdd/0x110
Jun 23 19:58:50 naomi kernel:  ? __pfx_kthread+0x10/0x10
Jun 23 19:58:50 naomi kernel:  ret_from_fork+0x2c/0x50
Jun 23 19:58:50 naomi kernel:  </TASK>