|

楼主 |
发表于 2022-2-15 14:25:06
|
显示全部楼层
Feb 15 13:56:59 kernel: rcu: INFO: rcu_preempt self-detected stall on CPUFeb 15 13:56:59 kernel: rcu: 1-...!: (60001 ticks this GP) idle=a2a/1/0x4000000000000002 softirq=38206947/38206948 fqs=91 Feb 15 13:56:59 kernel: rcu: (t=60000 jiffies g=65659837 q=5622)Feb 15 13:56:59 kernel: rcu: rcu_preempt kthread starved for 59636 jiffies! g65659837 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2Feb 15 13:56:59 kernel: rcu: RCU grace-period kthread stack dump:Feb 15 13:56:59 kernel: Call trace:Feb 15 13:56:59 kernel: __switch_to+0xe0/0x160Feb 15 13:56:59 kernel: __schedule+0x1ac/0x430Feb 15 13:56:59 kernel: schedule+0x38/0xa0Feb 15 13:56:59 kernel: schedule_timeout+0x19c/0x390Feb 15 13:56:59 kernel: rcu_gp_kthread+0x458/0x860Feb 15 13:56:59 kernel: kthread+0x118/0x150Feb 15 13:56:59 kernel: ret_from_fork+0x10/0x24Feb 15 13:56:59 kernel: Call trace:Feb 15 13:56:59 kernel: __switch_to+0x14c/0x160Feb 15 13:56:59 kernel: 0x60000005Feb 15 13:56:59 kernel: Call trace:Feb 15 13:56:59 kernel: dump_backtrace+0x0/0x150Feb 15 13:56:59 kernel: show_stack+0x14/0x20Feb 15 13:56:59 kernel: sched_show_task+0x128/0x160Feb 15 13:56:59 kernel: dump_cpu_task+0x40/0x4cFeb 15 13:56:59 kernel: rcu_dump_cpu_stacks+0x90/0xccFeb 15 13:56:59 kernel: rcu_check_callbacks+0x6b8/0x840Feb 15 13:56:59 kernel: update_process_times+0x2c/0x70Feb 15 13:56:59 kernel: tick_sched_handle.isra.0+0x30/0x50Feb 15 13:56:59 kernel: tick_sched_timer+0x48/0xa0Feb 15 13:56:59 kernel: __hrtimer_run_queues+0x110/0x1b0Feb 15 13:56:59 kernel: hrtimer_interrupt+0xe4/0x240Feb 15 13:56:59 kernel: arch_timer_handler_phys+0x30/0x40Feb 15 13:56:59 kernel: handle_percpu_devid_irq+0x80/0x140Feb 15 13:56:59 kernel: generic_handle_irq+0x24/0x40Feb 15 13:56:59 kernel: __handle_domain_irq+0x60/0xc0Feb 15 13:56:59 kernel: gic_handle_irq+0x5c/0xc0Feb 15 13:56:59 kernel: el1_irq+0xb0/0x128Feb 15 13:56:59 kernel: queued_spin_lock_slowpath+0x84/0x2f0Feb 15 13:56:59 kernel: _raw_spin_lock_bh+0x48/0x50Feb 15 13:56:59 kernel: emfc_input+0x8c/0x470 [emf]Feb 15 13:56:59 kernel: dhd_wmf_packets_handle+0xa0/0x210 [dhd]Feb 15 13:56:59 kernel: dhd_start_xmit_try+0x828/0xa10 [dhd]Feb 15 13:56:59 kernel: dhd_start_xmit+0x10/0x20 [dhd]Feb 15 13:56:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 13:56:59 kernel: sch_direct_xmit+0x120/0x310Feb 15 13:56:59 kernel: __qdisc_run+0x80/0xd0Feb 15 13:56:59 kernel: __dev_queue_xmit+0x520/0x820Feb 15 13:56:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 13:56:59 kernel: br_dev_queue_push_xmit+0x70/0x190Feb 15 13:56:59 kernel: br_forward_finish+0xc8/0xe0Feb 15 13:56:59 kernel: __br_forward.isra.0+0x114/0x1a0Feb 15 13:56:59 kernel: deliver_clone+0x58/0x90Feb 15 13:56:59 kernel: br_flood+0xfc/0x210Feb 15 13:56:59 kernel: br_dev_xmit+0x130/0x3c0Feb 15 13:56:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 13:56:59 kernel: __dev_queue_xmit+0x740/0x820Feb 15 13:56:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 13:56:59 kernel: ip_finish_output2+0x1d8/0x440Feb 15 13:56:59 kernel: ip_finish_output+0x148/0x210Feb 15 13:56:59 kernel: ip_mc_output+0x10c/0x320Feb 15 13:56:59 kernel: ip_local_out+0x44/0x60Feb 15 13:56:59 kernel: ip_send_skb+0x1c/0xb0Feb 15 13:56:59 kernel: udp_send_skb.isra.0+0x138/0x370Feb 15 13:56:59 kernel: udp_sendmsg+0x730/0x870Feb 15 13:56:59 kernel: inet_sendmsg+0x40/0xf0Feb 15 13:56:59 kernel: sock_sendmsg+0x18/0x30Feb 15 13:56:59 kernel: __sys_sendto+0xcc/0x130Feb 15 13:56:59 kernel: __arm64_sys_sendto+0x24/0x30Feb 15 13:56:59 kernel: el0_svc_common+0x70/0x170Feb 15 13:56:59 kernel: el0_svc_compat_handler+0x1c/0x30Feb 15 13:56:59 kernel: el0_svc_compat+0x8/0x18Feb 15 13:56:59 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:Feb 15 13:56:59 kernel: rcu: 0-...!: (13 ticks this GP) idle=c7e/0/0x1 softirq=32651862/32651873 fqs=26 Feb 15 13:56:59 kernel: rcu: 1-...!: (60103 ticks this GP) idle=a2a/1/0x4000000000000000 softirq=38206939/38206948 fqs=26 Feb 15 13:56:59 kernel: rcu: (detected by 2, t=60022 jiffies, g=25581401, q=140)Feb 15 13:56:59 kernel: Call trace:Feb 15 13:56:59 kernel: __switch_to+0x14c/0x160Feb 15 13:56:59 kernel: 0x60000005Feb 15 13:56:59 kernel: Call trace:Feb 15 13:56:59 kernel: __switch_to+0xe0/0x160Feb 15 13:56:59 kernel: 0x8400Feb 15 13:56:59 kernel: rcu: rcu_sched kthread starved for 59932 jiffies! g25581401 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2Feb 15 13:56:59 kernel: rcu: RCU grace-period kthread stack dump:Feb 15 13:56:59 kernel: Call trace:Feb 15 13:56:59 kernel: __switch_to+0x14c/0x160Feb 15 13:56:59 kernel: __schedule+0x1ac/0x430Feb 15 13:56:59 kernel: schedule+0x38/0xa0Feb 15 13:56:59 kernel: schedule_timeout+0x19c/0x390Feb 15 13:56:59 kernel: rcu_gp_kthread+0x458/0x860Feb 15 13:56:59 kernel: kthread+0x118/0x150Feb 15 13:56:59 kernel: ret_from_fork+0x10/0x24Feb 15 13:59:59 kernel: rcu: INFO: rcu_preempt self-detected stall on CPUFeb 15 13:59:59 kernel: rcu: 1-....: (239819 ticks this GP) idle=a2a/1/0x4000000000000002 softirq=38206947/38206948 fqs=45030 Feb 15 13:59:59 kernel: rcu: (t=240003 jiffies g=65659837 q=14164)Feb 15 13:59:59 kernel: Call trace:Feb 15 13:59:59 kernel: __switch_to+0x14c/0x160Feb 15 13:59:59 kernel: 0x60000005Feb 15 13:59:59 kernel: Call trace:Feb 15 13:59:59 kernel: dump_backtrace+0x0/0x150Feb 15 13:59:59 kernel: show_stack+0x14/0x20Feb 15 13:59:59 kernel: sched_show_task+0x128/0x160Feb 15 13:59:59 kernel: dump_cpu_task+0x40/0x4cFeb 15 13:59:59 kernel: rcu_dump_cpu_stacks+0x90/0xccFeb 15 13:59:59 kernel: rcu_check_callbacks+0x6b8/0x840Feb 15 13:59:59 kernel: update_process_times+0x2c/0x70Feb 15 13:59:59 kernel: tick_sched_handle.isra.0+0x30/0x50Feb 15 13:59:59 kernel: tick_sched_timer+0x48/0xa0Feb 15 13:59:59 kernel: __hrtimer_run_queues+0x110/0x1b0Feb 15 13:59:59 kernel: hrtimer_interrupt+0xe4/0x240Feb 15 13:59:59 kernel: arch_timer_handler_phys+0x30/0x40Feb 15 13:59:59 kernel: handle_percpu_devid_irq+0x80/0x140Feb 15 13:59:59 kernel: generic_handle_irq+0x24/0x40Feb 15 13:59:59 kernel: __handle_domain_irq+0x60/0xc0Feb 15 13:59:59 kernel: gic_handle_irq+0x5c/0xc0Feb 15 13:59:59 kernel: el1_irq+0xb0/0x128Feb 15 13:59:59 kernel: queued_spin_lock_slowpath+0x84/0x2f0Feb 15 13:59:59 kernel: _raw_spin_lock_bh+0x48/0x50Feb 15 13:59:59 kernel: emfc_input+0x8c/0x470 [emf]Feb 15 13:59:59 kernel: dhd_wmf_packets_handle+0xa0/0x210 [dhd]Feb 15 13:59:59 kernel: dhd_start_xmit_try+0x828/0xa10 [dhd]Feb 15 13:59:59 kernel: dhd_start_xmit+0x10/0x20 [dhd]Feb 15 13:59:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 13:59:59 kernel: sch_direct_xmit+0x120/0x310Feb 15 13:59:59 kernel: __qdisc_run+0x80/0xd0Feb 15 13:59:59 kernel: __dev_queue_xmit+0x520/0x820Feb 15 13:59:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 13:59:59 kernel: br_dev_queue_push_xmit+0x70/0x190Feb 15 13:59:59 kernel: br_forward_finish+0xc8/0xe0Feb 15 13:59:59 kernel: __br_forward.isra.0+0x114/0x1a0Feb 15 13:59:59 kernel: deliver_clone+0x58/0x90Feb 15 13:59:59 kernel: br_flood+0xfc/0x210Feb 15 13:59:59 kernel: br_dev_xmit+0x130/0x3c0Feb 15 13:59:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 13:59:59 kernel: __dev_queue_xmit+0x740/0x820Feb 15 13:59:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 13:59:59 kernel: ip_finish_output2+0x1d8/0x440Feb 15 13:59:59 kernel: ip_finish_output+0x148/0x210Feb 15 13:59:59 kernel: ip_mc_output+0x10c/0x320Feb 15 13:59:59 kernel: ip_local_out+0x44/0x60Feb 15 13:59:59 kernel: ip_send_skb+0x1c/0xb0Feb 15 13:59:59 kernel: udp_send_skb.isra.0+0x138/0x370Feb 15 13:59:59 kernel: udp_sendmsg+0x730/0x870Feb 15 13:59:59 kernel: inet_sendmsg+0x40/0xf0Feb 15 13:59:59 kernel: sock_sendmsg+0x18/0x30Feb 15 13:59:59 kernel: __sys_sendto+0xcc/0x130Feb 15 13:59:59 kernel: __arm64_sys_sendto+0x24/0x30Feb 15 13:59:59 kernel: el0_svc_common+0x70/0x170Feb 15 13:59:59 kernel: el0_svc_compat_handler+0x1c/0x30Feb 15 13:59:59 kernel: el0_svc_compat+0x8/0x18Feb 15 13:59:59 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:Feb 15 13:59:59 kernel: rcu: 0-....: (13 ticks this GP) idle=c7e/0/0x1 softirq=32651862/32651873 fqs=44802 Feb 15 13:59:59 kernel: rcu: 1-....: (239953 ticks this GP) idle=a2a/1/0x4000000000000000 softirq=38206939/38206948 fqs=44803 Feb 15 13:59:59 kernel: rcu: (detected by 2, t=240028 jiffies, g=25581401, q=140)Feb 15 13:59:59 kernel: Call trace:Feb 15 13:59:59 kernel: __switch_to+0x14c/0x160Feb 15 13:59:59 kernel: 0x60000005Feb 15 13:59:59 kernel: Call trace:Feb 15 13:59:59 kernel: __switch_to+0xe0/0x160Feb 15 13:59:59 kernel: 0x8400Feb 15 14:01:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:01:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:02:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:02:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:03:00 kernel: rcu: INFO: rcu_preempt self-detected stall on CPUFeb 15 14:03:00 kernel: rcu: 1-....: (419666 ticks this GP) idle=a2a/1/0x4000000000000002 softirq=38206947/38206948 fqs=89971 Feb 15 14:03:00 kernel: rcu: (t=420006 jiffies g=65659837 q=33877)Feb 15 14:03:00 kernel: Call trace:Feb 15 14:03:00 kernel: __switch_to+0x14c/0x160Feb 15 14:03:00 kernel: 0x60000005Feb 15 14:03:00 kernel: Call trace:Feb 15 14:03:00 kernel: dump_backtrace+0x0/0x150Feb 15 14:03:00 kernel: show_stack+0x14/0x20Feb 15 14:03:00 kernel: sched_show_task+0x128/0x160Feb 15 14:03:00 kernel: dump_cpu_task+0x40/0x4cFeb 15 14:03:00 kernel: rcu_dump_cpu_stacks+0x90/0xccFeb 15 14:03:00 kernel: rcu_check_callbacks+0x6b8/0x840Feb 15 14:03:00 kernel: update_process_times+0x2c/0x70Feb 15 14:03:00 kernel: tick_sched_handle.isra.0+0x30/0x50Feb 15 14:03:00 kernel: tick_sched_timer+0x48/0xa0Feb 15 14:03:00 kernel: __hrtimer_run_queues+0x110/0x1b0Feb 15 14:03:00 kernel: hrtimer_interrupt+0xe4/0x240Feb 15 14:03:00 kernel: arch_timer_handler_phys+0x30/0x40Feb 15 14:03:00 kernel: handle_percpu_devid_irq+0x80/0x140Feb 15 14:03:00 kernel: generic_handle_irq+0x24/0x40Feb 15 14:03:00 kernel: __handle_domain_irq+0x60/0xc0Feb 15 14:03:00 kernel: gic_handle_irq+0x5c/0xc0Feb 15 14:03:00 kernel: el1_irq+0xb0/0x128Feb 15 14:03:00 kernel: queued_spin_lock_slowpath+0x84/0x2f0Feb 15 14:03:00 kernel: _raw_spin_lock_bh+0x48/0x50Feb 15 14:03:00 kernel: emfc_input+0x8c/0x470 [emf]Feb 15 14:03:00 kernel: dhd_wmf_packets_handle+0xa0/0x210 [dhd]Feb 15 14:03:00 kernel: dhd_start_xmit_try+0x828/0xa10 [dhd]Feb 15 14:03:00 kernel: dhd_start_xmit+0x10/0x20 [dhd]Feb 15 14:03:00 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:03:00 kernel: sch_direct_xmit+0x120/0x310Feb 15 14:03:00 kernel: __qdisc_run+0x80/0xd0Feb 15 14:03:00 kernel: __dev_queue_xmit+0x520/0x820Feb 15 14:03:00 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:03:00 kernel: br_dev_queue_push_xmit+0x70/0x190Feb 15 14:03:00 kernel: br_forward_finish+0xc8/0xe0Feb 15 14:03:00 kernel: __br_forward.isra.0+0x114/0x1a0Feb 15 14:03:00 kernel: deliver_clone+0x58/0x90Feb 15 14:03:00 kernel: br_flood+0xfc/0x210Feb 15 14:03:00 kernel: br_dev_xmit+0x130/0x3c0Feb 15 14:03:00 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:03:00 kernel: __dev_queue_xmit+0x740/0x820Feb 15 14:03:00 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:03:00 kernel: ip_finish_output2+0x1d8/0x440Feb 15 14:03:00 kernel: ip_finish_output+0x148/0x210Feb 15 14:03:00 kernel: ip_mc_output+0x10c/0x320Feb 15 14:03:00 kernel: ip_local_out+0x44/0x60Feb 15 14:03:00 kernel: ip_send_skb+0x1c/0xb0Feb 15 14:03:00 kernel: udp_send_skb.isra.0+0x138/0x370Feb 15 14:03:00 kernel: udp_sendmsg+0x730/0x870Feb 15 14:03:00 kernel: inet_sendmsg+0x40/0xf0Feb 15 14:03:00 kernel: sock_sendmsg+0x18/0x30Feb 15 14:03:00 kernel: __sys_sendto+0xcc/0x130Feb 15 14:03:00 kernel: __arm64_sys_sendto+0x24/0x30Feb 15 14:03:00 kernel: el0_svc_common+0x70/0x170Feb 15 14:03:00 kernel: el0_svc_compat_handler+0x1c/0x30Feb 15 14:03:00 kernel: el0_svc_compat+0x8/0x18Feb 15 14:03:00 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:Feb 15 14:03:00 kernel: rcu: 0-....: (13 ticks this GP) idle=c7e/0/0x1 softirq=32651862/32651873 fqs=89740 Feb 15 14:03:00 kernel: rcu: 1-....: (419802 ticks this GP) idle=a2a/1/0x4000000000000000 softirq=38206939/38206948 fqs=89741 Feb 15 14:03:00 kernel: rcu: (detected by 2, t=420033 jiffies, g=25581401, q=141)Feb 15 14:03:00 kernel: Call trace:Feb 15 14:03:00 kernel: __switch_to+0x14c/0x160Feb 15 14:03:00 kernel: 0x60000005Feb 15 14:03:00 kernel: Call trace:Feb 15 14:03:00 kernel: __switch_to+0xe0/0x160Feb 15 14:03:00 kernel: 0x8400Feb 15 14:03:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:03:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:04:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:04:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:05:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:05:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:05:59 kernel: rcu: INFO: rcu_preempt self-detected stall on CPUFeb 15 14:05:59 kernel: rcu: 1-....: (599513 ticks this GP) idle=a2a/1/0x4000000000000002 softirq=38206947/38206948 fqs=134907 Feb 15 14:05:59 kernel: rcu: (t=600009 jiffies g=65659837 q=56541)Feb 15 14:05:59 kernel: Call trace:Feb 15 14:05:59 kernel: __switch_to+0x14c/0x160Feb 15 14:05:59 kernel: 0x60000005Feb 15 14:05:59 kernel: Call trace:Feb 15 14:05:59 kernel: dump_backtrace+0x0/0x150Feb 15 14:05:59 kernel: show_stack+0x14/0x20Feb 15 14:05:59 kernel: sched_show_task+0x128/0x160Feb 15 14:05:59 kernel: dump_cpu_task+0x40/0x4cFeb 15 14:05:59 kernel: rcu_dump_cpu_stacks+0x90/0xccFeb 15 14:05:59 kernel: rcu_check_callbacks+0x6b8/0x840Feb 15 14:05:59 kernel: update_process_times+0x2c/0x70Feb 15 14:05:59 kernel: tick_sched_handle.isra.0+0x30/0x50Feb 15 14:05:59 kernel: tick_sched_timer+0x48/0xa0Feb 15 14:05:59 kernel: __hrtimer_run_queues+0x110/0x1b0Feb 15 14:05:59 kernel: hrtimer_interrupt+0xe4/0x240Feb 15 14:05:59 kernel: arch_timer_handler_phys+0x30/0x40Feb 15 14:05:59 kernel: handle_percpu_devid_irq+0x80/0x140Feb 15 14:05:59 kernel: generic_handle_irq+0x24/0x40Feb 15 14:05:59 kernel: __handle_domain_irq+0x60/0xc0Feb 15 14:05:59 kernel: gic_handle_irq+0x5c/0xc0Feb 15 14:05:59 kernel: el1_irq+0xb0/0x128Feb 15 14:05:59 kernel: queued_spin_lock_slowpath+0x84/0x2f0Feb 15 14:05:59 kernel: _raw_spin_lock_bh+0x48/0x50Feb 15 14:05:59 kernel: emfc_input+0x8c/0x470 [emf]Feb 15 14:05:59 kernel: dhd_wmf_packets_handle+0xa0/0x210 [dhd]Feb 15 14:05:59 kernel: dhd_start_xmit_try+0x828/0xa10 [dhd]Feb 15 14:05:59 kernel: dhd_start_xmit+0x10/0x20 [dhd]Feb 15 14:05:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:05:59 kernel: sch_direct_xmit+0x120/0x310Feb 15 14:05:59 kernel: __qdisc_run+0x80/0xd0Feb 15 14:05:59 kernel: __dev_queue_xmit+0x520/0x820Feb 15 14:05:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:05:59 kernel: br_dev_queue_push_xmit+0x70/0x190Feb 15 14:05:59 kernel: br_forward_finish+0xc8/0xe0Feb 15 14:05:59 kernel: __br_forward.isra.0+0x114/0x1a0Feb 15 14:05:59 kernel: deliver_clone+0x58/0x90Feb 15 14:05:59 kernel: br_flood+0xfc/0x210Feb 15 14:05:59 kernel: br_dev_xmit+0x130/0x3c0Feb 15 14:05:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:05:59 kernel: __dev_queue_xmit+0x740/0x820Feb 15 14:05:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:05:59 kernel: ip_finish_output2+0x1d8/0x440Feb 15 14:05:59 kernel: ip_finish_output+0x148/0x210Feb 15 14:05:59 kernel: ip_mc_output+0x10c/0x320Feb 15 14:05:59 kernel: ip_local_out+0x44/0x60Feb 15 14:05:59 kernel: ip_send_skb+0x1c/0xb0Feb 15 14:05:59 kernel: udp_send_skb.isra.0+0x138/0x370Feb 15 14:05:59 kernel: udp_sendmsg+0x730/0x870Feb 15 14:05:59 kernel: inet_sendmsg+0x40/0xf0Feb 15 14:05:59 kernel: sock_sendmsg+0x18/0x30Feb 15 14:05:59 kernel: __sys_sendto+0xcc/0x130Feb 15 14:05:59 kernel: __arm64_sys_sendto+0x24/0x30Feb 15 14:05:59 kernel: el0_svc_common+0x70/0x170Feb 15 14:05:59 kernel: el0_svc_compat_handler+0x1c/0x30Feb 15 14:05:59 kernel: el0_svc_compat+0x8/0x18Feb 15 14:05:59 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:Feb 15 14:05:59 kernel: rcu: 0-....: (13 ticks this GP) idle=c7e/0/0x1 softirq=32651862/32651873 fqs=134672 Feb 15 14:05:59 kernel: rcu: 1-....: (599651 ticks this GP) idle=a2a/1/0x4000000000000000 softirq=38206939/38206948 fqs=134673 Feb 15 14:05:59 kernel: rcu: (detected by 2, t=600038 jiffies, g=25581401, q=141)Feb 15 14:05:59 kernel: Call trace:Feb 15 14:05:59 kernel: __switch_to+0x14c/0x160Feb 15 14:05:59 kernel: 0x60000005Feb 15 14:05:59 kernel: Call trace:Feb 15 14:05:59 kernel: __switch_to+0xe0/0x160Feb 15 14:05:59 kernel: 0x8400Feb 15 14:06:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:06:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:07:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:07:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:08:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:08:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:08:59 kernel: rcu: INFO: rcu_preempt self-detected stall on CPUFeb 15 14:08:59 kernel: rcu: 1-....: (779360 ticks this GP) idle=a2a/1/0x4000000000000002 softirq=38206947/38206948 fqs=179842 Feb 15 14:08:59 kernel: rcu: (t=780012 jiffies g=65659837 q=79574)Feb 15 14:08:59 kernel: Call trace:Feb 15 14:08:59 kernel: __switch_to+0x14c/0x160Feb 15 14:08:59 kernel: 0x60000005Feb 15 14:08:59 kernel: Call trace:Feb 15 14:08:59 kernel: dump_backtrace+0x0/0x150Feb 15 14:08:59 kernel: show_stack+0x14/0x20Feb 15 14:08:59 kernel: sched_show_task+0x128/0x160Feb 15 14:08:59 kernel: dump_cpu_task+0x40/0x4cFeb 15 14:08:59 kernel: rcu_dump_cpu_stacks+0x90/0xccFeb 15 14:08:59 kernel: rcu_check_callbacks+0x6b8/0x840Feb 15 14:08:59 kernel: update_process_times+0x2c/0x70Feb 15 14:08:59 kernel: tick_sched_handle.isra.0+0x30/0x50Feb 15 14:08:59 kernel: tick_sched_timer+0x48/0xa0Feb 15 14:08:59 kernel: __hrtimer_run_queues+0x110/0x1b0Feb 15 14:08:59 kernel: hrtimer_interrupt+0xe4/0x240Feb 15 14:08:59 kernel: arch_timer_handler_phys+0x30/0x40Feb 15 14:08:59 kernel: handle_percpu_devid_irq+0x80/0x140Feb 15 14:08:59 kernel: generic_handle_irq+0x24/0x40Feb 15 14:08:59 kernel: __handle_domain_irq+0x60/0xc0Feb 15 14:08:59 kernel: gic_handle_irq+0x5c/0xc0Feb 15 14:08:59 kernel: el1_irq+0xb0/0x128Feb 15 14:08:59 kernel: queued_spin_lock_slowpath+0x84/0x2f0Feb 15 14:08:59 kernel: _raw_spin_lock_bh+0x48/0x50Feb 15 14:08:59 kernel: emfc_input+0x8c/0x470 [emf]Feb 15 14:08:59 kernel: dhd_wmf_packets_handle+0xa0/0x210 [dhd]Feb 15 14:08:59 kernel: dhd_start_xmit_try+0x828/0xa10 [dhd]Feb 15 14:08:59 kernel: dhd_start_xmit+0x10/0x20 [dhd]Feb 15 14:08:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:08:59 kernel: sch_direct_xmit+0x120/0x310Feb 15 14:08:59 kernel: __qdisc_run+0x80/0xd0Feb 15 14:08:59 kernel: __dev_queue_xmit+0x520/0x820Feb 15 14:08:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:08:59 kernel: br_dev_queue_push_xmit+0x70/0x190Feb 15 14:08:59 kernel: br_forward_finish+0xc8/0xe0Feb 15 14:08:59 kernel: __br_forward.isra.0+0x114/0x1a0Feb 15 14:08:59 kernel: deliver_clone+0x58/0x90Feb 15 14:08:59 kernel: br_flood+0xfc/0x210Feb 15 14:08:59 kernel: br_dev_xmit+0x130/0x3c0Feb 15 14:08:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:08:59 kernel: __dev_queue_xmit+0x740/0x820Feb 15 14:08:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:08:59 kernel: ip_finish_output2+0x1d8/0x440Feb 15 14:08:59 kernel: ip_finish_output+0x148/0x210Feb 15 14:08:59 kernel: ip_mc_output+0x10c/0x320Feb 15 14:08:59 kernel: ip_local_out+0x44/0x60Feb 15 14:08:59 kernel: ip_send_skb+0x1c/0xb0Feb 15 14:08:59 kernel: udp_send_skb.isra.0+0x138/0x370Feb 15 14:08:59 kernel: udp_sendmsg+0x730/0x870Feb 15 14:08:59 kernel: inet_sendmsg+0x40/0xf0Feb 15 14:08:59 kernel: sock_sendmsg+0x18/0x30Feb 15 14:08:59 kernel: __sys_sendto+0xcc/0x130Feb 15 14:08:59 kernel: __arm64_sys_sendto+0x24/0x30Feb 15 14:08:59 kernel: el0_svc_common+0x70/0x170Feb 15 14:08:59 kernel: el0_svc_compat_handler+0x1c/0x30Feb 15 14:08:59 kernel: el0_svc_compat+0x8/0x18Feb 15 14:08:59 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:Feb 15 14:08:59 kernel: rcu: 0-....: (13 ticks this GP) idle=c7e/0/0x1 softirq=32651862/32651873 fqs=179605 Feb 15 14:08:59 kernel: rcu: 1-....: (779500 ticks this GP) idle=a2a/1/0x4000000000000000 softirq=38206939/38206948 fqs=179606 Feb 15 14:08:59 kernel: rcu: (detected by 2, t=780043 jiffies, g=25581401, q=141)Feb 15 14:08:59 kernel: Call trace:Feb 15 14:08:59 kernel: __switch_to+0x14c/0x160Feb 15 14:08:59 kernel: 0x60000005Feb 15 14:08:59 kernel: Call trace:Feb 15 14:08:59 kernel: __switch_to+0xe0/0x160Feb 15 14:08:59 kernel: 0x8400Feb 15 14:09:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:09:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:10:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:10:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:11:43 check_watchdog: [check_watchdog] restart watchdog for no heartbeatFeb 15 14:11:43 rc_service: check_watchdog 2927:notify_rc restart_watchdogFeb 15 14:11:59 kernel: rcu: INFO: rcu_preempt self-detected stall on CPUFeb 15 14:11:59 kernel: rcu: 1-....: (959207 ticks this GP) idle=a2a/1/0x4000000000000002 softirq=38206947/38206948 fqs=224776 Feb 15 14:11:59 kernel: rcu: (t=960015 jiffies g=65659837 q=102993)Feb 15 14:11:59 kernel: Call trace:Feb 15 14:11:59 kernel: __switch_to+0x14c/0x160Feb 15 14:11:59 kernel: 0x60000005Feb 15 14:11:59 kernel: Call trace:Feb 15 14:11:59 kernel: dump_backtrace+0x0/0x150Feb 15 14:11:59 kernel: show_stack+0x14/0x20Feb 15 14:11:59 kernel: sched_show_task+0x128/0x160Feb 15 14:11:59 kernel: dump_cpu_task+0x40/0x4cFeb 15 14:11:59 kernel: rcu_dump_cpu_stacks+0x90/0xccFeb 15 14:11:59 kernel: rcu_check_callbacks+0x6b8/0x840Feb 15 14:11:59 kernel: update_process_times+0x2c/0x70Feb 15 14:11:59 kernel: tick_sched_handle.isra.0+0x30/0x50Feb 15 14:11:59 kernel: tick_sched_timer+0x48/0xa0Feb 15 14:11:59 kernel: __hrtimer_run_queues+0x110/0x1b0Feb 15 14:11:59 kernel: hrtimer_interrupt+0xe4/0x240Feb 15 14:11:59 kernel: arch_timer_handler_phys+0x30/0x40Feb 15 14:11:59 kernel: handle_percpu_devid_irq+0x80/0x140Feb 15 14:11:59 kernel: generic_handle_irq+0x24/0x40Feb 15 14:11:59 kernel: __handle_domain_irq+0x60/0xc0Feb 15 14:11:59 kernel: gic_handle_irq+0x5c/0xc0Feb 15 14:11:59 kernel: el1_irq+0xb0/0x128Feb 15 14:11:59 kernel: queued_spin_lock_slowpath+0x84/0x2f0Feb 15 14:11:59 kernel: _raw_spin_lock_bh+0x48/0x50Feb 15 14:11:59 kernel: emfc_input+0x8c/0x470 [emf]Feb 15 14:11:59 kernel: dhd_wmf_packets_handle+0xa0/0x210 [dhd]Feb 15 14:11:59 kernel: dhd_start_xmit_try+0x828/0xa10 [dhd]Feb 15 14:11:59 kernel: dhd_start_xmit+0x10/0x20 [dhd]Feb 15 14:11:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:11:59 kernel: sch_direct_xmit+0x120/0x310Feb 15 14:11:59 kernel: __qdisc_run+0x80/0xd0Feb 15 14:11:59 kernel: __dev_queue_xmit+0x520/0x820Feb 15 14:11:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:11:59 kernel: br_dev_queue_push_xmit+0x70/0x190Feb 15 14:11:59 kernel: br_forward_finish+0xc8/0xe0Feb 15 14:11:59 kernel: __br_forward.isra.0+0x114/0x1a0Feb 15 14:11:59 kernel: deliver_clone+0x58/0x90Feb 15 14:11:59 kernel: br_flood+0xfc/0x210Feb 15 14:11:59 kernel: br_dev_xmit+0x130/0x3c0Feb 15 14:11:59 kernel: dev_hard_start_xmit+0x88/0x180Feb 15 14:11:59 kernel: __dev_queue_xmit+0x740/0x820Feb 15 14:11:59 kernel: dev_queue_xmit+0x10/0x20Feb 15 14:11:59 kernel: ip_finish_output2+0x1d8/0x440Feb 15 14:11:59 kernel: ip_finish_output+0x148/0x210Feb 15 14:11:59 kernel: ip_mc_output+0x10c/0x320Feb 15 14:11:59 kernel: ip_local_out+0x44/0x60Feb 15 14:11:59 kernel: ip_send_skb+0x1c/0xb0Feb 15 14:11:59 kernel: udp_send_skb.isra.0+0x138/0x370Feb 15 14:11:59 kernel: udp_sendmsg+0x730/0x870Feb 15 14:11:59 kernel: inet_sendmsg+0x40/0xf0Feb 15 14:11:59 kernel: sock_sendmsg+0x18/0x30Feb 15 14:11:59 kernel: __sys_sendto+0xcc/0x130Feb 15 14:11:59 kernel: __arm64_sys_sendto+0x24/0x30Feb 15 14:11:59 kernel: el0_svc_common+0x70/0x170Feb 15 14:11:59 kernel: el0_svc_compat_handler+0x1c/0x30Feb 15 14:11:59 kernel: el0_svc_compat+0x8/0x18Feb 15 14:11:59 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:Feb 15 14:11:59 kernel: rcu: 0-....: (13 ticks this GP) idle=c7e/0/0x1 softirq=32651862/32651873 fqs=224536 Feb 15 14:11:59 kernel: rcu: 1-....: (959349 ticks this GP) idle=a2a/1/0x4000000000000000 softirq=38206939/38206948 fqs=224537 Feb 15 14:11:59 kernel: rcu: (detected by 2, t=960048 jiffies, g=25581401, q=141)Feb 15 14:11:59 kernel: Call trace:Feb 15 14:11:59 kernel: __switch_to+0x14c/0x160Feb 15 14:11:59 kernel: 0x60000005Feb 15 14:11:59 kernel: Call trace:Feb 15 14:11:59 kernel: __switch_to+0xe0/0x160Feb 15 14:11:59 kernel: 0x8400May 5 13:05:15 kernel: klogd started: BusyBox v1.24.1 (2022-01-27 16:09:23 PST)这次重启花了两分钟前面各种报错持续了二十来分钟,日志有记录,已经点回报
Feb 15 14:13:34 rc_service: ntp 3473:notify_rc restart_diskmon
|
|