Application Freeze

[asadilan]

I have PXIe 8840 on Linux RT 21.8.

I have application freezing and the following was found in the kernel log. 

2023-06-22T13:15:04.925+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.020555] ACPI: NMI not connected to LINT 1!
2023-06-22T13:15:04.925+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.020556] ACPI: NMI not connected to LINT 1!
2023-06-22T13:15:04.925+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.020556] ACPI: NMI not connected to LINT 1!
2023-06-22T13:15:04.925+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.020557] ACPI: NMI not connected to LINT 1!
2023-06-22T13:15:05.015+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.020578] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x17 (or later)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.501130] Warning: Processor Platform Limit event detected, but not handled.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.501131] Consider compiling CPUfreq support into your kernel.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.568449] usb: port power management may be unreliable
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 0.572882] Warning: unable to open an initial console.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 2.098389] ext4 filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 3.143018] ext4 filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 3.225381] ext4 filesystem being mounted at /etc/natinst/share supports timestamps until 2038 (0x7fffffff)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254449] ACPI Warning: SystemIO range 0x0000000000001828-0x000000000000182F conflicts with OpRegion 0x0000000000001800-0x000000000000187F (\PMIO) (20200925/utaddress-213)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254456] ACPI Warning: SystemIO range 0x0000000000001C40-0x0000000000001C4F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR) (20200925/utaddress-213)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254459] ACPI Warning: SystemIO range 0x0000000000001C30-0x0000000000001C3F conflicts with OpRegion 0x0000000000001C00-0x0000000000001C3F (\GPRL) (20200925/utaddress-213)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254461] ACPI Warning: SystemIO range 0x0000000000001C30-0x0000000000001C3F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR) (20200925/utaddress-213)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254463] ACPI Warning: SystemIO range 0x0000000000001C00-0x0000000000001C2F conflicts with OpRegion 0x0000000000001C00-0x0000000000001C3F (\GPRL) (20200925/utaddress-213)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254464] ACPI Warning: SystemIO range 0x0000000000001C00-0x0000000000001C2F conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR) (20200925/utaddress-213)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.254466] lpc_ich: Resource conflict(s) found affecting gpio_ich
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.477438] nikal: loading out-of-tree module taints kernel.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.477438] nikal: loading out-of-tree module taints kernel.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.477441] nikal: module license 'Copyright (c) 2002-2022 National Instruments Corporation. All Rights Reserved. Any and all use of the copyrighted materials is subject to the then current terms and conditions of the applicable license agreement, which can be found at <http://www.ni.com/linux/>.' taints kernel.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.477441] nikal: module license 'Copyright (c) 2002-2022 National Instruments Corporation. All Rights Reserved. Any and all use of the copyrighted materials is subject to the then current terms and conditions of the applicable license agreement, which can be found at <http://www.ni.com/linux/>.' taints kernel.
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.477442] Disabling lock debugging due to kernel taint
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 4.852171] i915: unknown parameter 'enable_rc6' ignored
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 5.218439] ext4 filesystem being mounted at /etc/natinst/share supports timestamps until 2038 (0x7fffffff)
2023-06-22T13:15:05.018+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 5.312093] ext4 filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)
2023-06-22T13:15:21.422+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [ 35.223958] process 'local/natinst/labview/lvrt' started with executable stack
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067309] rcu: INFO: rcu_preempt self-detected stall on CPU
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067312] rcu: 1-....: (6101 ticks this GP) idle=5ba/1/0x4000000000000000 softirq=0/0 fqs=2302
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067316] (t=6001 jiffies g=4276329 q=137)
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067317] NMI backtrace for cpu 1
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067318] CPU: 1 PID: 3519 Comm: L586770108 Tainted: P U O 5.10.106-rt64 #1
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067320] Hardware name: National Instruments NI PXIe-8840 Quad-Core/NI PXIe-8840 Quad-Core, BIOS 2.1.10f0 02/03/2021
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067321] Call Trace:
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067323] <IRQ>
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067324] ? dump_stack+0x57/0x6a
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067328] ? nmi_cpu_backtrace+0x99/0xb0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067331] ? lapic_can_unplug_cpu+0xd0/0xd0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067334] ? nmi_trigger_cpumask_backtrace+0xc7/0x110
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067336] ? rcu_dump_cpu_stacks+0xa9/0xd5
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067337] ? rcu_sched_clock_irq+0x871/0xcf0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067339] ? update_rt_rq_load_avg+0x1f7/0x2f0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067341] ? update_rt_rq_load_avg+0x1f7/0x2f0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067342] ? cpuacct_account_field+0x16/0x60
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067344] ? tick_init_highres+0x20/0x20
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067346] ? update_process_times+0x55/0x80
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067347] ? tick_sched_timer+0x7a/0x1d0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067349] ? __hrtimer_run_queues+0x123/0x380
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067351] ? hrtimer_interrupt+0xe5/0x240
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067352] ? irq_work_queue_on+0xf1/0x110
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067354] ? __sysvec_apic_timer_interrupt+0x62/0x1a0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067356] ? asm_call_irq_on_stack+0xf/0x20
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067357] </IRQ>
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067358] ? sysvec_apic_timer_interrupt+0x52/0x80
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067361] ? asm_sysvec_apic_timer_interrupt+0x12/0x20
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067362] ? rt_spin_lock+0x16/0x40
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067364] ? futex_requeue+0x554/0x890
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067367] ? do_futex+0x2dd/0xa60
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067369] ? __schedule+0x417/0x5b0
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067370] ? run_posix_cpu_timers+0xbc/0x150
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067372] ? sysvec_irq_work+0x5e/0x80
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067374] ? asm_sysvec_irq_work+0x12/0x20
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067375] ? __x64_sys_futex+0x134/0x180
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067377] ? cpumask_next+0x16/0x20
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067379] ? rto_push_irq_work_func+0x97/0x140
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067381] ? irq_work_single+0x2e/0x60
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067383] ? do_syscall_64+0x33/0x40
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067385] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9

Anyone has ever experienced this processor stalling? 

TIA

[gratian.crisan]

RCU stalls like this:

2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067309] rcu: INFO: rcu_preempt self-detected stall on CPU
2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067312] rcu: 1-....: (6101 ticks this GP) idle=5ba/1/0x4000000000000000 softirq=0/0 fqs=2302

are usually a sign that one of the CPUs is overloaded running something at high priority and the background work the kernel is trying to do on that CPU got delayed. If the work gets delayed long enough it can cause the system to run out of memory and behave unpredictably.

It looks like the stall happened on CPU 1 which at the time was running a timed loop, specifically one labeled "L586770108":

2023-06-28T08:25:01.707+00:00 NI-PXIe-8840Quad-Core-P4S-Arriel kernel: [500909.067318] CPU: 1 PID: 3519 Comm: L586770108 Tainted: P U O 5.10.106-rt64 #1

There are several possible root causes:

• you could be hitting a bug in the kernel or the OS in which case I would recommend trying a newer version of the base system image (you can install an older version of LabVIEW RT on a newer base OS system image).
• it is possible that the timed loop in question "L586770108" uses 100% of a CPU and is causing this issue. Make sure you have the loop rate set appropriately to leave a bit of a headroom on that CPU. Also make sure you don't have other (for/while) loops inside that timed loop that monopolize the CPU for a long time.