cRIO: RT Linux locks up when running simple application

[bcl511]

Hello,

I am having trouble with my cRIO-9040.  The controller will periodically freeze when running a simple application.  It is not only the RT application that freezes - the entire linux kernel becomes unresponsive and I am forced to power cycle the device to regain control.  Sometimes this happens after only 10 minutes, and sometimes it runs for 1-2 hours, but never more.  My program controls a chemical process so I am reading temperature, flow, and pressure data (from a mix of c-series modules and MODBUS (RS485) devices) once a second and displaying the data in an embedded-UI user interface.  

While investigating this lock-up I found that I can also reliably cause the crash to happen by simply displaying the index of a while loop to the screen at 100Hz or faster:

This is my first cRIO project with the embedded UI, so it is possible that I am making a silly mistake.  I would understand if the vi just failed to work properly, but crashing the whole linux kernel is a bit of a surprise.  The actual program (that also crashes linux after an hour or two) is attached.  Does anyone have any suggestions?  

Many thanks!

[rolfk]

This sounds weird. I have done much more complicated applications using the integrated UI on both 903x and 904x controllers with several dozen top level VIs running potentially in parallel and displaying their front panel, so if your simple loop already crashes after less than an hour, something is really wrong.

Possible actions in about this order:

1) Do you run 2018SP1 with latest fixes? If not, I definitely would recommend installing that, both for the LabVIEW IDE and the Realtime extension. For the FPGA Toolkit there is no SP1 release.

2) Try to completely wipe the cRIO controller and reinstall the default CompactRIO base image from NI Max. Don't customize this for the time being!

3) If this doesn't help do you have access to a newer LabVIEW release such as 2020 or newer? If so try with that.

4) If all of these doesn't help I would strongly suspect a hardware problem that would require a lengthy and rather painful RMA procedure.

That all said, I'm pretty conservative in updating user interfaces at high update rates. The basic principle is, that the human eye can't really see more than 25 images per second, and if you want something to be intelligible to a human it needs to be even slower. So my loops hardly if ever update user interface elements at every loop, even on Windows, but even much more consistently on an embedded target. It simply makes no sense to tax the OS with things that a user can't possibly take in anyhow. It's part of defensive programming I suppose.

[AnalogKid2DigitalMan]

Status LED's show anything abnormal?

Did this problem just start to show up in the setup?

Abnormally warm ambient temperature area or cRIO running hot?

-AK2DM

[bcl511]

Rolfk,

Many thanks for replying, and apologies that I did not reply sooner (it seems I did not have my notifications set correctly).  I have been working with an NI technician on this issue.  Your suggestion to try a different labview version seems to have been the most on-point.  I was running labview 2023, using the RT-Linux system image Q1 (I attached my code in LV2018 only to be sure most people could read it).  The technician confirms that he sees the same behavior on his controller when running the same labview/RT-linux versions, but does not see this behavior in labview/RT-linux 2021.  I have switched to the RT-Linux system image 2023 Q4 and this appears to be better, but I cannot yet say if the problem is completely solved in this version.  

I agree with you about the fast updating - my actual application only updates the front-panel elements once a second.  The code snippet that I pasted (which updated 100x a second) was merely the most efficient way that I found to provoke this freeze-up on my controller.  I made it for debugging purposes.  More realistic applications required between 2 hours and 1 day to cause the freeze-up, but this one would freeze my controller in 10 minutes, allowing for faster iteration.  

I will post again when I have a more firm resolution to this issue.

Best regards

[bcl511]

Hello Everyone,

I received confirmation from NI technical support that this behavior is only present in the cRIO 2023 Q1 - Q3 drivers.  It has been fixed in the Q4 release, and it is also not present in earlier versions (multiple versions of LV 2021 were tested in detail).  Now I can move on to actually deploying my application!

Cheers

[bcl511 · Accepted solution]

Coming back to this thread after a few months for an update, in case anyone else is facing the same problem.  It turns out that while the LV 2023 Q4 drivers were much better, the problem is not completely solved.  When running a vi with a simple front panel (10-15 fields, updating once per second), I had no problem.  When I added 3 graphs to the front panel, the crashing behavior came back, this time happening anywhere between once a day to once a week.  The graphs were a hard requirement for my project so I went back to ni customer service and they were again able to replicate the issue on their side, on 3 different cRIO models.  They were really quite helpful, even if they ultimately weren't able to fix the problem.  There is apparently a bug either in the ni-linux system image or in one of the software modules (they haven't been able to find the root cause yet).  The key thing is that it only happens when using embedded UI and then only when updating more than some critical number of things on the front panel. 

I don't have my cRIO anymore as I have traded it in and switched to cDAQ.  The R&D folks at NI are still working on a fix as far as I know but I had to move forward with my project.  I replaced the cRIO with a PC and an old cDAQ chassis during the 5 months I was debugging this issue and the PC/cDAQ combination didn't crash once.  I didn't need the FPGA in the cRIO anyways, I had only bought it for the 'all-in-one' possibility of the embedded UI and on the (apparently mistaken) assumption that embedded linux would be less likely to crash than a windows PC.

Cheers

[Be-lock-eh]

I seem to be having a similar issue with an Embedded UI application. I only have a single graph. I'm using 2021 SP1 and its drivers. I tried 2024 and was unable to run the app from dev mode (although I don't remember if it was Q3 or Q4 drivers) and working with NI it was a known issue you couldn't run from dev mode and debug your app so I rolled it back to 2021. 

My app will freeze (in my case it would be better if it crashed and killed the outputs because keeping things on can actually melt product) randomly. Seems like even with the app at "idle" it can freeze after a few days and I've seen it freeze around 30 mins into a run.

Ugh my customer picked this cRIO to have greater stability and we've had more issues getting this up and running than I've ever had with Windows/cDAQ combos.