cRIO-9048 Startup VI Stalling

[Greg_Reynolds]

I'm running a simple Startup VI on a cRIO-9048, taking some AI data via DAQmx and writing data to a spreadsheet file on the SD card. Occasionally the VI stalls, still running, but no updates to the FP indicators, as observed on the cRIO USB-C driven display. 

 - I've added some cycle-counter indicators to the Producer & Consumer loops and those stop updating as well.   - I've added some error-line indicators to the two loops and no errors are shown there. 

An FGV is used to communicate the data from the Producer to the Consumer.  This populates the Consumer indicators and is sampled every 30s to write to file.  I don't need continuous data.   The Producer loop is timed by the DAQmx Read, with six channels of  AI sampling at 50S/s and reading 25 samples/channel at the DAQmx Read, so this is not a lot of data!  The Consumer is a Timed Loop running at dt = 500ms.

I just noticed that the last FP update shows a time of 05:37:37.430 and the last record of data on the SD card was at 05:37:06... So it's likely that the stall may be occurring at the write to SD!

I'm running LabVIEW 2022 Q3 Patch 1 with Real-Time 22.3.0 (on the PC) and Real-Time 22.3.1 (on the 9048)

[ZYOng]

Try Monitoring CPU and Memory Usage on Real-Time Embedded Targets and see if the cRIO stops responding or crashes at some point due to high CPU usage or memory leak.

[Greg_Reynolds]

Thanks ZYOng, I'll check that. 

I've run several more diagnostic runs to track down the sporadic failures.  I switched to the cRIO memory at /c/my_data.  Still got a stall (VI still running, but the FP no longer updates, and no indication that while-loops are cycling).  Long and short of it... this seems to be related to the write-to-file...

 - DAQmx Read and FP updates are at 0.5s intervals & write-to-file every 30s

 - last write-to-file was at 09:30:46

 - last FP updates occurred at 09:31:16.992 (should be a write-to-file at this time)

 - but, the last data file update on the cRIO ssd (Based on File Explorer) was at 11:59! 

The main VI is set up to create a new data file at midnight.  So the above is very curious; as if the Write Delimited Spreadsheet.vi is not completing the write-to-file, but keeps trying until the file name is changed at midnight.  Very strange!  (These are very small amounts of data, around 600KB over a 24 hour period.)

Greg

[Greg_Reynolds]

Update - several more runs with no clear indication as to the source of the stalling.  By stalling, I mean that the VI is still running as indicated by the Run button on the FP, but there are no updates to FP indicators.  Additionally, there are never any errors indicated.  I'm at my wits end after WAY too much time trying to diagnose this unreliable operation.  Some observations:

1. The VI runs OK, collecting AI data, doing some simple data averaging and logging data to file, creating a new data file at midnight.  As noted never any errors indicated, running sometimes a few hours before stall, but usually more than 24 hours.

2. Sometimes the stall is at a write-to-file, which occur every 30 seconds.  When this occurs, the data file appears to stay open until a) midnight when a new file is generated, or b) when I do a cRIO Reset to stop the VI and access the data. But more commonly the stall is at some other time than write-to-file.

3. Free Physical Memory is 1.26MB and stable.  This seems small given the available memory on the unit.

4. CPU Load is around 8-10%

5. Before the run shown here, I did a Firmware update on the cRIO-9048, going from 8.8.0 (7/14/21) to 22.8.0 (10/2/22).  This made no apparent difference.

Greg

[aascencio]

Are you using property nodes or references for your clusters?

Also do you think we can take a look on your FGV? If possible maybe the code?

I also would like to know what would happen if you try to deploy my application as a source distribution with a startup VI instead: Configuring Startup VIs on LabVIEW Real-Time Targets Without Application Builder - NI

[Greg_Reynolds]

Hi aascencio,

I've tried the following...

 - disk reformat and reinstall software on the cRIO-9048... no change

 - factory reset and reinstall software on the cRIO-9048... no change

The VI stalls with any DAQmx AI application (still running, but no FP updates)

I've crated the Source Distribution according to the link that you sent, Configuring A Startup VI, and I can build and deploy it.  But I don't understand how to configure the Startup VI (as opposed to the Startup Executable, startup.rtexe, that results from the RT Application). 

I see the reference to ni-rt.ini, and I find the following on my target running NI Linux RT 22:

 - \etc\natinst\share\ni-rt.ini

 - \etc\natinst\share\lvrt.conf

It looks like I'm supposed to add into one of these files: 

 - RTTarget.StartupVIs=/home/lvuser/natinst/bin/mystartup.vi

Any insight as to which of these files I should modify... the advisability/advantage of doing this!?

Thanks much,

Greg