What Causes "Loop Hang" Issue With File I/O When Using Multiple Timed Loops on a NI-cRIO 9012?

[William_Griffin]

I have been experiencing an issue with an application. I believe I have isolated the issue in the attached source code "Timed Loop Issue - Source Code.zip". A detailed list of my results can be found in the attached document "Timed Loop Issue - Detailed Results.docx".

General Overview - It appears to be closely related to file I/O and the use of "Timed Loops" on the NI cRIO-9012

1.     If I enable the I/O in both loops, then both loops crash simultaneously

2.     If I only enable the I/O in one loop, then only that loop crashes

3.     If I enable the I/O in both loops and make one of the loops a "While Loop", then it does not crash

4.     If there is nothing written in the file I/O, then it does not crash

5.     If the reference to the file is left open instead of opening and closing it with each call, then it does not crash

6.     If I try to run it on "My Computer", then it does not crash

7.     If the I/O is only enabled in one loop and programmed to wait one second between file accesses, then it still crashes

In my actual application, any given loop is only writing (at most) once per minute, but I could still get my application to crash within a 24 hour period by writing a single error from a single loop once per minute with a low processor utilization. This was the impetus for testing the seventh item above.

It seems like the best solution was to open the reference to the file and leave it open. I believe that this is a good practice in general and I will make this update in the application. This should solve the problem, but it is very hard to verify this due to the issue's sporadic nature. As such, I would like to fully understand what was causing the issue so that I can verify that it has been fixed. Thus far, my attempts at creating a single, consistent theory capable of explaining all of the aspects of this issue have failed.

[iZACHdx]

 Hello,

Looking at your code, I would not be surprised that you are getting the behavior that you describe. File IO operations are inherently non-deterministic and thus should not be placed in a time-critical or high-priority loop. Opening references to files, writing to them, and then closing the reference consumes significantly more resources than opening the file outside of the loop and then writing to it within a loop. This has to do with creating the reference and allocating space from the LabVIEW memory manager - another shared resource. In the code that you have provided, you have two timed loops - a high priority loop at a period of 20ms and a lower priority loop at a period of 100ms. Within these loops, you are accessing the same subVI. Since this subVI is not set to a reentrant execution scheme, it has become a shared resource as well. This means that if you are trying to write to a file with the lower priority loop and the higher priority loop starts its turn for execution, it creates a priority inversion. The way this is handled is that the subVI temporarily inherits the priority of the time-critical loop in order to finish executing and release the resource. All of these things combined would attribute to why you are getting a crash. You could try increasing the loop period to account for the extra jitter that is inherent with File IO and accessing shared resources.

-Zach

Message Edited by iZACHdx on 05-20-2010 02:42 PM

[William_Griffin]

 Thank you for looking this over Zach. I have attempted to address your main points below. Please let me know if you have any additional thoughts or clarifications.

1."File IO operations are inherently non-deterministic and thus should not be placed in a time-critical or high-priority loop. Opening references to files, writing to them, and then closing the reference consumes significantly more resources than opening the file outside of the loop and then writing to it within a loop. This has to do with creating the reference and allocating space from the LabVIEW memory manager - another shared resource."

        This comment is pointing out an issue with the speed of the loop. Please note that the issue is not with the speed of the loop, although I am using the timed loop functionality to both monitor the speed and prioritize the code contained within. The issue that concerns me is that it crashes at all. As I note in number three below, the speed of the loop only appears to affect the amount of time it takes to crash, not whether it crashes at all.

2."In the code that you have provided, you have two timed loops - a high priority loop at a period of 20ms and a lower priority loop at a period of 100ms. Within these loops, you are accessing the same subVI. Since this subVI is not set to a reentrant execution scheme, it has become a shared resource as well. This means that if you are trying to write to a file with the lower priority loop and the higher priority loop starts its turn for execution, it creates a priority inversion. The way this is handled is that the subVI temporarily inherits the priority of the time-critical loop in order to finish executing and release the resource."

        The issue occurs even when I do not access the same subVI in both loops (see General Overview #2).

3."You could try increasing the loop period to account for the extra jitter that is inherent with File IO and accessing shared resources."

        When the application is run at only 1 write/second or 1 write/minute, the problem still occurs (see General Overview #7 and the paragraph that followed it).

[iZACHdx]

Hello,

To clarify:

Yes, it has to do with the speed of the loop, but also how memory is allocated through a shared resource (the LabVIEW Memory Manager). By continually opening, writing, and the closing the file, the file that is having to be opened in each iteration of the loop is a slightly larger file each time. This explains why giving longer loop periods would affect the amount of time it takes to crash. This also goes hand-in-hand with why placing the code in a normal while loop does not crash. In that case, you are not placing the same timing constraints on the code. Have you tried unchecking both discard missed periods and maintain original phase?

-Zach

[StephanieA · Accepted solution]

I just wanted to let everyone know that there has been a corrective action request (CAR#214156) filed for this issue to be fixed in a future release. This appears to be an issue with the real time OS we are using VxWorks. We have additional cRIO chassis that have a Pharlap OS where this does not occur. For now, you can look for this CAR ID in future release notes to see if this issue has been resolved.

Currently there are two workarounds we have identified to resolve this issue:

1. Open the file before the loop and close it after. (recommended)
2. Perform the File I/O in a normal loop and pass data to it from a queue.

Message Edited by StephanieO on 05-24-2010 01:36 PM

[Robert_North]

Hi William,

Can you tell me if this issue (#214156) has been fixed in LabVIEW 2011 RT?

Regards,

Rob

[William_Griffin]

Sorry Rob, I can't help you with that. I searched NI.com for the CAR#, but I haven't found any references to additional work on this. You may want to contact National Instruments directly regarding this.