Timed loop slows down on built exe only

[blahblahblahxxjhxjh]

Hi,

I'm having a very strange issue with timed loops in built exes. I originally noticed this problem in a much larger development program that I have been using successfully for the past 5 years with no issues. I made an update to it and when I rebuilt the exe I suddenly couldn't get it to run properly. I tracked the issue down to some timed loops in my larger program, so in order to diagnose the issue I created a small program (attached) that has nothing but two parallel timed loops in it. Bear with me while I try to explain this...

• If I run the attached code in the development environment it works perfectly, loop times are as expected
• If I build the code into an exe and run that on the same machine the program runs normally for around 30 seconds and then the loop times start becoming very sporadic, with loops taking up to 3 or 4 seconds to complete instead of the 100 ms they are timed for
• If I shut down the exe and open it again I repeatedly get the same behaviour, sporadic times after about 30 seconds of running
• I tried it on another completely separate machine by building an installer with the run-time included, same behaviour

This is where it gets really weird:

• I built the exe 10 days ago and it wouldn't run properly on either of the two machines I tried it on
• I haven't looked at it for the last 10 days as I've been away, having come back the exe now runs as expected on both machines, no issues with the timing of the loops - it has miraculously fixed itself
• If I use Windows to copy the now working exe to any other location on the same computer, the copied version of EXACTLY the same program now shows the original fault with loop timings not working properly
• The original (10 day old) version of the program keeps working properly, no issue with loop timing
• I have tried this multiple times, copying to many different locations on the PC (C drive, desktop, program files etc) and the result is always the same. The newly created copy does not run properly, the 10 day old copy continues to work properly
• I see this exact same behaviour on both the PC I did the original development on and the second PC that I used to trial the exe - Both computers have 10 day old versions that work fine while any copy of that version doesn't work properly

How is that possible? 

Very grateful for any ideas!

Thanks

[Bob_Schor]

I agree that you have found Something Really Strange!  But I have a few other Remarks and Observations.

• Back when I was starting with LabVIEW (LabVIEW 7.0) and was doing a LabVIEW Real-Time Project, I remember learning that Timed Loops were meant for LabVIEW-RT code running on a Real-Time OS.  You could use them on a PC, but many of the "features" either weren't supported in Windows or didn't make sense in Windows, so I reserved this feature for my RT Target's code.
• Nevertheless, I ran your code (which I see you developed in LabVIEW 2018) with LabVIEW 2018 on a Windows 10 (64-bit, Version 18.09) laptop.  I made a minor change by setting the Timed Loops to run for 5000 times (instead of running forever) so I'd have a known base for my tests.
• I initially only noticed the top loop (the bottom loop didn't appear on my small laptop screen until I scrolled), so I'm repeating the tests having modified the loops identically.
• The two loops ran with a Max Loop Time of 0.10534, which is 5% "slow" on one loop.  The max Code Time was on the order of 0.01, clearly not a problem.
• Replacing the Timed Loops by While Loops and using Wait for Next Ms functions with 100 wired to them (and one preceding the loops to get them "in synch") yields Max Loop Times of 0.102114 and 0.102563, with max Code Times again around 0.01.
• Replacing the Timed Loops by While Loops and using Wait (ms) functions with 100 wired to them yielded Max Loop Times of 0.102615 and 0.102702, with max Code Times around 0.01.
• It appears that you can get equivalent functionality (and simpler code) by replacing the Timed Loop with a While + a Wait (ms) function inside (since you don't need to worry about being notified if you "miss" a tick, and most of the other functionality of the Timed Loop is meaningless in a Windows System).

This is testing things in Development Mode.  When running in Execution mode, I wonder if Windows "plays by its own rules" and might make its own decisions about allowing itself to "preempt" your system while it did things for itself.  I confess I have not (yet) tried your executable, but will, shortly, and will report back if I find that it seems to "hang" on my system, as well.

One personal experience -- I recently was scheduled to give a talk (at NIWeek, as it happened).  When I booted my laptop, the screen stayed black, and I figured the Video card had chosen a bad time to die.  I had the slides on a USB Key, so I used another PC for the slides.  When it came time for the Demo, however, I was stuck, until an audience member said "Hey, your Laptop just came alive!".  Reattached laptop, finished talk, and discovered that Windows had decided to push the latest Upgrade which took about 40 minutes ...

Be wary of Windows for time-critical work!

Bob Schor

• when a light-bulb went off in my head.  

[CFER_STS]

At a guess, I would suggest that your loops are being pre-empted by the Windows scheduler (since Windows is not a deterministic OS), and specifically to accommodate the LabVIEW memory manager, since you are dynamically building arrays within your FOR loops.  This is poor practice.  Instead, you should be initializing those arrays to the desired size before entering the FOR loop, and then wiring the iteration counter to Replace Array Element instead of using Build Array on every iteration.  Preallocating the memory block used for the array prevents fragmentation and reduces the need for the memory manager to consume system resources. As you have currently implemented, every time the array grows beyond the available contiguous memory block, the memory manager needs to find a new place to put it.

On a deterministic real-time system, it is recommended to only use one timed loop per available CPU core in order to preserve code execution priorities.  On a preemptive operating system, there is no advantage to using timed loop structures over while loops with wait functions.

[blahblahblahxxjhxjh]

Thanks for the replies both.

I am definitely only having this issue with timed loops, I also tried switching to while loops with a wait function and that solved the issue. I had thought that timed loops could be used on any OS though, I can't see anything in the documentation that recommends against using timed loops in Windows.

This also only happens when I have parallel timed loops, with a single loop I don't see any issue. I agree it seems like something the operating system is doing with resource allocation somehow. I have no idea why leaving the built executable for a prolonged period should suddenly change that though, although I suppose that is a side issue...

I realise the code in my loops is not good practice, it's really just some nonsense code to waste a small amount of time. I see the same effect no matter what the code within the loop is.

[Bob_Schor]

---

@blahblahblahxxjhxjh wrote:

I am definitely only having this issue with timed loops, I also tried switching to while loops with a wait function and that solved the issue. I had thought that timed loops could be used on any OS though, I can't see anything in the documentation that recommends against using timed loops in Windows.

---

I have also looked for such documentation -- it doesn't appear to be specifically mentioned, but many LabVIEW developers who work with LabVIEW RT (including my original LabVIEW Mentor) mention that the additional code in the Timed Loop was largely irrelevant in a non-real-time OS like Windows and strongly recommended not using it.  I'm just passing on "received wisdom" (which also makes "logical sense").

Bob Schor

[blahblahblahxxjhxjh]

So just to update for anyone having a similar issue in the future, I got to the bottom of this with the help of a service request with the NI team. It seems to be an interaction problem between multiple timed loops and hyper-threading on my machines, if I disable hyper-threading in the bios then the issue goes away. 

[pallen]

---

@Bob_Schor wrote:

...Reattached laptop, finished talk, and discovered that Windows had decided to push the latest Upgrade which took about 40 minutes ...

Talk about bad timing!  

I've heard that one way to bring some control back to updates is to tell Windows 10 that it has a Metered Connection.  This is supposed to force Windows to ask you when to update.  But I have read a few posts claiming that even this does not always work.  

[CFER_STS]

That makes sense.  Timed loops inherently inline all code within them, so that the contents of the structure will execute within a single thread.  As such, you can't do things like explicitly code parallelism into multiple FOR loops contained within a timed loop, for example.  A single timed loop only runs on a single CPU core.  The hyperthreading feature of the system processor attempts to create multiple logical cores on any single physical core, in order to improve parallelization by executing code in parallel on the logical cores.  The problem is that these logical cores do not duplicate all execution resources the same way that multiple physical processors do, and the OS thread scheduler is not necessarily aware of the hyperthreading behaviour of the system processor, so you can encounter processor stalls.  This is particularly true when you have one scheduler (LabVIEW) which has optimized execution of timed loop code for a single core, or multiple timed loops to be executed on one core per loop, and then the Windows scheduler arbitrarily attempting to distribute the executing code across the logical processors.  I wonder if the processor assignment of the two timed loops was being made to two separate logical cores which happened to coexist on the same physical core?