Intermittent memory hangup using Functional Global

[vituning]

I've been chasing this one for over a month, now I turn to you all for help.

I have a Functional (LV2-style) global that stores all the data that my aplication uses in its shift register-driven memory. I have a 'GetData' function that simply reads a large waveform (1MB) from the shift register and makes a copy which is written to an indicator on the front panel (for use by the calling VI). Every once in a while (1 in 10,000 times) instead of taking the usual 5-10ms to execute, this function takes over 5 seconds to complete. This vi is not reentrant and is not being called simultaneously.

Here's a picture of the block diagram at the 'GetData' function. The 'Read' function is called with after this, which just puts a copy of all the data to the front panel.

My main question is, is there any known issue with indexing medium-sized data from memory in windows that could cause this hangup? Has anyone had similar experiences?

Thanks.

[altenbach]

Can you try once more to attach the picture? THanks!

[johnsold]

You did not succeed in attaching your VI. If you "Preview Post" after attaching the file, the attachment may get lost.

I am guessing that your problem is memory allocation. LabVIEW automatically allocates memory for arrays and other large data structures as needed and usually does a very good job of it. However, if a large data space is needed where perviously a small one was already allocated, sometimes this can take a long time. Arrays and strings which grow in a manner which the compiler cannot predict are usually the sources of this type of problem.

How big is your data set when the problem occurs?

Lynn

[vituning]

Sorry, must have lost the attachment when I previewed. Here it is again.

[vituning]

---

@johnsold wrote:

I am guessing that your problem is memory allocation. LabVIEW automatically allocates memory for arrays and other large data structures as needed and usually does a very good job of it. However, if a large data space is needed where perviously a small one was already allocated, sometimes this can take a long time. Arrays and strings which grow in a manner which the compiler cannot predict are usually the sources of this type of problem.

How big is your data set when the problem occurs?

Lynn

---

The data set is big - say around 1MB. I would pre-allocate the memory for this array if I knew the size ahead of time. However, I need to read the waveform to get the total allocation size. How can I be sure that LV won't hang when allocating the memory in the first place?

Thanks.

[unclebump]

I see some build array functions in there, I think this will cause you to reallocate memory on the fly. Preallocate more memory than you need ahead of time, then when you know the size you need, use the reshape array function. Do a search of the discussion forums for 'reshape array altenbach'. I can't find his exact message that I wanted to show you.

[Ben]

Since this code work most of the time, we should see if there are other things with the OS that could be causing it to forget about your app and make it go off and waste its time. Here are some ideas.

1) Set Windows to optimize background services.
Windows by default will attempt to optimize its scheduling to make foreground processes perform well. For single threaded applications this is fine but for LabVIEW, some of its background threads can suffer.

A) Start >>> Control Panel
B) Open "System" and choose the "Advanced" tab
C) In the "Performance" section click "Settings"
D) On the "Advanced" tab in the "Processor Scheduling" section select "Background Services"
E) Save or apply everything.

2) Stop Indexing services.
Windows will by default do file indexing. File indexing is functionality that tells windows to maintain in memory a cache of files on a disk or in a folder. This makes files show up in the Explorer faster and also makes searching for files faster.

A) Start >>> Accessories >>> Windows Explorer
B) Right-click C: drive and select "Properties"
C) On the "General" tab un-check "Allow Indexing Service to Index This Drive for Fast File Searching"
D) Save and apply everything

3) Set LabVIEW priority in Task manager. Windows will by default treat all processes that are ready to run as peers and will share the CPU evenly.

A) Open the Task Manager and select the "Processes" tab.
B) Locate "LabVIEW.exe" in the list of processes.
C) Right -click "LabVIEW.exe" and choose Set Priority >>> High
Note: Setting LV to run at higher than "High" will put it on equal footing with interrupt service routines etc. This could result in the machine "locking up" because LV is using all of the CPU and there is no opportunity to respond to interrupts from your mouse moving.

4) Shutdown Virus Checking. Virus checking gets its hooks into everything!

5) Disconnect Ethernet cable. Ethernet traffic requires intervention by the OS. No cable, no traffic, no distractions.

With all of that in-place watch your Task Manager while the "problem" is occuring. If the CPU drops off, and you see the LV process is page Faulting, then memory allocations is definately a suspect.

If the CPU spikes, this may be due to the work required to graph the data. If disconnecting the data from the graph/chart eleminates this issue, concider using "defer FP Updates" before updating the indicator.

Does any of the above help?

Ben

[vituning]

Ben -

Thank you for the very informative reply. I tried your suggestions, and unfortunately, it did not solve my problem. The idea that windows is doing something in the background that interrupts the memory allocation process is an interesting, however. I know that disabling hyperthreading is another good trick, but the two machines that I am testing on do not have hyperthreading processors.

Do you know of any other settings like these?

thanks,

scott

[vituning]

Both for loops in this case are run or only one iteration. Even though I am explicitly building an array in the first loop, it is only one element large and contains a single I32. In the second loop (where the arge data array is being indexed), I am using the indexing of the loop itself to create the output array. Doesn't this do a better job of allocating memory than the usual build array function? I was always under the impression that by eabling indexing at the tunnel of a for loop would handle the memory allocation for arrays much more efficiently.

That said, I am trying a method that will avoid the for loop indexing/array building altogether. I'll keep you posted on the progress.

thanks again for your help.

[Ben]

Ok Scott,

That was the easy stuff.

Warning: We may not finish this up without some sweat and tears.

Could you please post a zip of of your LV2 and let us know what version of LV you are working with?

You wrote;

"

 I was always under the impression that by eabling indexing at the tunnel of a for loop would handle the memory allocation for arrays much more efficiently.

"

I think it was LV 6i or there-abouts were LV learned how to optimize building arrays on the edge of For loops. It does this by looking at how many times the loop will iterate and pre-ALLOCATES a buffer for the data.

In your code, the "build array" has to tickle the memory manager every interation. If you combine your logic in your two For loops you can eliminate the build array completely. The trick will be handling the un-defined waveform names. That can be handled as well but the solution tha comes to mind is convoluted. It involves buffers that are created when the LV2 is init and these are re-used when do the getdata work.

But before I'd get to serious about that, I'd first think about moving all this "figure out the index" work to another state. Here is what I have in mind.

The consumers of the data in the LV2 would pass a "Task ID" instead of the list of channels names. THe "Task ID" can then be used to index out a previously defined set of indexes. If the consumer needed to change its list of channels it want to recieve, it would pass a new list along with its "task ID" (index). The Task ID is then used to update the list of channels using the logic that is now being repeated everytime this client asks for an update.

Another reason I'd like to see your code running in a demo that lets us test what is happnening.

I am "showing my hand" by admitting that the exact circumstances when arrays of waveform datat types wqill require buffer allocations. Sharing your code in a demo form will help us poke around. I'd like to see where buffers are being allocated. Of some concern is the "Use default" on the tunnels coming out of some of your cases. That may be forcing buffer copies. Not sure without gettin gmy hands on this code.

I'll watch for your code.

Ben

Message Edited by Ben on 04-15-2006 10:18 AM