Array subset uses twice as much memory

[Timmar]

I have been chasing a problem with running out of memory when manipulating large ammounts of acquired data.

My attached vi exhibits some very strange behavior.

My original code used "Array Subset" to clear the data in a shift register.

My revised code just uses an empty constant.

Question #1. Why should it take at least double (perhaps triple) the ammount of memory to store data using the Array subset method.

and.. Why is the memory allocated before the "Array subset" is used.

Question #2. Why am I seeing significant residual memory usage that is not being re-allocated

Labview Version 9.0.1 (32-bit)

[Ray.R]

I'm not sure I understand what you are trying to do.. 

You create a sinusoidal waveform which consists of 10000 samples, so ultimately, you have an array of 10000 elements to create the sine wave.

Then you initialize an array to have 1000 elements of the waveform that you created.. So that means 1000 duplicates of the original waveform...  Which consumes 1000 times the amount of memory from your original waveform.

I have no clue at what you are doing with the loop which has a Wait until next ms multiple.  A delay would do the trick... but why??  Did you do that so that you can loop at your memory useage?

Then you have an array of the waveforms with a waveform below it.  Now I am really curious.  Do you really want or need an array of waveforms? What are you trying to do?

[Timmar]

There are 1100 vi's in the code where it was discovered.

This is a simplified vi to illustrate what looks like a labview bug.

Step #1 Allocate a whole bunch of memory.

Step #2 Release the memory using 2 different methods.

For some reason, the ammount of memory used is dependent on the method used to release it.

[GregFreeman]

That's interesting...the memory usage is different when you first run the program, before you ever get to the replace array subset or empty array case. My guess is for the array subset case, the LabVIEW compiler is automatically allocating a buffer for the data copy that may be made (assuming you actually grabbed a subset of data, which in your case you do not). Then after the case executes it sees you aren't getting a subset and frees up the allocated memory. As for the empty array taking up less memory, I'd just assume the compiler is smart enough to know it doesn't have to allocate memory. This is just my guess though...and if this is the problem then the question is why isn't the compiler smart enough to know that you have that 0 constant wired so it should treat it like an empty array.

I'm sure others will add their input. On my PC though, it only shows and increase in about 2 times the memory between the two cases, not 3. That's using LabVIEW 10 on windows xp.

A question I have for you is...why wasn't an empty array used in the first place haha?

[Timmar]

I was hoping you wouldn't ask that final question.

Let's just say that our coding practices have improved since it was written.

That said, it would not be unreasonable to empty your functional global variable, (Shift register) and return a subset of waveforms to the calling routine.

[GregFreeman]

---

@Timmar wrote:

I was hoping you wouldn't ask that final question.

Let's just say that our coding practices have improved since it was written.

---

Yes, my bet is teh compiler just isn't smart enough to see that zero constant wired, and it assumes worst case (a subset of the whole array) so it allocates that memory. Then when the case executes, it frees what's unused (in this case...all of it).

I'd be interested to know the answer, or see if someone from NI can add their input.

[altenbach]

There are a lot of lose ends here, for example since there are no indicators on the big array, we don't really know how much gets eliminated by the dead code elimination algorithm. I don't really understand the idea of using array subset to clear an array. Typically you would just delete the wire and set the output tunnel to "use default if unwired". The default value is an empty array of the correct type, no need to clutter the diagram with empty diagram constants.

Some quick test:

Get rid of the diagram disable structure, it might falsify some optimization. Change the wiring manually for the various tests.

Disable debugging. Debugging forces the data to be in memory for probing. Now there won't be a memory spike at all, no matter what the algorithm.

Add an indicator to the array inside the loop after the case structure. Now the code cannot be eliminated by the compiler. Try with array subset (as you did), with a default output tunnel, or using reshape array with a size of zero. compare the memory spike.

[Ray.R]

I agree with Altenbach, which is why I didn't understand that this was setup as a test to verify why an array subset used more memory than expected.  The test itself has a lot of loose ends and did not make sense from a program point of view (programming logic).

I would try what Altenbach suggested before calling this one a LabVIEW bug. 

[Timmar]

OK,

The origins of this discovery was unexpected memory "behavior" in an application containing in excess of 1250 vi's and is attached to NI-DAq's, CAN and GPIB based hardware.

We were storing 6 minutes of 50 channels sampled at 5khz and recived an out of memory error.

To stop it failing, we reduced the sample rate to 1khz and gathered some metrics

We did the calculatons on how much memory it should take to store this many samples (360x50x1000x8) = 144 meg

The memory profiler was showing that on first run the memory would rise to an additional 432 meg at the end of sampling and at the point where the shift register was "zeroed" it would drop back to only 144 meg. On the next run it would rise back to 432Meg again.

For intellectual property and simplicity reasons, I wrote a basic VI to demonstrate the problem we found.  I was not expecting it to be critqued for programming methods or whether or not it was a valid application.

The Fundamental problem is that Labview should not allocate 3 times the memory required and not give back a third of it on completion.  Debug or otherwise.

The condensed .vi was first written without the diagram disable, which was added later as a tool to aid peers and NI LV Guru's a way to easily see the problem.

On another note, Why doesn't Labview access virtual memory to store this data instead of using RAM and running out?

We are planning a code extetion to sream the data to TDMS to compensate for this "characteristic".

[DFGray]

Many of your questions are answered in this tutorial (which, unfortunately, is somewhat dated).  Also check out the In Place Element structure and the Data Value reference.

You may also want to look at the Request Deallocation primitive.  By default, LabVIEW keeps the memory allocated by a subVI around under the assumption that it will probably be used again if it was used once.  You can get LabVIEW to release memory by using Request Deallocation.  Be careful, however, as this can lead to major performance hits (which is why the default behavior exists).

Another technique to clear memory is to use Reshape Array to clear an array (set to zero length).  This lets LabVIEW know that you will no longer be using the old array size.  It is also an inplace operation (as much as possible), so is fairly fast.

If you have further questions, let us know.