Array subset uses twice as much memory

[Ben]

Don't have time to get too deep in the investigation.

1) LV does not give up memory quickly I think ti was called lazy or something like that. By allocating and not giving up the kernal swaps to allocate virtual memory would be a performance killer. THere is a "de-allocate" function on one of the palette that will return what the VI used when it COMPLETES.

2) Un-do buffers have to stor the old data. Get rid of control indiators.

3) An array of queues is a nice trick for buffering data. THe data will stay in the memeory resurved for the queue entry until you mess with it.

4) LV running under Windows uses Virtual Memory. But virtual memory is just a magic trick the OS performs to present an memory environment to the applications that "appears" to be larger than the actual total of physical memory. The OS uses hardware to map the address memory locations to the actual physical location inside physical memory. most computer I have touched can only acces data taht is in A) Register in the CPU, B) Data from Cache , and C) physical  memory.

Virual memory is after all "virtual" it does not really exist.

Ben

[Ray.R]

---

@Ben wrote:

2) Un-do buffers have to stor the old data. Get rid of control indiators.

---

That would consume a lot of memory especially if you re-write an entire array.

Undo can't be "turned off", but you could set it to 1.  However, if the last transaction is to resize an entire array, you're back to square one..

(In LV8.6), this is set under Tools>Options>Environment and set the "Maximum undo steps per VI".  Setting it to 1 may release the contents of the undo buffer which would be the contents of the array, when modifying it.  However, I do not know if LV allocates a memory pool, which means although it ignores the past contents, the pool of memory (in blocks of given size) is still allocated until it needs to use it again.  If that is the case, the only time you would see the memory being fully deallocated is when the VI is released from memory.

Maybe someone at NI can de-mystify that one...

[MarkMoss]

---

@Timmar wrote:

On another note, Why doesn't Labview access virtual memory to store this data instead of using RAM and running out?

---

As other's have pointed out LabVIEW does use virtual memory.  However, an array must occupy a contiguous region of the virtual memory address space.  On 32-bit windows, the virtual memory address space is only 2 GB (actually 4GB, but 2GB of that are reserved for system drivers and other shared components).  144 MB is over 7% of the virtual memory space.  It's easy to see how there might not be that large of a block available in the address space, especially after the program has been running for a while and allocating/freeing large data structures.

I ran into this issue trying to use a 700 MB+ array as a circular buffer for several minutes worth of data from 160 channels sampled at 500 Hz, even after being extremely careful to avoid operations that would create copies.  The solution for we involved two changes:

1. Storing the arrays in a queue to avoid creating a copy when unbundling from the class's private data structure.  Queuing/dequeing does not create a copy of the data.

2. Breaking the array up into several smaller arrays.  For example, channels 0-15 might be store to one array, 16-31 in another, etc.

Mark Moss

Electrical Validation Engr.

GHSP.

[Timmar]

Mark, Ben

Thanks for the info on "virtual" (Hard drive swap) memory I had not considered the 32 bit limitations on address space.  I had incorrectly assumed that the limit was the 2G Ram in my machine.

[Timmar]

I have updated the test vi and added an in-place structure in one of the diagram disable cases.

It works!

It also works for re-shape array, without adding an in-place element.

Thanks to all for the information on how labview manages memory,  Quite frankly it scares me a lot, needing in-depth knowlege of how Labview manages memory and that not all VI's are equal when it comes to memory allocation.

I still don't agree that it is appropriate to pre-allocate double the required size because get array subset is in the block diagram.

I stand by my statement that I believe that it is is a labview bug.

I think that the compiler should be clever enough to see that it's source and destination are the same shift register and automaticaly apply in-place logic. I have seen this work in other elements such as unbundle, replace array subset and reshape array.

[altenbach]

As I said, some of it has to do with extra debugging code. Run your test, the disable debugging, the run the test again. 😄

[Timmar]

Altenbach,

Same Result, Peaks are as follows:

1.Debug

2.No Debug

3.In Place

[Ben]

---

@Timmar wrote:

...

I stand by my statement that I believe that it is is a labview bug.

...

---

You are in a place were I was years ago but I am now of a different mind, let me share.

A "bug" is a condition were software fails to perform as specified. Since the specifications of the "In-placement" alogorithm are not published, we can not declare it a bug.

Only a small group of people in the Ivory Tower are capable of making that decision.

Now if you ind a white paper article that states something different thatn what we find in our code, then we can start suspecting bug.

Just sharing my view in LV, memory and bugs.

Ben

[DFGray]

As explained in the tutorial I linked earlier, the combination of Windows and LabVIEW puts a lot of restrictions on memory use.  On a standard 32-bit system, only 2GBytes is available for program use.  Of that, the top half gigabyte is used by system DLLs.  So we are down to 1.5GBytes.  LabVIEW requires a contiguous data space to allocate an array.  In LabVIEW 8.0 and above, the largest contiguous data space is about 800MBytes in size due to memory fragmentation (in earlier versions, it was about 1.1GBytes), assuming nothing else in memory.  I have confirmed these numbers experimentally on Windows XP.  64-bit LabVIEW on a 64-bit OS still has the contiguous array issue and still limits you to 2G points per array (max I32), but otherwise will use as much memory as you have.  32-bit LabVIEW on a 64-bit machine should give you a lot more space, but I have not looked at it, so cannot tell.

[Ray.R]

---

@altenbach wrote:

As I said, some of it has to do with extra debugging code. Run your test, the disable debugging, the run the test again. 😄

---

I'm surprised with the results..  I was expecting the memory useage to decrease with debug disabled. 

Hey... I learned something new... 🙂

The learning never ends...