memory full when running read file subVI

[mugger]

I was using the spreadsheet read VI to read data, then I was stopped by the prompt saying the "Memory is full" when running the read file VI. I read the 1st spreasheet data file, which is 60MB, then when I tried to read 2nd file of 70MB, the Labview stopped me. So I have to exit labview and read 2nd file directly, then it was OK. It seems that after I read first 60MB, even though I stop the execution of the current VI, the memory is somehow not cleared, then then 70MB file reading kills the program. I was wondering is there a way to clear the memory everytime when the stop execution is clicked? or is there a way to allocate more memory to labview? I have 4GB memory in total, I really want to allocate more to labview.

I enclosed the prompt pics.

[RavensFan]

Post your VI so we can see what you are doing.  You may be making unnecessary copies of the data within your VI, or doing something else with arrays or references that are eating up your memory.

[mugger]

The issue seems to be simple, the read from spreadsheet.vi cannot read a data file with size larger than 80MB on my computer. I'm running the Windows 7 on VMware Fusion 3.0 virtual machine, the total RAM is 2GB. I enclosed the snapshot of the error prompt, VI and data file in zip. When uncompressed, the size is about 100MB, which will fail the "read from spreadsheet.vi"

My question is that can I find a way to allocate more memory to labview when running a VI?

[johnsold]

You may need to use something other than Read from Spreadsheet File.vi. If you open the diagram and look inside, you will find that it reads the file as a string, then converts it to an array, transposes if requested, and indexes out the first row. Using the Show Buffer Allocations tool shows as many as 6 buffers created for string and arrays. This probably explains why you are having problems.

What is the expected size of the array resulting from these files?  Your data file did not make it.  Possibly the Forum filters rejected it due to size.

You may need to read the file in chunks and construct the final array from the pieces.

Lynn

[mugger]

you are correct, the data file (spreadsheet format) is 100MB in size, so the server does not allow it to be uploaded. I made 100MB 2D array data into a single file, now when I wanna reopen it, I got the problem. But I don't wanna break the data file into two files of 50MB each because it makes the whole thing more complex. Is there a solution?

[johnsold]

1. Can you read the file as a string without memory problems?  Do not wire the string to an indicator.  Just read the file as a string.

2. How many numeric elements does your array have?  Can you initialize an array of that size in the same program as the file read, again without displaying the array in an indicator?

Lynn

[Timmar]

This sounds like a classic .csv problem.

my advice would be to wite a vi that scans from file 1 row (or column) at a time and process the data in small chunks.

You could store the data in Labview's .TDMS format which will make it a lot easier to work with.

In fact in future I would recommend streaming the dat direct to one in the first place.

[mugger]

In one of my recent test, the spreadsheet file has the array of 1162619x6. What really confuses me is I can use the attached code to read the file once, then I move to another file of similar size, then system give me memory full error. It seems that the memory has not been released after first loading, then the 2nd loading crash the system with the memory full error. 

[mugger]

1. Can the ReadfromSpreadsheet.vi read data in partial?  

2. The VI can read a file with array of 1162619x6. But it cannot read more files of similar size when letting labview to read one by one. Sometimes system crash with memory full error at 2nd file, sometimes 3rd file. It seems there is a memory release failture. 

[smercurio_fc]

---

@mugger wrote:

1. Can the ReadfromSpreadsheet.vi read data in partial?  

---

Did you look at the Help for that VI? That VI has a "number of rows" input. Did you try that? It also has a start of read offset and a mark after read that tells you where the file marker was when it stopped reading.

---

2. The VI can read a file with array of 1162619x6. But it cannot read more files of similar size when letting labview to read one by one. Sometimes system crash with memory full error at 2nd file, sometimes 3rd file. It seems there is a memory release failture. 

---

This is a very common question. The issue is always that you must have an adeqate amount of contigous memory available, since arrays must be contiguous. If you run many apps and close them frequently your memory can get chopped up. You may also wish to read this article: http://zone.ni.com/devzone/cda/tut/p/id/3625