Allocating enough memory to open a large data file

[BrockB]

Hello,

I am currently running an experiment which requires me to collect data at a very high rate, creating enormous data files. At first, I ran into the problem of LabVIEW not having enough memory to save all of the data to the hard-disk at once (it seeme to have no problem with a huge array during data collection, only when it went to save it to a file), so I programmed LabVIEW to only save 100000 samples at a time. This seemed to work fine, and I was able to collect a 550MB data file in this way. However, now that I would like to analyze that data, LabVIEW can't read the data from the file into an array, giving me the same insufficient memory errors as before. My system has 3GB of memory, so in theory LabVIEW should be able to get enough to open it, however Windows doesn't seem to want to allocate what it needs. Is there any way that I can override windows and allocate enough memory to LabVIEW for it to be able to open this file and work on the data?

[TCPlomp]

In short:

No.

LabVIEW is a high level programming language and protects you from memory allocation.

For an array all the data needs to be in a sequencial memoryspace. Is there a reason you can't analyse in parts? For plotting 550 Megs of data is more than can fit on a screen, so writing your own memory data manager might be what you need.

Ton

[BrockB]

I'm not worried about plotting any data right now. If I want to do that later, I can do it in gnuplot. Right now I'm just trying to process the raw data. Even if I did do it in parts, I'd still have to open the file in the first place to break it into parts, and that seems to be where labview is getting stuck. There has to be a way, either through windows or through labview that I can allocate enough memory to get the job done.

[TCPlomp]

---

BrockB wrote:

.....Right now I'm just trying to process the raw data. Even if I did do it in parts, I'd still have to open the file in the first place to break it into parts, and that seems to be where labview is getting stuck. ..... 

---

Hmm, strange, could you show your code?

Ton

[altenbach]

---

BrockB wrote:

Even if I did do it in parts, I'd still have to open the file in the first place to break it into parts, and that seems to be where labview is getting stuck.

---

You can use lowlevel file IO functions to read a defined amount of data from a defined file position.

I assume that your files are in binary format and not e.g. ASCII formatted.

Use "set file position" to the desired location, the wire a "count" to the binary read function to only read parts of the file.

[BrockB]

I'm just home for lunch right now, so I can't show you my code until I get back, but all I'm doing is using the "read from spreadsheet" vi to read the data from my file. The data is all in a 2xN tab-delimited array, and I'm using the 1d array output of the "read from spreadsheet" vi, as I only need the first of the two columns. What I meant to say earlier was that I would still have to read all of the data from the file first if I wanted to break it up into pieces to use later. Labview seems to get stuck on reading the data from the file, not actually on opening it. It also seems like breaking the data up would be a much bigger hassle than just allowing labview to use more of the 3GB that I have available (most of which is sitting unused anyway).

[altenbach]

---

BrockB wrote:
The data is all in a 2xN tab-delimited array, and I'm using the 1d array output of the "read from spreadsheet" vi, as I only need the first of the two columns. What I meant to say earlier was that I would still have to read all of the data from the file first if I wanted to break it up into pieces to use later. Labview seems to get stuck on reading the data from the file, not actually on opening it. It also seems like breaking the data up would be a much bigger hassle than just allowing labview to use more of the 3GB that I have available (most of which is sitting unused anyway).

---

First of all, any ASCII formatted file is an extreme waste of space and comparatively expensive to write and read. It is only recommeded for small files intended to be later read by humans. 🙂

"Read from Spreadsheet file" must first internally read the entire file as a string, then chop it up looking for tab and linefeed characters, then scan the parts into DBL. If you want the first column (instead of the first row), it then also needs to transpose the entire 2D array, requiering another copy in memory. As you can see, you'll easily have quite a few data copies plugging up your RAM. ("Read/Write from/to spreadsheet file" are plain VIs. You can open their diagram to see what's in there. :))

For datasets this size, you MUST use binary files. Try it and your problems will go away. 😄

Message Edited by altenbach on 11-11-2008 12:06 PM

[altenbach]

Some additional tips.

Since you are acquiring data, most likely it is in a relatively small datatype. For example if you have an 16 bit digitizer, the raw measurments is only I16 (or U16) and the binary file would only require 2 bytes/point, retaining the full information. (You currently waste 2bytes/point just for the decimal point and the delimiter! 🐵

You mentioned a 550MB file, but since this is ASCII formatted, it is impossible to tell how many points you have. Depending how many points there are, you might still want to process it in parts if you run into memory problems. With binary files this is no problem because there is a simple relation between array index and file offset.

[TCPlomp]

One more additional advice:

have a look at TDMS files, they have groups, channels, properties, names, timestamping provided in the datatype so you don't have to reinvent the wheel.

Ton

[BrockB]

Thank you very much for your help. I'm actually using an 18-bit digitizer, so I'll need the full 4 bytes per point.

I've run into a couple of new problems though

It seems that although it had no problem initializing a 2x50M array of doubles before (the problem was in trying to write the array to a file after collection finished), it won't let me do so now, saying it doesn't have enough memory (although it still lets the program run and just uses an empty array instead of a 2x50M one, therefore no data gets saved).

When I try saving 100k data points at a time, it seems to work, but I have trouble retrieving it all at once, as it seems to have saved it as multiple individual 2x100k arrays instead of one long 2xN array. When I try to retrieve the data, I only get the first one. How can I append to the binary file while keeping all of the data in a single array? Is it possible?