Large data array to disk

[sjandrew]

Ya, probably didn't write that post in a very literate way.

So now I've tried removing the loop, and I get a "generic I/O error" code 6, which looks like the same problem I was having with the "Write to Spreadsheet Array" - trying to allocate an entire (new) array, before writing the data to disk. I originally put the loop in to avoid this problem, but now that's starting to make less sense, especially because I get the same error running with only 1 million samples (3MB).

Code 6 can also indicate improper file permissions, but I'm overwriting the exact same file as I was before. Also my disk is gigantic, so it can't be that I'm out of room on it.

[nathand]

How are you opening the file? If you wire a path directly to "Write to Binary File" it appends to the existing file, according to the help. If you want to overwrite the existing file, you need to open that file first, with the mode set to Create/Replace. Are you doing that? Do you get an error when you open the file, or when you try to write to it?

[sjandrew]

I'm opening the file with the open file block, with the mode set to create/replace. The error comes when I try to write.

[Trager]

And the error is coming from the write primitive, not from the file open?

[nathand]

I would guess there's something about your particular code that's causing the problem; I don't think it's the size of the file. Have you written a simple VI, such as the attached code, to test writing a large file? I just ran this on my machine; it runs in a matter of seconds and creates a 38.1MB file. Maybe you could install LabVIEW on another machine - one that's on the network? If I'm not mistaken, the per-user LabVIEW license does allow it to be installed on more than one development machine (up to 3, I think) so long as only one is in use at a time.

[Trager]

I thought maybe the issue had something to do with you running XP and LV 8.5, but I just ran a similar test to Nathan's using that setup (well, LV 8.5.1, to be precise). No error (and it typically executes in under a second on my system).

[crossrulz]

As much data as you are gathering, you might want to look into using a Producer/Consumer to stream your data to disk while you are acquiring it.  This works best if you have many data points (an array) being sent to your consumer loop for logging.  This will save you time and memory.

[sjandrew]

1. Yes, the error (out of memory) was coming from the write primitive.

2. I tried the simple vi that nathand suggested, and it runs much faster - a few seconds or so.

3. I am using LV 8.6, not 8.5 - not sure if that should make much of a difference, since I'm not trying to do anything fancy here.

4. I'll give that producer/consumer thing a shot, but at first glance, I'm not sure I'll be able to implement it; accessing my measurement device involves calling a dll (through LabView's "Call Library Function" node), which returns a (pointer to) an array. The rate, then, at which data is being "produced," is probably the rate that LabView copies them to its own array (assumption). I could pop off each element of the LV array and queue it, and then dequeue it to write to disk. This seems like more operations than necessary.

[akovaski]

"calling a dll, which returns an array"

Are you saying all 10 million samples are from one single call to the dll? If so, yeah, producer consumer doesn't help speed unless you can break it into smaller chunks of data.

Also, this may be a primitive question, but have you checked the length of the array which you get from your dll call to make sure it's 10 million and not something larger?

[sjandrew]

Yes, the single call to the dll function returns the array. It then gets passed up the chain through a few subVi's, and then written to disk. I tried moving the write op into the the VI which calls the DLL, to see if that sped anyhing up - it did not.

The function in the DLL also returns the size of the array, and it is 1 million. Additionally, LabView's "get array size" function returns 10 million.