LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Large data array to disk

Ya, probably didn't write that post in a very literate way.

 

So now I've tried removing the loop, and I get a "generic I/O error" code 6, which looks like the same problem I was having with the "Write to Spreadsheet Array" - trying to allocate an entire (new) array, before writing the data to disk. I originally put the loop in to avoid this problem, but now that's starting to make less sense, especially because I get the same error running with only 1 million samples (3MB).

 

Code 6 can also indicate improper file permissions, but I'm overwriting the exact same file as I was before. Also my disk is gigantic, so it can't be that I'm out of room on it.

0 Kudos
Message 21 of 37
(1,775 Views)

How are you opening the file? If you wire a path directly to "Write to Binary File" it appends to the existing file, according to the help. If you want to overwrite the existing file, you need to open that file first, with the mode set to Create/Replace. Are you doing that? Do you get an error when you open the file, or when you try to write to it?

0 Kudos
Message 22 of 37
(1,758 Views)

I'm opening the file with the open file block, with the mode set to create/replace. The error comes when I try to write.

0 Kudos
Message 23 of 37
(1,743 Views)

And the error is coming from the write primitive, not from the file open?

0 Kudos
Message 24 of 37
(1,736 Views)

I would guess there's something about your particular code that's causing the problem; I don't think it's the size of the file. Have you written a simple VI, such as the attached code, to test writing a large file? I just ran this on my machine; it runs in a matter of seconds and creates a 38.1MB file. Maybe you could install LabVIEW on another machine - one that's on the network? If I'm not mistaken, the per-user LabVIEW license does allow it to be installed on more than one development machine (up to 3, I think) so long as only one is in use at a time.

CreateLargeDataFile.png

0 Kudos
Message 25 of 37
(1,712 Views)

I thought maybe the issue had something to do with you running XP and LV 8.5, but I just ran a similar test to Nathan's using that setup (well, LV 8.5.1, to be precise). No error (and it typically executes in under a second on my system).

0 Kudos
Message 26 of 37
(1,714 Views)

As much data as you are gathering, you might want to look into using a Producer/Consumer to stream your data to disk while you are acquiring it.  This works best if you have many data points (an array) being sent to your consumer loop for logging.  This will save you time and memory.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 27 of 37
(1,633 Views)

1. Yes, the error (out of memory) was coming from the write primitive.

 

2. I tried the simple vi that nathand suggested, and it runs much faster - a few seconds or so.

 

3. I am using LV 8.6, not 8.5 - not sure if that should make much of a difference, since I'm not trying to do anything fancy here.

 

4. I'll give that producer/consumer thing a shot, but at first glance, I'm not sure I'll be able to implement it; accessing my measurement device involves calling a dll (through LabView's "Call Library Function" node), which returns a (pointer to) an array. The rate, then, at which data is being "produced," is probably the rate that LabView copies them to its own array (assumption). I could pop off each element of the LV array and queue it, and then dequeue it to write to disk. This seems like more operations than necessary.

0 Kudos
Message 28 of 37
(1,618 Views)

"calling a dll, which returns an array"

Are you saying all 10 million samples are from one single call to the dll? If so, yeah, producer consumer doesn't help speed unless you can break it into smaller chunks of data.

 

Also, this may be a primitive question, but have you checked the length of the array which you get from your dll call to make sure it's 10 million and not something larger?

0 Kudos
Message 29 of 37
(1,607 Views)

Yes, the single call to the dll function returns the array. It then gets passed up the chain through a few subVi's, and then written to disk. I tried moving the write op into the the VI which calls the DLL, to see if that sped anyhing up - it did not.

 

The function in the DLL also returns the size of the array, and it is 1 million. Additionally, LabView's "get array size" function returns 10 million.

0 Kudos
Message 30 of 37
(1,600 Views)