Large data array to disk

[Trager]

Oops, I misread the description of the data processing. I thought the performance issue was with using a single write after using a loop to generate the data (which is how I think you should do it). Nathan is entirely right that repeated disk accesses are going to chew up resources unnecessarily.

[sjandrew]

I probably seem like an ass, but neither can I take a photo of the screen. In pseudocode:

open file;

FOR num_samples

{

 binary_write sample[i]; 

}

close file;

But what you're saying is that I should do:

open file;

binary_write_entire_array samples;

close file;

And this will have the effect of preallocating the proper amount of space on disk? So I make no changes to my diagram, except removing the for loop? And wire the array straight to the binary write block?

[sjandrew]

The zero problem was with the text version. I'm going to be reading it with various programs later, but to verify that it was properly written, I used Matlab (specifically fread(file, array, 'single'), which, as long as the endianess is correct, will read back the array). Then I plotted it, and see the data as I expect, up until sample 10,000. Using the aforementioned value of 11 bytes per textualized single, I come up with only 100 kB used at that point.

[nathand]

That's correct. And you should set the "Prepend array or string size" input, and the "byte order" input, according to how you plan to read the data later. If the only thing the file contains is that single array, then you don't need to prepend the array size (that feature is useful if there will be more binary data following the array, because it lets you know how many bytes to read - but if the entire file is a single array, you can simply read the entire file). If you'll be reading the data in another program, you may want to set the byte order to little-endian since that's what most non-LabVIEW applications expect.

[Trager]

Yes, that's one way that should work (of course, this being LabVIEW, there are several others as well). I would point out that if you're using the normal Write to Binary File primitive, there is an option for prepending the array size that defaults to true. This can sometimes mess up reading the file in other programs, if you aren't aware of it, and you may find that it should be set to false.

[nathand]

---

@sjandrew wrote:

The zero problem was with the text version. I'm going to be reading it with various programs later, but to verify that it was properly written, I used Matlab (specifically fread(file, array, 'single'), which, as long as the endianess is correct, will read back the array). Then I plotted it, and see the data as I expect, up until sample 10,000. Using the aforementioned value of 11 bytes per textualized single, I come up with only 100 kB used at that point.

---

Well, here you wrote the values as text, and tried to read them as binary in Matlab, so no surprise that you didn't get what you expected.

[sjandrew]

Not necessary to prepend the size, as I've always got 10 million samples, but that's a handy feature.

As for the byte order, I went with the default big-endian. And thus a (one-man) industry-wide standard was born...

[sjandrew]

"Well, here you wrote the values as text, and tried to read them as binary in Matlab, so no surprise that you didn't get what you expected."

Nah come on, I'm not that much of a rookie. I was reading text at that point.

[Trager]

Actually, more often than not, I find that to be a decidedly UN-handy feature. Prepending the size tends to break (or at least, make more complicated) the process of reading the data array.

[nathand]

---

@sjandrew wrote:

"Well, here you wrote the values as text, and tried to read them as binary in Matlab, so no surprise that you didn't get what you expected."

Nah come on, I'm not that much of a rookie. I was reading text at that point.

---

Sorry, maybe I misunderstood what you wrote. I saw "The zero problem was with the text version... to verify that it was properly written, I used Matlab (specifically fread(file, array, 'single'), which, as long as the endianess is correct, will read back the array)." Endianness is, of course, irrelevant when discussing text, so I thought you'd tried to use a binary read on the text file.