Parsing binary file when writing a 2D array

[AlecSt]

Hi all,

My lab partner and I are writing a program to write a 2D array of numbers to a file. The first 32 bits are a seconds count, the second 32 bits are a microseconds count, and the remaining data is stored as unsigned 8bit integers which come from the array.

My lab partner is writing a program to parse the data and has so far been unable to. My question is: what is, schematically, the way the data is written? Is it just 32bits + 32 bits + 8 + 8 and so on?

I've provided the VI in case there are any questions.

[altenbach · Accepted solution]

What's the point of the FOR loop? You can remove it and nothing will change. (2D arrays are stored flat. There are no "row delimiter" or similar)

Why do you set the file pointer to the end? Since you "replace or create", that seems illogical and useless. Just omit that function.

---

@AlecSt wrote:

My question is: what is, schematically, the way the data is written? Is it just 32bits + 32 bits + 8 + 8 and so on?

---

Yes.

---

@AlecSt wrote:

My lab partner is writing a program to parse the data and has so far been unable to.

---

Are you trying to read it using LabVIEW or some other program?

[AlecSt]

Thanks for the reply. Replace or create was not the previous setting -- "open or create" was. The data is meant to be written to the end of a file.

Thank you for answering the question.

[altenbach]

---

@AlecSt wrote:

The data is meant to be written to the end of a file.

---

Is the 2D array always the same size? How would you later know when one set ends and the new one begins? Or are the two 32bit number written only once at the very beginning of the file?

[AlecSt]

The structure is the following: every time the file is called, the program ought to go to the end of the file and write the following things: the 32 bit seconds count, followed by the 32 bit microseconds count, which is lastly followed by a 2D array (which is always the same size.) This forms one chunk. The file is a lot of chunks concatenated together.

[altenbach]

OK, of you know the chunk size, you are all set. You can easily calculate the file offset for each chunk.

Do you still have problems with reading it later? If so, please explain how you are trying to read it.

Note that the file does not contain size information for the 2D array, so you need to read it as a flat 1D U8 array and reshape it later to the correct 2D dimension sizes. You could even read the entire chunk as a binary string and parse it afterwards.

[Bob_Schor]

I hate to disagree with Christian (who is almost always correct), but LabVIEW, by default (unless you tell it otherwise), dpes write the array size(s) when writing a binary array, and will read it when reading a binary array.  So all you need to know is that the file contains two I32s and 2D arrays of whatever type you want (they can even all be of different sizes -- you only need to know the dimensionality and type.

Here's a snippet that writes two single I32s, then a 3x4 array of dbls (I chose this type as it should be really clear if you mix up the data).

Bob Schor

[altenbach]

---

@Bob_Schor wrote:

I hate to disagree with Christian (who is almost always correct),

---

If you look at his code, the array sizes are all disabled (wired with a false). I was assuming that this is required for the file to be processed in another (possibly non-LabVIEW) application. 😉

I still agree with myself! 😄

(If he would change it now, the old and new files will not be compatible without special decision code, so this seems offlimit for the current discussion)

[Bob_Schor]

As they used to say on Mr. Roger's Neighborhood, "Correct, as usual, King Friday".  I did not look at the original code, and did not notice that the (very helpful) option to save array sizes for binary writes was turned off.  This does make the problem of recreating the data much more problematic, as the information required is not only to know the type (I32, U8, Dbl, etc.) of each element, but, if an array, the precise dimensions of each one.  Saving the array size as part of the binary write makes the subsequent binary read much more "reliable" and less prone to "user error".  It also allows multiple binary arrays to be different sizes without worrying about it.

BS

[Bob_Schor]

I just realized that if the Array data are written without the array size indicators (as the Poster and Christian noted), you can still simplify reading the array (I'm assuming that once the two I32 variables have been written, everything else is one, or possibly more, 2D arrays).  Simply wire a 1D array of the appropriate type into the Binary Read function -- this will read all of the remaining data into a single 1D array.

If you know the size of the data arrays being written, you can reshape the 1D array into a single 2D array of the appropriate dimension, or into a 3D array, where the third dimension refers to multiples of the 2D array being read/wriitten.  If you do not know the siize of the datal, you can guess by factoring the size of the 1D array.  For example, if the 1D array was of size 28, then the underlying 2D array is a 2x14, 4x7, 7x4, or 14x2.  See which guess seems to fit the data best.

Of course, all bets are off if the file contains multiple arrays of different sizes ...

BS