Reading data from file

[AndreaD]

Hi, 

I need to analyze data in two ways, one through collecting data through an instrument (where I'm using Queue, data is streaming in live), and the other one I'm reading through a file. In the second method (reading through file), how can I read the data like it is coming in through the instrument - streaming, instead of reading static numbers just stored in an array? 

Thanks, 

Andrea 

[RavensFan]

Read in the entire array, and index through the array one element at a time.

Or read the file one piece of data at a time.

Either way, you'll want some sort of timing control in your Read Loop that slows it down to a rate equivalent to what the streaming method is, otherwise it would process the entire file super quickly.

[Bob_Schor]

When data are written to a file, they are usually written according to some rules, or format.  For example, your data might be a numeric record of voltages from 8 channels, recorded at 100 Hz, written as a string of 8 numbers separated by commas, one line of text per sample (or 100 lines of text per minute).  You could choose to read this back a line at a time, with your "read" routine acting to "stream" a sample from your 8 channels to you each time you did the read.

Other ways of saving data impose other "formatting" rules.  An interesting exception is what is often called a "binary" file, where what is written is a bit-for-bit representation of the data as it exists in computer memory.  Thus an I32 would be written as 4 bytes that hold the 32-bit integer, while a double float would be written as 8 bytes with the sign, the exponent, and the "mantissa" expressed according to IEEE standards.

When you read binary data, you need to know exactly how the data were written, and take care to read it in with the same type of variable.  For example, if the data on the disk were double floats and you read the values into an array of I32 integers, you'd get data, but it look like gibberish and would be a challenge to interpret.

So what format are the data written?  Is the format one that has a "natural division" that makes "streaming" logical?  How (and why) do you want to stream the data?  Note that most file i/o routines let you read data in "pieces", so you (usually) do not have to read it in "all at once" (I can think of some exceptions, for one of which I've formulated a work-around).

Bob Schor

[AndreaD]

I've been trying to figure it out, don't think I'm getting anywhere ... (there are a few things which I've tried, so some things go to nowhere) 

Thanks, 

Andrea 

[Bob_Schor]

Andrea,

     Do not post a picture of part of your Block Diagram.  I can't see the whole thing, I can't "modify" it, I can't test it.  Post the VI itself, please, and I (and other Forum members) will look at it and get back to you.

Bob Schor

[AndreaD]

Here's the code ... I had to take the portion out of the whole code, since the entire thing would be too large. I have the test file attached as well, zipped in the file. 

Thanks, 

Andrea 

[Bob_Schor]

OK.  I started with the Data File, paying no attention to your code nor your note so as not to bias my observations.  I was hoping to see a binary pattern in the organization of the file, and partly succeeded, but it was strange.  Records seem to be about 1657 bytes in size, but they are not all the same length, varying by a few bytes.  The first two bytes of a record seem to be 0xA5 and then a two-byte count that starts at BF 70 (with BF being the least-significant bits).  This corresponds to 28863.  The final record would be number 29352, for a total of 490 records.  If we divide the file size in bytes by 490, we get 1654.37, close to the 1657 noted above.

The Bad News is that I can't find anything else "recognizable" in the data.  But if you want to give it a try, here's one way to proceed:

• Read the entire file into a single 1D Array of U8 (bytes).
• Locate all of the Bytes with a value of 0xA5.  There should be 490 of them.
• Verify that there are 490, and that the spacing of their indices is roughly 1657.  If there's an extra one or two, you may be able to "flag" them as a "false beginning" by their relative index.
• To get Record N, extract the Array of Bytes from Index(N) to Index(N+1)-1 (you might want to add an extra "End-of-File" index).  Record 0 (the first one) will start at position 0 (the index of the first A5) and go to about 1650 (the index of the second A5, minus 1).
• What you do with this collection of Bytes is a mystery to me, as I haven't made much sense of them.

Bob Schor

[AndreaD]

What's the best way of ID'ing all of the A5 values? You should be able to see the values in Data, and ReadData. Data should be in hex, ReadDataArray and ReadData are in decimal (although I forgot to change the format to hex). A5 is the beginning of a packet, the value in between are science data which are decoded in the rest of my code. 

[Bob_Schor · Accepted solution]

Andrea,

     You inspired me to Byte the Bullet (pun definitely intended).  I "gave up" looking to understand the data (though parts of it do look like integers, parts do not), but decided to look at finding all of the A5's.  To my surprise, they were exactly 1651 characters apart, and there are exactly 491 "records" of this length starting with A5 (or 165 in decimal).

     This snippet reads in the entire array as an array of U8 and reformats it into 491 "rows" of 1651 "columns", all of which start with 165.  By scrolling up and down, you may be able to see more patterns and deduce more structure.  Once you think you know how the other bytes are arranged, you can either modify my code or send me a Private Message and I'll provide "off-line" help, with the promise that if we do figure this out "privately", I'll post the full solution so that others with similar issues can benefit from our efforts.

Bob Schor