LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

How to determine binary file data set size

Solved!
Go to solution

Hi all

 

I am writing specific sets of array data to a binary file, appending each time so the file grows by one data set for each write operation.  I use the set file position function to make sure that I am at the end of the file each time.

 

When I read the file, I want to read only the last 25 (or some number) data sets.  To do this, I figured on using the set file position function to place the file position to where it was 25 data sets from the end.  Easy math, right ?  Apparently not.

 

Well, as I have been collecting file size data as I have started the initial tet run, I am finding the the file size (using file size command and getting number of bytes as a result) that the size is not growing the same amount every time.  My size and format of my data being written is identical each time, an array of four double precision numbers.

 

The increments I get are as follows, after first write - 44 bytes, after 2nd - 52 bytes, 3rd - 52 bytes, 4th 44 bytes, 5th - 52 bytes, 6th - 52 bytes, 7th - 44 bytes and it appears to maintain this pattern going forward.

 

Why would each write operation not be identical in size of bytes.  This means that my basic math for determining the correct file poistion to read only the last 25 data sets will not be simple and if somewhere along the line after I have accumulated hundreds or thousands of data sets, what if the pattern changes.

 

Any help on why this is occuring or on a method of working around the problem would be much appreciated.

 

Thanks

 

Doug

Doug

"My only wish is that I am capable of learning each and every day until my last breath."
0 Kudos
Message 1 of 22
(7,796 Views)

Apparently, the sizes differ by 8 bytes. Is all writing done with the same code fragment? (coud it be you sometimes prepend the sizes and sometimes not?)

 

If you would flatten the data before each write to a a string, what is the string size each time?

 

It would help if you could attach some simple code that demonstrates the problem. There are millions of ways to write binary data. we need more specifics.

0 Kudos
Message 2 of 22
(7,794 Views)

I have stripped out the DSC module functions from the vi and attached it here.  I also set default values to all the inputs so it will run with no other inputs.  I also included my current data files (zipped as I have four of them) though the file names are hard coded in the vi so they can be changed to whatever works locally. In fact probably will have to be to modified for the path anyway.

 

If you point to a path that has no file, it will create a new one on the first run and the file size will show zero since there is no data in it. It will start to show the changes on each subsequent run.

 

As I am creating and appending four different files, each with it's own set of data but always the same format (array of four double precision numbers) and the file size information always increments the same for all four files (as will be seen in the File Size Array) I don't think it is a function of the size of the acutal numbers but something idiosyncracy with how the binary file is created.

 

If this proves to be a major hurdle I guess I could try a TDM file but I already got everything else working with this one and need to move on to other tasks.

 

Thanks for any continued assistance

 

Doug

 

Doug

"My only wish is that I am capable of learning each and every day until my last breath."
Download All
0 Kudos
Message 3 of 22
(7,787 Views)

@dacad wrote:

I have stripped out the DSC module functions from the vi and attached it here.  I also set default values to all the inputs so it will run with no other inputs.


The "SPC test results array" is empty, thus no data will be produced in the current default state.

 

 

0 Kudos
Message 4 of 22
(7,770 Views)

I double checked the file I uploaded but have insured there is data in the SPC Test Results Array and made sure to make current values default and uploaded it again.

 

If for some reason it is empty, there only needs to be a 2 col x 4 row set of data.  Only the Measurement, Ref_Point and Current fields need values for this.  The Measurement and Current fields are measured values so can be pretty much anything for testing.  The Ref_Point is an identifier and needs to be set to a value of  4  in the first column and  12  in the second column (in each row for the 4 rows)

 

Not sure why it would have been empty on the first upload.

 

The problem I am having right now is, even if I select a random offset for the Set File Position, the Read always returns data starting at the beginning of the file.    For some reason, if I wire the refnum from the Set File Position to the Read vi, I get an error on the Read.  In order to not get the error, I have to wire the original file path to the Read vi.  I think this removes the pointer from the Set File Position and resets it to zero.   Not sure why the Read vi won't take a refnum for the file path.

 

I have looked at the TDMS format but it appears to be more tedious to work with.

 

Appreciate the continued assistance

 

Doug

Doug

"My only wish is that I am capable of learning each and every day until my last breath."
0 Kudos
Message 5 of 22
(7,760 Views)

anothe approach is to use teh DB approach and use and index file that has nothing but the file pointer values as to build up the file.

 

Since that file should have easy math you could indix into it find you rpointer then use that to read your file.

 

Probably easier than switching to TDMS...

 

Ben

Retired Senior Automation Systems Architect with Data Science Automation LabVIEW Champion Knight of NI and Prepper LinkedIn Profile YouTube Channel
0 Kudos
Message 6 of 22
(7,753 Views)

Maybe something like this would help.  I do not prepend the array size, so each element is a double.  There are 8 bytes per double.  So if you know how many samples to grab, just multiply by 8 and set the offset from the END (bet you didn't know you could do that, no need to get the size).  You can then read all of the rest of the data to the end of file.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 7 of 22
(7,747 Views)

Ben, Not surewhat you mean by the DB approach.  Database ? (Access?)

 

Crossrulz,   Tried this code and I continue to have the same issue with wiring the renum to the Read Binary vi.  It gives me an error every time and I get no data.  If I wire the original file path, it reads the file but not from the point intended.

 

I am uploading another cleaned up version that I have verified runs on it's own.  The only thing that would need to be changed would be the file paths for the bin files.

 

Running as is, I get my 4 x 25 2D array as desired but it is from the beginning of the file.  I tried setting the file position but the Read Binary just won't take a refnum for me and My guess is that it is indeed resetting the position when wired with the original file path as I have it.

 

Thanks for the continued assistance......  Doug

Doug

"My only wish is that I am capable of learning each and every day until my last breath."
0 Kudos
Message 8 of 22
(7,734 Views)

@altenbach wrote:
The "SPC test results array" is empty, thus no data will be produced in the current default state.


Sorry, I only mistook it as empty because it was greyed out due to the missing typedef. After disconnecting from the typedef things are OK.

0 Kudos
Message 9 of 22
(7,725 Views)

I don't see any problems to wire the file refnum from the Write File to the Read File without any need to reopen the file for read. Of course there would need to be some Set Offset in between to set the pointer backwards.

 

But you do never close the file refnums!! BAAAD!

 

Especially with the situation where you write to the file and then "reopen" it to read from it. The OS caches any writes to disk and you can only be sure that the data is on the disk after you have either flushed the file or closed it successfully. So when you try to read it in the current situation, no data may be on disk at that point, and since it is a new file refnum it is not aware of the intermittent state of the file in the other refnum.

Rolf Kalbermatter
My Blog
0 Kudos
Message 10 of 22
(7,722 Views)