read large data from binary file(.dat) and plot

[hhtnwpu]

hi, I want to read large data(nearly 100 million rows) from binary file, the original format of the file is .txt, through rename as .dat we get the binary file. The attachment is my vi.When the file is larger than 1 million lines(the data is singal Column), there is a wrong with "the memory is full".I want to read the data and plot a graph in time domain, through the graph I can see the detail by zoom tools; and then do some anlysis about FFT and Statistics .if who can help me improve my VI in the attachment about read data and plot?Thanks!

[hhtnwpu]

my labview version is 8.6

[GerdW]

Hi hh,

no, your file will not be a "binary file" by renaming it from *.txt to *.dat. Atleast you still consider the file content to be ASCII/human readable numbers as you use SpreadsheetStringToArray to convert the contents into numbers...

- When your file only contains one column of data you should convert the string directly into a 1D array instead of the 2D array you're creating now!

- When it still comes to problems you should read your file in small pieces. This has been suggested a lot of times before - probable you will find also examples demonstrating this suggestion here in the forum!

[hhtnwpu]

thanks for your suggestion. I maybe use some data file which have several columns, but now I must slove the problem with large. the data from a DAQ,the sample rate is 10KHz, about 2 hours.As you said , the data is not binary data ,just text file data, can i convert the format to TDMS ?

you sail that "When it still comes to problems you should read your file in small pieces. This has been suggested a lot of times before - probable you will find also examples demonstrating this suggestion here in the forum"

I searched the example for a long time but get none result, can you give me a example? thanks!

[mikeporter]

When dealing with datasets of that size, you need to remember the resolution of your screen. For example, say your graph indicator is 700 pixels wide and 600 wide. This display only has 420,000 pixels within its borders. Clearly every datapoint in a million point dataset will not have its own pixel on the screen.

What you are wanting to do is not simple. Say you want to start by displaying the entire dataset and you are using the 700x600 graph I mentioned above. As you read the data in, you will need to decimate it such that it has no more than 700 unique X axis values before plotting.

Now when the user zooms the display, you will need to define an event (Scale Range Change) that see the scale change and will read in only the data associated with that new range and again decimate it such that it has no more than 700 unique X axis values before plotting.

The effect visually is even pretty cool. when they first zoom in they see the same low resolution data (but larger) that then changes to higher resolution a monent later.

Obviously, if your data is in asscending x-axis order and not an XY dataset, this will be much easier - and may even help relieve your memory problems. This will also be much easier if your file really is in binary...

MIke...

[mikeporter]

Another thought on the decimation:

Say you have a graph that has 1 million datapoints in it and a graph that is 700 pixels wide. That means every horizontal dot on the graph will be "representing" over 1400 datapoints. The question is: What value do you want to use to represent all that data? The maximum value? the average? the median?

That is something you will have to decide as there is no single "correct" answer.

Mike...

[hhtnwpu]

"This will also be much easier if your file really is in binary..."

the file I think is not a really binary,I just renamed the text file as *.dat. when i convert the file to tdms, the volume of the file doesn't smaller the the original.

[hhtnwpu]

"The maximum value? the average? the median?"

 maybe i should do a chose bottom, let the users to chose which one they want.thanks!

[hhtnwpu]

anybody could supply a example for me ? thanks a lot! i searched online, but find none.

[deppSu]

TDMS is just a one kind of data format that belongs to the binary data. In your case, you should write a program read the txt data out convert them to numeric type instead of txt strings and store them to a TDMS file.