What is the best way to write 10 channels of data each sampled at 4kHz to file?

[jcannon]

Hi everyone,

I have developed a vi with about 8 AI channels and 2 AO channels... The vi uses a number of parallel while loops to acquire, process, and display continous data.. All data are read at 400 points per loop interation and all synchronously sampled at 4kHz...

My questions is: Which is the best way of writing the data to file? The "Write Measurement To File.vi" or low-level "open/create file" and "close file" functions? From my understanding there are limitations with both approaches, which I have outlines below..

The "Write Measurement To File.vi" is simple to use and closes the file after each interation so if the program crashes not all data would necessary be lost; however, the fact it closes and opens the file after each iteration consumes the processor and takes time... This may cause lags or data to be lost, which I absolutely do not want..

The low-level "open/create file" and "close file" functions involves a bit more coding, but does not require the file to be closed/opened after each iteration; so processor consumption is reduced and associated lag due to continuous open/close operations will not occur.. However, if the program crashes while data is being acquired ALL data in the buffer yet to be written will be lost... This is risky to me...

Does anyone have any comments or suggestions about which way I should go?... At the end of the day, I want to be able to start/stop the write to file process within a running while loop... To do this can the opn/create file and close file functions even be used (as they will need to be inside a while loop)?

I think I am ok with the coding... Just the some help to clarify which direction I should go and the pros and cons for each...

Regards,

Jack

[PiDi]

Hello Jack

I'd go with TDMS - it is easy to use, flexible, safe (in contrast to low-level file operations) and (most importantly?) fast. One drawback is that they are binary files, so you can't simply view them in notepad. However, you can view them without any problems using TDMS File Viewer.vi in LabVIEW, and also in Excell, using plugin shipped with LV, or in DIAdem. Here are some starting informations about it:

http://www.ni.com/white-paper/3727/en

http://www.ni.com/white-paper/3539/en

[jcannon]

Hi PiDi,

Thanks for the reply... I was hoping to avoid TDMS measurements as I am not that familiar with them.... But I will give it a try... Can you pronot any comments about using the "write measurement to file.vi" (which opens and closed each interation, think??) vs. the low -level TMDS functions? Is the open/close operations with the vi likely to interfere with data collection?

Thanks,

Jack

[deppSu]

I think either apporches will not lead to interefere with data collection if they are used properly. For the TDMS VIs, you can use "TDMS Flush" to empty the buffer at intervals if you worry data lose with corruption.

[jcannon]

Hi deppSu,

is there any way I can check or confirm that no data re being lost? For example, let the instrument run during a mock test and return the data, while also having indicators so I can observe the change in for buffer size/#elements (with and without recording) just so I cna "eyeball" what is happening..?

Thanks,

Jack

[deppSu]

I think you can replay (read data out) when logging to make sure data are written into file.

[PiDi]

Here's a little benchmark to check the speed of various methods. Compare  "TDMS open/close once" to each other - you'll see the enormous difference in performance. This is absolutely the best way to write measurement file, and it is safe - even without closing TDMS file all the data should be there. And of course you should close the file you opened somewhere, so I can't see
why you are afraid of loosing data? You may also add TDMS Flush, as deepSu suggested.

Write to Measurement File.vi with TDMS set works pretty well too, but I believe it just does exactly the same as "TDMS open/close once", with additional (quite big) overhead from the express VI.

[jcannon]

Sorry !!

I am using LV10.0... Any chase you can re-save...?

Thank you very much for this..

Regards,

Jack

[PiDi]

Sure, saved for LV2010

[DFGray]

One thing you have not mentioned is how you are consuming the data after you save it.  Your solution should be compatible with whatever software you are using at both ends.

Your data rate (40kS/s) is relatively slow.  You can achieve it using just about any format from ASCII, to raw binary and TDMS, provided you keep your file open and close operations out of the write loop.  I would recommend a producer/consumer architecture to decouple the data collection from the data writing.  This may not be necessary at the low rates you are using, but it is good practice and would enable you to scale to hardware limited speeds.

TDMS was designed for logging and is a safe format (<fullDisclosure> I am a National Instruments employee </fullDisclosure> ).  If you are worried about power failures, you should flush it after every write operation, since TDMS can buffer data and write it in larger chunks to give better performance and smaller file sizes.  This will make it slower, but should not be an issue at your write speeds.  Make sure you read up on the use of TDMS and how and when it buffers data so you can make sure your implementation does what you would like it to do.

If you have further questions, let us know.