Best Method for Saving Data to File?

[Meg T]

Hi, 

We're using labVIEW 2009 to acquire data from our instrument.  In the past we have used the "write to labview measurement file" express configuration tool to save data to a file. However we had some issues with our VI - occasionally we would lose data or column headers somewhat erratically and were never able to sort out the problem. We would like to rewrite our VI with the best possible solution. 

 Here's a summary of what we would like to do:

Save 30 variables at a rate of roughly 1 Hz. We would like one set of column headers per file so that the data can be easily imported into labVIEW with the variable names intact. We will be collecting data continuously, so we would like to divide the data into 3-4 files per day. Ideally, the program to start new files at the same times from day to day and the filename could be configured to include the date and time/file number.

I am hoping that users can provide a little feedback about methods that were most successful and reliable. From what I have read there are a few different ways to do this (express VI, tdms, "write to text file"). Any thoughts or relevant examples would be quite useful for us!

Thanks for your help!

[tbob]

One part at a time.  Here is how to create a filename with a timestamp:

[tbob]

This code shows how to append a column header to the data array, then write it to a spreadsheet file.  Since the columns headers are strings, you must convert the data into strings.

[tbob]

This code will gather data for a number of hours, then write the data to a file.  The main loop repeats until the stop button is pressed.  Notice that the headers are written before anything else.  Then the data is appended to the file inside the loop (notice the True wired into the append terminal).  You will have to change the filename on each outer loop iteration to include the timestamp.  See my previous code to do this.

Mistake:  You need to put the first file write inside the loop so that it can create a new file upon each loop iteration. 

[Meg T]

Hi tbob,

Thanks for your reply. I have a quick question about how this column header would work in practice. Is it correct to say that in your method, the indexing results in building up the data into one large array of data before saving it to the file with the column headers and filenames appended?  If we are writing data at 1Hz for 6 hours of time, will we run into an issue being able to store all the data?

In the past, we used the express vi configured to "save to one file" and the "append" option so that the "write to labview measurement file" vi was called every second. I have been playing around with using the same vi configured to "write to a series of files" since it has the option to start a new file at a specific time of day and/or after a certain period of time. In addition to the fact that there is some bug relatin to the 2009 version of "write to a series of files", it also seems difficult to incorporate headers into express VI if you are writing the data continuously as a part of a loop with the "append" option.

Thanks again for your help!

[tbob]

---

@Meg T wrote:

Is it correct to say that in your method, the indexing results in building up the data into one large array of data before saving it to the file with the column headers and filenames appended?  If we are writing data at 1Hz for 6 hours of time, will we run into an issue being able to store all the data?

---

Yes the indexing will build up one large array.  If this is a problem due to array size and memory, you will have to write more often.  You can write every second, that should not be a problem.  You should still write the column names to the file first, and have the append input set to False (or nothing wired in since the default is false).  This will create a new file with the headers only.  Then the data write has a True wired in so that the append takes place.  If running for 6 hours and gathering data once per second, your array will contain 36,006 rows of data.  I'm not sure if that would cause a memory problem or not.

---

@Meg T wrote:

 it also seems difficult to incorporate headers into express VI if you are writing the data continuously as a part of a loop with the "append" option.

---

If you write the headers before the loop as I have shown, and use append inside the loop, you will not have problems.

[Meg T]

Thanks! I'll work on implementing the method you have suggested! I really appreciate your suggestions.

[Thehammer]

do you the VI application for this Best Method for Saving Data to File thanks

[billko]

The user hasn't been here since 2011.  You probably aren't going to get anywhere by asking them.  Besides, it shouldn't be too hard to assemble given previous messages.