Question regarding the "date modified" info of a TDMS file

[Nadweb]

My VI is writing data to a TDMS file every few seconds, but when I look at the file in its folder in windows explorer, the date modified doesn't change until I stop the VI. Is this normal, since the file doesn't close until the VI stop? I want to make sure the file is actually being updated 

[Nadweb]

The file size also says 0 until the VI is stopped.

[Nadweb]

Should I be using the TDMS Flush function? If I'm understanding correctly, it seems that the TDMS Write function is only writing to a buffer until the TDMS Close function runs. 

[cbutcher · Accepted solution]

---

@Nadweb wrote:

Should I be using the TDMS Flush function? If I'm understanding correctly, it seems that the TDMS Write function is only writing to a buffer until the TDMS Close function runs. 

---

Yes - if you want it to write to disk, call Flush.

It will occasionally flush by itself, but I guess it depends on the rate at which you're writing data - when we acquire high-sample-rate data, it generally writes pretty quickly, but our temperature data ~1Hz*5 channels (so 5S/s) might take a very long time without manual Flush calls.

This also has an impact on "oh no, my computer crashed/restarted/did something stupid"... Flushing saves you losing more time etc.

[Nadweb]

Thank you for that information!

Since I'm collecting data pretty slowly, but for extended periods of time (20 signals collected every 30 seconds or every 1 minute for multiple days, or weeks) would it hurt performance to flush every time data is collected? If or when we need to collect high speed data, I could then change it so it doesn't flush every iteration. 

[cbutcher]

I would definitely use the Flush VI in that case. It may not make a difference if everything is going smoothly, but it makes a huge difference if something goes wrong or you need to unexpectedly abort the VI or something...

[Kevin_Price]

2 brief remarks:

1. Try refreshing the file view in Win Explorer.  I've noticed a similar staleness of info on some (not all) of our test systems (file existence, size, date/timestamp) until I did a manual refresh in Explorer.  Others seem to automatically refresh every second or two.  Haven't had the inclination to dig in further to understand what drives the difference.

2. If/when you do need to support higher-speed logging, you could consider something like the Elapsed Time vi to let your decision to Flush or not be based on time since previous Flush.  That's the kind of approach I've tended to use for long running general-purpose loggers, usually with intervals around 30-60 seconds.

-Kevin P

[Nadweb]

Thanks for your input Kevin!

I just have a quick follow up question for you regarding part 2. of you post.

Are you saying you also collect data every 30-60 seconds, or you flush the data every 30-60 seconds? I'm wondering how often I should flush the data. I guess it depends on how fast your collecting and how many data points you don't mind losing if something were to go wrong.

[Kevin_Price]

I was talking about doing an explicit flush every 30-60 seconds.  But you're exactly right -- it depends on the situation.

The situations I was thinking of were generally long-term tests that ran for somewhere in the neighborhood of a day to a couple weeks.  So most of that time the testing is autonomous and unattended.  I came to the conclusion that for the kinds of things that could stop my test early (power outage, Window crash), a couple minutes worth of data give or take was unlikely to be any kind of difference maker.  If I intended a 48 hour test, the difference between collecting data for 27.10 hours vs. 27.15 hours before a crash probably won't influence my decision about whether to continue the test from where things left off or whether to start all over from the beginning.

There could be other situations where something unexpected *about* the test can cause the unexpected crash, in which case the very most recent possible data before the crash might be invaluable for investigating.  Then you might want to flush after every write.

-Kevin P

[Nadweb]

That all makes a lot of sense, thanks for talking that through with me.

It seems like for my slow data acquisition (every 30-60 seconds), I'll just flush the data after every write, mainly for simplicity and it won't slow anything down. If I need to run fast data acquisition, I'll decide on how often it would be appropriate to flush while still optimizing speed/performance. In either case, I don't think losing a few minutes or seconds of data, if something went wrong, would really effect us. My tests aren't changing that rapidly. 

Thanks!