回复： TDMS Defrag Progress Switch Question

[Hooovahh]

Starting in LabVIEW 2015 (I think) there is a shipped example with LabVIEW which shows how to defrag a TDMS file, and get the progress of the defrag process.  This uses two new VIs, one to set and clear the flag telling the defrag to provide the progress, and another VI to actually get the progress.

My current design is using an actor based architecture where each testing location will have a reentrant VI handling the logging of the test for that location.  And I wanted to have this logging actor check to see if the file is fragmented, and to defrag it periodically.  I'd also like to defrag on test completion.  I figured having a progress of this defrag would be a great feature but with these two VIs being undocumented I'm unsure how they will work when there are potentially parallel processes also performing defrags on other files.

My specific question is in regards to the disabling of the progress switch.  For the progress to be reported you need to enable the switch, then you can read the progress, and in the example it shows disabling the switch.  But if I am defragging with a progress bar in a parallel process, disabling the switch will disable it for all files and that's not what I want.  So what repercussions might I see if I leave the defrag switch on throughout the life of the application?  Attached is a quick example of what I mean.  It generates two files, one which takes a longer defrag than the other, but they both start at the same time.  If I disable the switch when the first stops, then progress won't be reported for the second file.  Am I okay in leaving it enabled forever?  Or do I need to track what files are being defragged, and to disable the switch when they are all done?

By the way the lack of documentation on these functions exists in 2016 still.

[magpie15]

I have a couple of questions for you. First, when you say the lifetime of the application, how long are you talking about? Do you mean forever or a couple of hours? Also, are you wanting to be continuously defragging the same file or new ones as they are generated?

I ran your example and saw the error that you are seeing. I am not sure what the consequences are of leaving the switch on.

[Hooovahh]

---

@magpie15 wrote:

First, when you say the lifetime of the application, how long are you talking about? Do you mean forever or a couple of hours?

---

The application will likely run for many months (maybe up to 😎 without being shutdown.

---

 Also, are you wanting to be continuously defragging the same file or new ones as they are generated?

There's multiple operations going on here so let me see if I can explain what my intent is.  Lets say there are two nest locations in a tester, where each location can have a single UUT inserted and tested independently.  Each UUT will create a TDMS file with the data for that UUT, for that test.  At the end of a test I want a defrag to take place as part of a finalization process.  These files will likely be transfer to other processes, and sending more data than is necessary isn't something I want to do.  Since either nest location can be started or stopped independently, there could be a case where UUT 1 finishes a test and starts the defrag process, then before finishing UUT 2 finishes a test and starts the defrag process.  Now I will have the status of the finalization of each UUT, but if UUT 1 finishes the defrag first and sets the flag false, then UUT 2 progression won't be reported.  The reason I was asking about just leaving this flag on all the time, is because I didn't really want to clear the flag, not knowing if another position was using it, and it would be easier to just leave it on.  

But I think a easy enough solution might be to use a VIG and just track what positions are defragging, and if they all finish then clear the flag.

In addition to defragging at test completion I intended on periodically querying the fragmentation status of the log file, and defrag it if it needs it, then continue logging to it.  The thought process here is defragging a 1GB file might take a long time, but if I can defrag the file at every 100MB then the finalization of the file will take less time, since the majority of the file has already been cleaned up.

[Craig_]

This is tangential, but can you update how your code is writing these TDMS files to not cause fragmentation?

There's a pretty good article on this here: https://devs.wiresmithtech.com/blog/tdms-fragmentation-cslug/

Basically, the key is to write everything to the TDMS file with a single TDMS Write that always gets the same data, so that the file structure doesn't have to re-write the headers constantly resulting in fragmentation. In my experience there's a lot you can do to reduce fragmentation to a point that you don't need to worry about defragmenting them!

[Hooovahh]

Yes I do my best to not create fragmented files, and in practice I think the files won't be very fragmented.  But again one of the goals is to have a finalization process be quick, and with that in mind I either can not defrag at the end, or to make sure the defrag process at the end is quick.  One way of making the final defrag complete quickly is to periodically defrag, so there is less to do at the end.  Development hasn't been completed but I'm expecting files on the order of a GB or so.  I don't intend on creating fragmented files, and I don't intend on defragging constantly, but I want to test for these types of situations.

[emmazheng · Accepted solution]

As you found that if you disable the progress switch, all the defragmentations for different files cannot be shown in progress. Leaving it eanbled forever is fine but can make the defragmentation a little slower for case of not needing to show progress.

Jie

NI R&D