TDMS Defrag Locks up LabVIEW

[CarstenPXI]

The de-frag crash is a known issue in LabVIEW 9.0.  Basically, we are now avoiding any use of de-frag until LabVIEW 2010 releases when it should be fixed.  In my opinion it is just too dangerous, because its crash is unpredictable and therefore can end up crashing a critical data collection session.  (It crashes LabVIEW).  This is described in the following thread:

http://forums.ni.com/ni/board/message?board.id=170&message.id=470686&query.id=4830813#M470686

I am nowusing a per-channel buffer of about 1 Mbyte, this seems to keep the index files small, and removes the need for de-frag,  greatly improves TDMS readperformance, and reduces the locking issues you are seeing, because they are primarily (my guess) due to the time consumed by the de-frag operation.

In any case, even when the de-frag gets fixed, I still think it might be better practice (peformance wise and reliability-wise) to run the de-frag once, after all data acquisition is finished.  I have this (perhaps irrational) nervousness about the risks inherent in constantly re-packing index files.  Maybe Herbert or some of the other NI experts can comment onthis.

Carsten

Carsten

[zou]

Thanks, Carsten.  I read your previous post before I posted.  I'm aware of the crash problem.  I think defrag frequently can overcome the problem.  I have 8 GB RAM.  I ran into this crash problem only when not defrag during the test, and defrag at the end.

I'm using buffer (disable buffer set to FALSE when open the TDMS file).  How do you set 1 MB buffer per channel?  How small is your index file? How long it takes to open your file?

[blawson]

Won't calling TDMS flush cause the reading VIs to be able to get the data written after they opened the reference?  I think it needs to be called by the writing VI though.

Edit: I believe that Carsten is talking about handling a buffer outside of TDMS in order to minimize the number of calls to TDMS Write.  This is what I do also and it seems to work fairly well, but it complicates concurrent read-write situations.

Message Edited by blawson on 04-13-2010 01:25 PM

[Mac671]

Hello I would like to add to this thread.

I am streaming a TDMS file @ (200 MB/s) in order to be able to work quickly with the data after the acquistion is complete, I would like to be able to defragment the file after my measurement is done.

The measurement can run in to many Gigabytes.

I've been considering two options.

1) Create a Software Tool with a GUI, that would use the TDMS defragment vi, and allow the user to select the files for defragmentation manually.

2) Make a Deamon that will start / stop defrag on files allready acquired, if measurement is not in progress.

The issue I'm running into is that for example for a 1.8 GB file. When I run the defrag ... the LabVIEW UI Freezes , LabVIEW stops responding until the defrag is done,and I have no way of telling how it is progressing until it is done.

I can't use my development environment for that time ... and hence can't give the user any feedback on where the defrag process currently is, when will it complete ( like a progess bar ).

So I'm having diffiiculty with figuring out on how to create the GUI tool.

I've not played with option number 2 yet. But I do not see an option to start and stop defragmentation, once it has started.

For example if I start the measurement, I would like to send an event, stop defrag. and resume it from when it was stopped once the measurement is not running ).

Ant thoughts on this would be appreciated.

I am using LV2010.

[blawson]

You can't cancel a defrag operation, so your only option is #1.

Defragging these files is very resource intensive so it's not surprising that it freezes up your UI.  In this situation, the best you can probably do is to warn the user ("Defragging, go make some coffee") and set a busy cursor.

[Mac671]

In the era of multi core processors, and multi threading using the resources in a way that locks up the UI isn't really somethign I belieave anymore.

Thanks for the tip though, as this lock up might mean that the whole TDMS Defrag VI runs in the User Interface Thread ... I will try to change the VI execution system or play around with other stuff maybe this will help.

It is too bad that there doesn't seem to be an ability to Pause the defragmentation... Perhaps I should read in pieces of the file, defrag them? Is that even possible?

Is there any documentation on how the defrag commences so I can implement my own version of this?

( I know there are papers on the TDMS file strucutre, so I might try and read in one TDMS and Convert it to another defragmented one ?)

Thanks again,

Maciej

[YongqingYe]

Yes, it's possible to write your own Defragment one. TDMS Defragment basically reorganizes your data objects (raw data and meta data) and reduce the file size. If you really want to implement your own defragment one, you can list all the objects in the file by using TDMS List Contents and read out all the meta data (like properties) and raw data from each object and rewrite them to a new TDMS file from object to object.