TDMS Defragment Function crashes labview

[Blueprint]

---

@Hooovahh wrote:

That's generally a sign that your TDMS file is quite fragmented.  If it weren't fragmented then the majority of the file would be the binary data which isn't really all that compressible, but what is compressible is the header sections that get written periodically.  If you open the TDMS file in Notepad++ you should see very little human readable text.  The more you see, the more fragmented it is.  Another sign it is fragmented is the size of the index file, it should generally be in the order of less than a MB, for me is is usually a couple of KB.

Anyway in Windows 7 x64, LabVIEW 2015 32 bit, Excel 2010.  The TDMS file could be opened and displayed just fine in Excel.  The defrag however caused the crash as mentioned.

---

so you are saying you also got the crash?

I know that the file is fragmented, that is why I am defragmenting it.

I think I was a little unclear in my first post, I shouldnt have written "trying to open", but rather trying to defragment. 

So let me explain better.

Since the TDMS Viewer.vi provided by NI is 1. not defragmenting fragmented files and 2. does not decimate big data sets, I have made my own TDMS viewer which defragments automatically files with index files >20 kB. Furthermore, if the index file does not exist, my TDMS viewer program will first open the file, then close it again to make the index file, then check the size and defragment if needed. My TDMS program also plots large data sets decimated. The performace of my TDMS viewer program is (in all modesty) excellent, since I can plot 6 hours of 18 channel 200 Hz data in matter of seconds. but only if the tdms file is defragmented. 

For those not experienced with defragmentation of TDMS files, here are example times of opening (normally) the same file before and after defragmentation:

Before defragmentation (650MB file) - 45 seconds to open

After defragmentation (400 MB file) - > 1 second to open

By open I mean using the TDMS open.vi.

The TDMS Defragment function is the reason we are using tdms in the first place, because for our use the TDMS format is unusable if we cannot defragment the files....

The reason that our TDMS files are fragmented is because we are logging lots of different data at different times. So that is no surprise 🙂

[emmazheng · Accepted solution]

Hi  Blueprint,

Thanks for your attached file.

I think all the channels of group TopsidePSU_Measurements from your attached file are never written with raw data, or explicitily written with zero length raw data. That's the key of the crash issue. 

The crash is not relative with OS or file size. In LabVIEW 2015, TDMS Defragment function cannot defragment a .tdms file including channels with 0xFFFFFFFF raw data index. That kind of channel not has NI_ChannelLength property when you use TDMS Data Viewer to watch it. All the channels of group TopsidePSU_Measurements have 0xFFFFFFFF raw data index. Thus LabVIEW 2015 crashes during TDMS Defragment. We will improvement TDMS Defragment function.

From your side, I can provide a workaround.:

1. Write zero length raw data for these channels under the identical group in a new .tdms file, as following.

2. Copy the binary of the new .tdms file at the beginning of your attached file. Then you can successfully call TDMS Defragment function in LabVIEW 2015.

The modified file is attached below.

Hope this will help.

Jie

NI R&D

[Blueprint]

---

@emmazheng wrote:

Hi  @Blueprint,

I think all the channels of group TopsidePSU_Measurements from your attached file are never written with raw data, or explicitily written with zero length raw data. That's the key of the crash issue. 

The crash is not relative with OS or file size. In LabVIEW 2015, TDMS Defragment function cannot defragment a .tdms file including channels with 0xFFFFFFFF raw data index. That kind of channel not has NI_ChannelLength property when you use TDMS Data Viewer to watch it. All the channels of group TopsidePSU_Measurements have 0xFFFFFFFF raw data index. Thus LabVIEW 2015 crashes during TDMS Defragment. We will improvement TDMS Defragment function.

---

Hi,

Thanks for investigating this, I really appreciate it 🙂 I can confirm that this also happens using LV2015 on a win7 machine. LV2014 works however.

Yes, there are several channels/groups that does not have any data stored to them (yet). This was not a problem in LV2014, so never thought of writing empty data to them. Would writing the NI_ChannelLength to 0 at initialization of the program making the TDMS files do the same trick (for new files)?

Another workaround for the already logged files is to defragment them using LV2014.

I think we are closing in on an explanation of the problem. It does not, however, explain why it works fine on a 80MB file which also has the channels/groups with no data? Any suggestions?

Øyvind

[emmazheng]

Hi  Blueprint,

I will explain why it works fine on a 80MB file which also has the channels/groups with no data.

In LabVIEW 2015, we improved the time performance for TDMS Defragment for general cases. When the .tdms file size is less than 100MB, TDMS defragments the whole file one time in memory; otherwise (file size>=100MB), if the total segment number meets the limit, TDMS divides the original file into multiple parts and defragments the partial file separately. The crash happens during separate defragmenting, not defragmenting the whole file.

Jie

NI R&D.

[Blueprint]

Hi Jie,

Thanks for the explanation! It is always good to understand why crashes happen 🙂

Do you think the fix will make it to the february patch?

Thanks again for the good help!

Øyvind

[Hooovahh]

Very cool information thank you so much for your insite.  I do think a CAR should be filed for this now that a cause has been identified.  If performing a defrag, and it is defragging in sections (meaning the file is larger than a the threshold) then cause it to not crash on defrag.  Maybe the easy solution is to see if there are channels without data, and if there are then to process the whole file once which I assume is the slower method of defrag but won't cause a crash.

[emmazheng]

I don't think the fix will be in patch for LabVIEW 2015. .

[emmazheng]

Hi  Hooovahh,

Thanks for your reply.

We do create a CAR recording what we just discussed.

Jie

NI R&D

[Hooovahh]

---

@emmazheng wrote:

We do create a CAR recording what we just discussed.

---

That's great...do you mind telling us what it is?

[sassyaspy]

Hey all,

The CAR created from this thread is number 569567.