Make TDMS functions robust!

[TiTou]

TDMS could be very useful if they were not so badly handled. The need for files that you can read and write concurrently does exist but TDMS fail to address it right now because of the lack of stability.

Create a TDMS, close the ref and then pass the closed ref to any TDMS function, LabVIEW crashes instantly...

Try use NI's TDMS viewer for a very large file, you get a memory error.

For now I still use WAV file and when customer send me their large TDMS files I always have to struggle around to parse them.

Really TDMS could be cool with a little effort from NI.

[TiTou]

typing mistake in the idea description... I meant TDMS, not TMDS, sorry.

[JeffBoettcher]

Thank you for the feedback.  We take TDMS stability & robustness very seriously.  You mention two specific issues:

The crash is very concerning, but from your description I was unable to reproduce this.  Do you have a VI that demonstrates this?  Also, which version of LabVIEW and the TDMS DLL ([Program Files]\National Instruments\Shared\TDMS - right-click the DLL and view details to view the version) are you using?

As for the memory issue using the TDMS File Viewer, you're right, it is a relatively simple viewer VI to address basic use-cases.  For more full featured data analysis, you could modify the tool or write your own viewer to read data from large files in smaller segments, perhaps adding decimation of data displayed.  Another option might be to try a product like DIAdem which is designed for analysis and display of especially large data sets.

If you have had any other specific experiences that make you feel like TDMS is not robust, please let us know.

[TiTou]

I'm (still) running LV8.6, my TDMS.dll is version 2.2.1546.609 and the attached VI systematically crashes LabVIEW when I run it, no message, nothing, LabVIEW just vanishes. The TDMS file is created and not corrupted though.

(for some reason I can't upload the screenshot of the VI)

I don't have an example of a big TDMS file that I can share (to remain polite I have to keep to myself what I think about NDAs), but just generate one that's big (~a few Gbyte) and read it with NI's TDMS viewer I often get memory error, and to work around those errors with the (too few) TDMS primitives is not handy.

Since I'm in the complaint department, let me also complain about the fact that the TDMS API is Windows only, that makes me sad as well.

Oh, and if I have time, I'll look for a little VI I've made, basically it does with TDMS what OpenG "Read / Write section cluster" VI do for INI files ; that's something I think should come as a primitive.

[G-Money]

TiTou,

          Sounds like you might have run into a bug (closing the TDMS ref and then LV crashes) and a limitation (TDMS Viewer). Did you have any more specific feedback on functionality we could add to the TDMS API? So far I read your post as some issues we could track as bugs but not really talk to R&D about adding a new feature.

[TiTou]

True G-Money, the first part should be a bug report...

Oh, by the way another bug report (I'll go through the full bug report one day, maybe), using the "TDMS deframent" primitive with very large files will crash LabVIEW.

Now just above are two feature requests :

- multiplatform

- more primitive to read/write information about a channel (config cluster of the channel for instance)

And on a general note make it easier to load data by chunks of pre-defined size.

[TiTou]

couldn't post image with firefox 3.6 or 4b11... had to use IE 😮

[YongqingYe]

Hi Titou,

Thank you for your feedback, we'd like to take a look into the problems you reported. 

First of all, I tried the VI in LV 8.6 and 8.6.1 with the dll version 2.2.1546.609 on my machine, there was no crash and LV returned the error code -2501 as expected. So would you please double check and do you have any new version of LV installed on your machine, if so, could you please try with it?

And for TDMS File Viewer, if you navigating in the tree in the left part, it only loads the .tdms_index file and when you viewing the data values in the "Value (table)" tab in the right part, it needs to load the .tdms file which is maybe a huge file. Technically, TDMS File Viewer only loads the first 1,000 values of the channel and you can configure it by clicking the button "Settings..." in the left bottom part, but if you want to load many data values, it's possible for LabVIEW running out of memory, is that your case?

And you also mentioned a new feature:

- more primitive to read/write information about a channel (config cluster of the channel for instance)

What do you mean for this? Do you mean supporting writing cluster or setting cluster properties for a channel or something else? And any more detail information?

Thank you again!

Yongqing Ye

NI R&D

[YongqingYe]

Hi Titou,

Another option you can try for the crash of passing a closed refnum to TDMS is, maybe you can try with the latest tdms.dll, that would help us to identify where the problem is.

The latest TDMS installer is here:

http://zone.ni.com/devzone/cda/tut/p/id/9995

Would you please let us know the result of running it with LabVIEW 8.6 for this problem?

You also mentioned TDMS defragment would also crash LabVIEW when handing with large files, so how big the file is in your case? And are you sure it's a crash of LabVIEW, since we noticed that in some cases with big files, defragment would cost quite a long time, like several minutes?

Thank you,

Yongqing Ye

NI R&D

[TiTou]

YongqingYe,

LabVIEW 8.6.1 is the most recent version of LabVIEW on my computer, I do have runtime engines of more recent versions though (2009 and 2010).

On a computer dedicated to a specific system with only a clean install of only LV 8.6.1 and nothing else I have the same issue.

Then I tried on a co-worker's computer who has LV 8.6, same TDMS.dll version, no runtime engines newer than 8.6, when he runs my VI, LabVIEW doesn't vanish away, but just hangs until we kill it (after waiting 5 minutes during which even opening the windows explorer is not possible).

more primitive to read/write information about a channel (config cluster of the channel for instance)

What do you mean for this? Do you mean supporting writing cluster or setting cluster properties for a channel or something else? And any more detail information?

At the moment the TDMS set property primitive has a variant input for property value, but it will only accept a very restricted list of data type (cluster won't be supported), I was very disappointed when I saw this error : LabVIEW does not support properties of the specified data type after flattening my cluster to variant and giving it to the "property value input.

I'm not telling you how you should implement it, you probably know better than me the best way, I'm just expressing my need.

So here's what I wish I could do :

And here's what I had to do :

with one case for each data type... I just show you one more and spare you the list.

And same for writing properties.

I'm now going to get the latest dll version and let you know what happens.

[TiTou]

Following your link I could be sure that TDMS 2.0 would work with LV 8.6... Would it work?

Also I saw this article : http://zone.ni.com/devzone/cda/tut/p/id/3727

When was the Advanced TDMS palette introduced? I've never seen it, does it come with V2?