TDMS functions much slower in CVI 2010

[MarcelK]

Hello everyone.

Today I noticed that at least some TDMS functions are much slower in CVI 2010 compared to CVI 2009 SP1 and prior. I have created and attached a simple sample project that creates a TDMS file with about 3000 file level properties and tries to read it back in afterwards. On all releases prior to 2010, this needs less than 10 seconds. On 2010 it's around an hour, if not more! Unfortunateky this is pretty much a show-stopper for me. Any comments?

Thanks, Marcel 

[nickb]

Hello Marcel -

What you've reported is actually a known issue, and is unfortunately considered to be expected behavior.  Let me try to explain:

There was a relatively large refactoring of the underlying TDMS code in LabWindows/CVI 2010.  This refactoring was intended to more closely align our internal implementation to that of LabVIEW.  As a result of this refactoring, we were able to address some internal issues we had previously been unable to address, as well as more correctly handle the data stored in the TDMS file.  Unfortunately, this refactoring unmasked a performance issue that had always been present when reading a large number of properties one at a time.

This performance issue was not uncovered for LabWindows/CVI 2010 because we had previously focused our performance testing on reading and writing data to a file, not metadata.  We considered it unlikely that a customer would have more than dozens of properties for any one channel or group or file, and as a result, the performance issues you've reported were overlooked.

However, we did recently find the performance issues you've reported.  As a result, there will be a handful of undocumented functions for returning all (or a subset of) properties on a channel, group, or file in LabWindows/CVI 2010 SP1.  This will allow for performance in line with what you'd seen in LabWindows/CVI 2009 SP1 and earlier, as long as you are OK with grabbing all the properties at once.  These functions are undocumented because, in general, we don't release new features with service packs.  Also, the functions are a little more difficult to use than normal CVI APIs, so we have not yet determined how or when they will be publicly documented.  When LabWindows/CVI 2010 SP1 releases (later this summer), feel free to reply back to this post or send me a private message, and I'll work with you on the details of calling these undocumented functions.

Out of curiosity, we'd like to know your use case for creating that many properties.  You're the first customer we've encountered using such a large number of properties, and we'd like to ensure that we are able to satisfy your use case in future versions of the API.

Thanks for the report, and I'm sorry for any inconvenience this has caused,

NickB

National Instruments

[MarcelK]

Hi Nick!

Thank you for your fast and thorough response. Though it's not quite what I wanted to hear...

We're currently in the process of adopting TDMS company wide as our standard test result format for automated tests (mostly TestStand sequences). We've got a wide range of very different products (automation industry) and thus the tests also vary very much. Some collect mostly mass data like voltage or pressure curves, which are then stored in channels, others collect more discrete data points, which are stored in properties. All this is pretty much up to the designer of the test script. In this particular case it was a test report that included some 3000 discrete data points, for whatever reasons. I only noticed it because my test execution/report viewer application automatically sends a memory dump to me when it hangs for a certain amount of time, in this case in the TDMS loader. Fortunately we've not yet updated many machines to 2010...

This in mind, the data types of the properties are usually not uniform (well, 2980 of the 3000 mentioned above were, but others not). Can your secret mass-property API still help in this case?

I must say, considering that we're betting the house and everything in it on TDMS here, I'm really on the verge of rolling my own code in order to not experience any more surprises and remove some other annoyances, like the inability to read back the channel list before the file is saved to disc, which due to my pretty abstract API layer above the TDMS library I have to do very often. 😞

Still, thanks a lot for your efforts. Cheers, Marcel

[nickb]

The API will be able to handle your case.  The function will return to you 3 associated arrays, the first containing the name of each property, the second containing the datatype of each property, and the third containing the value each property.  Then, for a given property name, you just need to find the same index in the datatype array to know how the same index in the values array should be cast.  The level of usage complexity (relative to more conventional CVI APIs) is the primary reason we've not made a decision on when to officially release a public version of these functions.

Out of curiosity, when you mention the "inability to read back the channel list before the file is saved to disc", do you mean that after you have created the file and written some things to it, you are unable to query aspects of the file because you get back error TDMS_RequestCannotBeCompletedWithUnsavedData (-6618)?  If so, it coincidentally happens that this limitation was removed as a result of the refactoring I mentioned earlier.  You can now (as of CVI 2010) write to and then query an in-memory file, before you save it to disk.

We're very interested in ensuring you have success using the TDMS API - please don't hesitate to let us know if you run into any other annoyances/problems -

NickB

National Instruments

[YongqingYe]

Hi MarcelK, 

Here is the latest TDMS installer on web:

http://zone.ni.com/devzone/cda/tut/p/id/9995

Please try with the latest TDMS installer to see whether the problem is solved or not. Currently I also couldn't make sure whether this can work or not, but at least deserve we trying it.

Please let me know the result. 

Thank you!

Yongqing Ye

NI R&D

[MarcelK]

@Nick: the new API sounds good to me, no matter how difficult to use. Even if I have to push the bits one by one to the stack it's still better than code that needs an hour to run 😉 I'll probably have to support both versions dynamically, while we're still supporting older RTEs, but that I can manage. Yes, I think it was the TDMS_RequestCannotBeCompletedWithUnsavedData error, *very* good to know that this limitation is removed! Given this I'm actually starting to look forward to the SP1 release, if there's ever a pre-release, don't hesitate to send it my way 😉

@Yongqing Ye: 

Downloaded the file at the given URL and got

"Cannot install
- TDM Streaming (higher version already installed)"

All the best, Marcel