Supress storing .tdms_index files (useful when generating logs on cRIO and other SSD devices)

[QFang]

When using TDMS on cRIO systems, there are a couple of considerations that doesn't normally play in too much when storing data as TDMS files and they are:

* The current version of the file system used on cRIO controllers degrades significantly in performance if -any- folder on the cRIO contains more than ~100 files. (work-around> more elaborate folder structures, but a lot of this structuring would be only to work around this shortcoming of the (old) version of the file system 

* The drives are SSD with limited life-length and wear-leveling etc. Writing and re-writing these index files add un-necessary overhead and wear on the disks

* They use up space which is (very) limited on some cRIO's (even if not much). (people may be quick to point out that you can add a thumb-drive, but down-sides to that is the thumbdrives (as far as I know) needs to be FAT. Compared to storing on the cRIO file system which is atomic and fail-safe where you pretty much don't have to worry about sudden power outages and interruptions mid-write.. on a thumb drive you would have all these issues that could worst case corrupt your whole thumb drive.)

I propose to add a boolean (default to false) on the TDMS Open called "supress writing tdms index to disk" or some smart name along those lines. What this would do is still allow for the tmds index to be created, but it will remain in memory only and never be written to disk. When the TDMS Close is called, the memory is released and the tdms file is written to disk without the index file. If the same file is opened again, extra time would be needed since the index file would be re-created (again in memory only if boolean indicates this), but I think for the most part this overhead would be more than acceptable.

I'm not sure how "simple" modifying the TDMS open and close functions would be, but I do know that there are many cases where this flag would make sense. 

[QFang]

Something like this 

[YongqingYe]

Thank you QFang, for your post of TDMS idea!

The purpose of TDMS index file is to help quickly load TDMS file when calling TDMS Open. In quite a lof of use cases, the .tdms file contains raw data and is much larger than the .tdms_index file. So if the index file existing when opening the file, we will just load the inde file, it would be much faster than load the entire .tdms file. 

However, I can understand the siutation on RT, especially when the disk size is very limited. 

Thank you again!

[QFang]

Yongqing Ye, yes, I absolutely agree the index file makes perfect sense in almost all use-cases, however, on the RT system, I'd be willing to trade off access speed when opening, for less file clutter on the disk. The index file would still "exist", just as an "in-memory only" instance, so writing to the tdms file itself should still be fast, just the initial open would be a little slower.

Thanks for the comments! 

[ba_0101]

Check the TDMS Advanced Open VI. http://zone.ni.com/reference/en-XX/help/371361J-01/glang/tdms_adv_open/

"Opens a .tdms file for reading or writing in the byte order of the host computer. You also can use this function to create a new file or replace an existing file. Unlike the TDMS Open function, the TDMS Advanced Open function does not create a .tdms_index file."

[QFang]

Many TDMS advanced functions are not supported on RT targets.  

nonsupported advanced TDMS functions

[deppSu]

Hi QFang,

The TDMS group has put this feature into implementation in LV2013. Welcome to have a try!

[QFang]

Hi deppSu,  indeed they have! Thank you TDMS Group.. and thanks for telling me, I don't know when I would have noticed if you didn't! 

-Perhaps this idea should be marked/moved as implemented?

I'll do some testing later, but from the documentation this is perfect!

[deppSu]

Yes, this idea is not "new" anymore and its status should be updated.