Three level TDMS hiearchy

[CarstenPXI]

We use TDMS extensively for large analysis tasks (multi Gbyte) files, and find the current two levels very restrictive, at minimum would like to see an one more level than Group and Channel. Without this, storing series of spectra requires overloading the current levels, or the Channel concept gets bastardized, by creating a new channel for each spectrum. Carsten Thomsen

[YongqingYe]

Hi Carsten Thomsen,

You are not the first guy who requesting this before 🙂 But make this in TDMS will be a big change. 

What we want to understand is more details about your request, like how many levels you want? like 4, 5 or even more, or just a customized number? Another question is now TDMS has 3 levels and users can store data only on channel level, do you also want to store raw data on other levels if TDMS has more levels?

Thank you.

[CarstenPXI]

Currently we are overloading the three level concept, (which in reality is only two level), it only is three levels with respect to properties.

In an application that we just finished, we had a need for four levels:  Patient, Measurement Channel, Analysis Type, Post Processing Results.  You may argure that why don't we just have different TDMS files for different patients, but since this is a research application where we are doing batch processing on large data sets, TDMS gives great large file performance.

Otherwise our disk would be filled with hundreds of separate files.

In practice, we put the Patient and Measurement Channel  info into an overloaded name into the TDMS Group, and the Analysis Type and Postprocessing overloaded into the TDMS Channel. When we write and read the data to TDMS we then have to encode/decode the Group/Channel names to extract additional "virtual" levels i.e.

TDMS Group

     Patient_MeasCh

TMDS Channel

     Analysis_PostProcessing

Some of the measurements are "array type measurements", i.e. FFT spectra,  which are stored as time slices.

This again complicates the data structure, because if we feed a series of spectra directly to TDMS it automatically creates a series of Untitled Channels.  This points tothe need for TDMS to have a mechanism that can append array type measements as easily and beautifully as TDMS appends time domain signals.  Currently we do this with another "hack", where the spectra are converted to arrays, and these arrrays are then streamed to TDMS as if they were time domain data.  To decode then we have to read a "key" which is stored as a TDMS property, telling us block size and df.  Not pretty, and it ruins interoperability with Diadem.

As absolute minimum we would like to see one additional level in TDMS.

We would also like the ability to have Spectra type data be a native data type in TDMS we can be appended without creating news TDMS Channels.

Anything beyond that is "nice to have", and would certainly be appreciated.  A user-defined number of levels would be wonderful, but I assume it may have a performance impact, which is OK if it is not more than 10-30% or so.   Also impact on file defragging should also be taken into account.

Storing data at multiple levels can also be very useful for our applications.   Again, here our desired for expanding the TDMS data types to handle other types than time domain data is important.

Hope this makes our needs more specific.

Carsten Thomsen

[YongqingYe]

Thank you for sharing the ideas!

[Tamarack]

I'd like to throw in my vote for a feature like this.  I was working on fleshing out a similar file structure and an extra "layer" of TDMS formatting would be very helpful.  For another example of how I'd like it formatted, I'm trying to create a file the defines the motion a custom CNC machine must follow to wind a composite part.  Each file would define a single part (first layer) and would have certain global part specific attributes like raw material type, tape size, part ID, etc.  Each part would be comprised of any number of plies (second layer, each with a few attributes for documentation's sake like ply wind angle and offset) and each ply would consist of a number of strips (third layer).  Finally, each strip is broken into a sequence of segments (theoretical fourth layer).  The data would be stored on the segment level although the ability to store data on the strip level would be handy.

[Darren]

Any idea that has received less than 9 kudos within 9 years after posting will be automatically declined.