Decrease size of TDMS

[SinisterMJ]

I am using the NI DLLs to save as TDM files, and my main issue is that it basically just writes the raw data on the disk, without any sort of compression.

My code:

struct ScopeData {
 std::vector<__int16> channel1;
 std::vector<__int16> channel2;
 int byteDepth;
 double maxMVCh1;
 double maxMVCh2;
 double maxMVCh3;
 double maxMVCh4;
};

void WriteNIFile(Scope::ScopeData* sdDataNI,
        std::string fileName,
        std::string filePath) {
        int	ddcError = 0;
	    DDCFileHandle file = 0;
	    DDCChannelGroupHandle group;
	    DDCChannelHandle channels[2];
        if (!DirectoryExists(filePath.c_str())) {
            CreateDirectory(filePath.c_str(), NULL);
        }
        std::string saveFile = filePath + "\\" + fileName;
        ddcError = DDC_CreateFile(saveFile.c_str(),
            "TDM",
            "OphthaMetrics data",
            "OphthaMetrics data acquisition",
            "",
            "",
            &file);
        ddcError = DDC_AddChannelGroup (file, "RAW Data",
            "Raw scope data from two channels", &group);
        ddcError = DDC_AddChannel (group, DDC_Int16, "Channel 1", 
            "Transducer to preamp", "Millivolts", &channels[0]);
        ddcError = DDC_AddChannel (group, DDC_Int16, "Channel 2", 
            "Transducer to preamp", "Millivolts", &channels[1]);
        ddcError = DDC_SetChannelProperty(channels[0],
            DDC_CHANNEL_MAXIMUM, sdDataNI->maxMVCh1);
        ddcError = DDC_SetChannelProperty(channels[0],
            DDC_CHANNEL_MINIMUM, -sdDataNI->maxMVCh1);
        ddcError = DDC_SetChannelProperty(channels[1],
            DDC_CHANNEL_MAXIMUM, sdDataNI->maxMVCh2);
        ddcError = DDC_SetChannelProperty(channels[1],
            DDC_CHANNEL_MINIMUM, -sdDataNI->maxMVCh2);
        ddcError = DDC_SetDataValues(channels[0], &sdDataNI->channel1[0],
            sdDataNI->channel1.size());
        ddcError = DDC_SetDataValues(channels[1], &sdDataNI->channel2[0],
            sdDataNI->channel2.size());
        ddcError = DDC_SaveFile(file);
        ddcError = DDC_CloseFile(file);
        return;
    }

Only channel 1 and 2 are being used in our tests, which is why only 2 are saved. The DLL saves them just subsequently I guess, and this results in 500 MB files (each channel is around 136 million 16-bit unsigned integers, resulting in a combined 500 MB). When I use zip on the files, it will decrease drastically in size (from 511 MB to 144 MB - which is expected, only 12 bits are set, the dataset is a lot of values ranging from -100 to +100, and only occassionally going for outside that range, etc). 

How can I decrease the data streams in size, but still have the files usable in LabView or similar tools? Can I use zip compression on the data before saving it?

[Stephan_D]

Hey SinisterMJ,

The TDMS file format doesn't provide a zip-like compression, as far as I know. Data are stored uncompressed in the channels. TDMS doesn't compress them, because this would reduce the throughput on data saved to disk, which conflicts with the purpose of highspeed streaming to disk (the S (Streaming) in the TDMS abbreviation).

It would be possible to compress the whole TDMS-file to a ZIP-file, but you need to decompress the files every time you want to work with them.

Regards, Stephan

[Hooovahh]

The data as it is stored in a TDMS is going to be stored in the same way the data was stored in memory.  It will likely be the most compressed it can be, with the exception of the header information.  If you zip a TDMS file and it gets small you probably are compressing the header information, and that is a sign that your TDMS file is fragmented.  Perform a TDMS defrag and see if the file size goes down.  If it does your file is fragmented and is filled with duplicate header information.  Another sign of this is having a index file that is about the same size as your data file.  Here's a few links on how to avoid fragmented files.

http://www.wiresmithtech.com/tdms-fragmentation-cslug/

https://decibel.ni.com/content/docs/DOC-20522

https://www.dmcinfo.com/latest-thinking/blog/id/205/labview-data-storage-tdms-performance-tweaking

Basically avoid writing single point data.  Write N samples and N channels calling the TDMS Write function as few times as possible.  Create your own internal buffer, or use the minimum buffer size property.  Or close defrag and re-open the file periodically.  Oh and in 2015 defrag is faster, and there is a way to get a progress bar of the defrag process.