3 PXI-6115 cards, daqmx compression, win32 streaming to disk

[biographie]

I am hoping to get some insight from people that have done all three of the following things, and integrating them together.  I have included a zip of the project folder.

I have an old PXI system with an embedded controller that only has a Pentium 3, and 512 of RAM.  I am trying to run the system as quickly as possible to get the data rate and streaming I need.  My current needs are strictly what I am doing now, but this program is suppose to expand in the future.

I have three PXI-6115 cards with 32MS buffers on each card.  I am setting up a single task using all 12 channels across the 3 cards, and using PFI0 on the first card as my TTL trigger line.  Sample rate of 1MS, and to acquire 30,000 points per channel with 10,000 of those points in a pre-trigger setup.  And to complicate the situation the data I am wanting is pulsed, so I need to constantly stop and start the task to retrigger the setup.

I have reviewed many examples on the NI website:  rf_stream_to_disk_resource_kit, 8_0_compressed_streaming, etc.  I also have add the code to take advantage of the lossless compression feature to cut the data amount.  However this cause some unknowns for me.  I have tried to adapt the VIs from the compressed_streaming example for the 6115 cards.  That example states it was designed for U8, but  I am wanting to use raw data I16.  What exactly needs to change in those VIs for I16 usage?  I have made notes in parts of the code to give some insight into what I think needed to be done.  Please tell me if I am on the right track, and what I have messed up.  I have made no changes in the decompress VIs yet.

To make matters more complicated for myself, I am trying to incorporate the non-buffered win32 operations for streaming to disk.  Because of this I am having to manage the data before I can write it to disk.  Again, I tried to make notes on what I think I need to do to pull this off.

I hope this is making sense, and I appreciate any changes, comments, and questions you may have to help me.  I hope providing everyone with the code with make more sense than my rambling.  Once I have this part solved, I will need to make the modification to write the timestamps into the binary as well, and get the decompress read VIs working as well with the timestamps.

[samantham]

Hello,

Are you getting an error messages and if so what are the code numbers?

You mentioned that you wanted to change the U8 to I16 in the compression VIs.  However, the only compression VI I saw in the block diagram was to compress the header and this VI does not output any U8 representation values.  Please clarify where you would like to make some changes.  I know of the Developer Zone titled Data Compaction for High-Speed Streaming to Disk Examples.  Please verify that these are the example VIs you want to change.  At most, I saw that the Win32 Open file VI (from the posted code) uses U32 as an output.  I failed to notice your “notes in parts of the code to give some insight into what I think needed to be done”.

The notes I did see in-regard to lossless compression where very detailed.  However, the default for compress in None.  Please review the following link titled Data Compaction for High-Speed Streaming to Disk on how to setup lossless compress with a property node. There are also help files in the Labview Help.  Please search AI.RawDataCompressionType in the Search tab.

Data Compaction for High-Speed Streaming to Disk

Data Compaction for High-Speed Streaming to Disk Examples

 

[lukko]

Hi Samantha,

I have tried to play with AI.RawDataCompressionType and AI.LossyLSBRemoval.CompressedSampleSize according to the example you mentioned above?

The issue is that I do not want to log data into binary files using Raw 1D I32. Instead of that I want to save Unscaled 2D I32 array in TDMS file. Is it possible to use lossless (24bits from PXI-4472 in I32 to TDMS) or lossy (20bits from PXI-4472 in I32 to TDMS) compression supported by DAQmx?

Afterwards I would like to open TDMS files directly in Matlab using Dominonilibddc (nilibddc.dll). Also I like the way how data are stored in TDMS files and it can be supported by more application than binary files.

Thanks!

--

Lukasz

[AndrewMc]

Hey Lukasz,

You might have seen my mention of the TDMS logging feature that was added in DAQmx 9.0 on this thread (http://forums.ni.com/t5/LabVIEW/Why-the-TDMS-file-is-larger-than-it-should-be/m-p/836518).  Note that this feature will skip Windows buffering and even use an advanced feature of the OS for overlapped (or asynchronous) file I/O.  It is the fastest way possible to stream to disk in DAQmx.

Additionally, lossless compression is supported in this feature.  That is, if your device supports hardware compression (24-bits represented in 3 bytes) and you enable lossless compression, only 3 bytes will be written to disk per sample.

http://zone.ni.com/devzone/cda/tut/p/id/9574

Let me know if you have any questions on this feature.

[lukko]

Hi Andy,

It seems that my quite old DSA 4472 does not support hardware compression. I bought it because it has 8 channels and simply I prefer to have 2 times 8-channel DSA instead of newer 449x with 16 channels. Now I can build two GPS-synchronized measurement systems with 8-channels DAQs but unfortunately without hardware compression.

Is there another way to apply lossless and lossy compression (sometimes my sensors sensitivity is bigger than LSB) for measurements using PXI-4472? I would like also avoid binary formats because measurements will be available for many users.

Thanks and best regards,

Lukasz Kocewiak

[AndrewMc]

Hey Lukasz,

For lack of hardware compression, you could enable software compression instead by setting attributes for AI.RawDataCompressionType and AI.LossyLSBRemoval.CompressedSampSize.

Note, however, that with the logging feature, software compression is not supported.  One of the main reasons for this is that it would add a buffer to that logging operation, which would greatly reduce overall available system bandwidth (by stressing the memory controller); the extra cost for the logging operation far outweighs the byte per sample that you would save (unless of course, file size is your only concern as opposed to streaming performance).

As to your concern about binary file formats, note that disk footprint and performance will suffer if you don't use binary.  Additionally, something that greatly helps TDMS is the TDM Excel Add-In, an easy way to open .tdms files in Excel.

[lukko]

Hi Andy,

And this was my first thought. As I observed my portable measurement set-up can handle long-term multichannel data acquisition with software compression enabled.

When I log raw data using DAQmx Read and TDMS Write Function with 1D I32 representation and software compression enabled is not so straight forward task to read TDMS files afterwards using TDMS Viewer.

If I log data without compression I can use Decimate 1D Array and afterwards save it using TDMS API but in this case there is no difference with Unscaled 2D I32. When I use directly Unscaled 2D I32 there is no difference in files content and sizes but in case of Raw data it seems to work and lossy file (20 CompressedSampSize) is almost 37% smaller than without compression and lossless file is almost 25% smaller than without compression. This would help me to save a lot of disk space.

Is it possible to save data in TDMS files with lossy or lossless software compression and read it directly in TDMS Viewer or Matlab afterwards without any binary data management algorithms?

Best regards,

Lukasz

[AndrewMc]

Hey Lucasz,

Many of the DAQmx scaling methods that are used in the DAQmx TDMS logging case are exposed in the standard TDMS API as well via the "Create Scaling Information" VI (Linear, Polynomial, etc).  However, there would not be a good way to express some of the complex binary data manipulation that happens with software data compression.

That being said:

1) Here are examples showing how to use software data compression and subsequently do that binary manipulation: http://sine.ni.com/devzone/cda/epd/p/id/4828

2) If you did not use software data compression, you could leverage the "Create Scaling Information" VI to store the scaling information.  For example, if your DSA device has a linear calibration of 9.5 (for example), you could describe that with the VI such that the scaling will automatically be applied whenever you read the file.

[lukko]

Hello Andy,

Thank you for your fast response.

In this case I will stay with TDMS API and Producer/Consumer solutions for long-term one-minute file logging measurement run-time software.

I also create linear scales and it work pretty nice. I am quite sad regarding the software lossy and lossless compression support in TDMS API, but I will just buy few more TB-capacity HDDs.

Right now I am not going to use binary files for compacted streaming purposes. Mainly due to lack of support for external applications. When I have lots of TBs I do not want to convert it in LabVIEW, save new files and afterwards process in third-party software.

Thanks!

--

Lukasz Kocewiak