Request for Assistance on Data Acquisition Configuration to Prevent Overwriting

[jangjational@kriss.re.kr]

I am currently engaged in a project that requires the long-term storage of substantial data volumes from various sensors, and I am encountering challenges in avoiding data overwriting. Your guidance on this matter would be immensely valuable.

Project Overview:

The system must store large data volumes over extended periods.

AE sensors, requiring a minimum of 100kHz sampling rate, will be connected to a USB-2020 device(mcc, Measurement Computing Corp.), with one sensor per two of the four channels.

Vibration acceleration sensors, requiring a minimum of 5kHz sampling rate, will be connected to a USB-1808X device, with one sensor per two of the eight channels.

The aim is to construct a system capable of storing data from all four sensors for at least one month. The buffer size for each sensor has been determined by multiplying the sampling rate by 80.

Challenges Encountered:

First Method Attempt: I specified the number of samples per channel as -1 with a wait time of 200ms to periodically empty the buffer. This resulted in data overwriting shortly after commencement.

Second Method Attempt: Without a wait period and setting the number of samples per channel to one-tenth of the target sampling frequency, overwriting still occurred.

Inquiry:

Are the USB-2020 and USB-1808X devices capable of supporting these specifications without data loss over a minimum of ten days?

How should I configure the buffer size, number of samples per channel, and wait times to effectively manage data without exceeding a total memory usage of 5TB for the ten-day period?

Given the overwriting challenges (error:10017) faced with both methods attempted, could you recommend an approach that either reduces the likelihood of real-time processing issues or suggests an optimal setup that might involve longer wait times but ensures data integrity?

I plan to attach VI files pertinent to each method attempted for your review.

Your recommendations on configuring our setup to meet these requirements and any advice on alternative strategies would be greatly appreciated. I understand the complexities involved but am hopeful for a feasible solution.

Thank you very much for your time and support. I look forward to your expert advice.

Best regards,

[Kevin_Price]

My initial advice:

1. Consider using NI's devices and DAQmx instead of MCC and ULx.   I don't have experience with MCC devices and drivers, but from threads over the years I have developed significant suspicions that their lower cost carries with it some significant limitations.

    I'd personally also gravitate *away* from USB-based DAQ devices and toward desktop PCIe devices (unless you already have PXI available as an option).

2. If you move to DAQmx, you could potentially make use of TDMS logging.  However, as you start to get to TB+ sized files, I'm not so sure about that any more.  Not saying it won't work, just that I simply don't know.

3. I'd be inclined to set up a producer/consumer structure where you can have a file writing loop that starts a new file periodically.  As a starting point, you've got a data rate of (8 bytes/DBL) * (210 kSamp/sec) = 1.68 MB/sec ~= 6 GB/hour.   In my experience, multi-GB files start getting unwieldy.

    There will be ways to compress that bandwidth if needed, but your order of magnitude is going to remain "really big."   Others around here probably have better specific tips for mid-speed long-duration storage.

-Kevin P