Can't keep up with data acquisition on timed data collection

[aolima]

Hi,

I'm trying to develop a system to collected timed data from multiple channels and modules (9215,9237, and 9213). The limiting sampling rate here is the 9213 which has a 100S/s maximum rate.

I only need to record one data point every 5 minutes (and preferably have that be the mean of a few hundred samples given noise).

Unfortunately I'm running into issues with keeping up with hardware acquisition that I cannot seem to solve. I've considered using the Configure Logging vi but couldn't figure out how to have that be timed to the 5 min intervals i need. I'm now considering the produced-consumer architecture but wanted to ask here if that is actually my best option before trying to build it out.

See attached my current VI.

Thank you in advance,

[ZYOng]

The Elapsed Time Express VI blocks the execution of DAQmx Read VI, causing data to not be read from the buffer in time, resulting in a buffer overflow error. I would recommend using the FGV Timer SubVI instead. Keep the DAQmx Read VI running non-stop and only log the data when the FGV Timer elapses.

[Yamaeda]

You're setting the sample buffer very tight. You're using a buffer of 100 samples and read 100 at a time, leaving no room for wiggle. Make it atleast the double size.

[GerdW]

Hi,

---

@Yamaeda wrote:

You're setting the sample buffer very tight. You're using a buffer of 100 samples and read 100 at a time, leaving no room for wiggle. Make it atleast the double size.

---

To expand on that suggestion: read the LabVIEW help for the DAQmxTiming function, especially the explanations for the "Continuous sampling" mode. After doing so you should connect NOTHING to the buffer size input as DAQmx does most often a very good job on setting a reasonable buffer size!

Other problems:

• Most often the case structure inside the loop will kill the TDMS file reference! Never use "unwired if default" tunnels for (any kind of) references!
• Why do you need an ElapsedTime ExpressVI with a target time of 1E17 s??? Why not use two GetDateTime functions (before and in the loop) and a subtract function to determine the elapsed time?
• You should also implement error handling for those file functions…
• Boolean functions also accept error wires…
• Are you sure you want to log median values when your requirements asks for mean values?
• I recommend the FormatIntoString function to create your filename, see image… (In your specific case you could also change the format string for the FormatDateTimeString function to get rid of the ConcatString function.)

[aolima]

Thanks you for the thoughts and recommendations,

I've implemented most of them in the attached code in which I also decided to implement the Producer-Consumer architecture which now seems to run without crashing.

However, the new issue I'm running into is that the VI will sometimes stop adding more data to the file so I have to stop and restart the VI. This happens without any consistency, sometimes file will stop updating with a few hours and sometimes it will take days for it to happen.

Any thoughts on what could be causing this issue?

[GerdW]

Hi aolima,

why do you need 8 SplitArray nodes when you could do the same with just one IndexArray function???

---

@aolima wrote:

However, the new issue I'm running into is that the VI will sometimes stop adding more data to the file so I have to stop and restart the VI. This happens without any consistency, sometimes file will stop updating with a few hours and sometimes it will take days for it to happen.

---

Did you debug your VI in such cases? (Using probes and/or additional indicators on your frontpanel?)

Are there any errors occuring?

Your charts are rather small (in pixel size), but they are set to a history length of 1024. So they will display 1024 waveforms of 100 samples each = 102400 samples! Does it make sense to you to plot 100k samples on a chart area of just 200 pixels width? (For me it doesn't make any sense!)

All you do here is wasting memory and forcing LabVIEW to shuffle data in memory…

[GRCK5000]

Hi Aolima,

I've modified your code a little bit. Give it a try and let us know. (See attached code).

On your front panel, make sure you change save data every 5 minutes as shown below (the circled control) because you want collect a data point every 5 minutes. You can change it to whatever time you would like.

[aolima]

Thank you for the modifications...

Code ran well for 48hrs until the system ran out of memory... so not sure if it still if using too much buffer or other memory? I've tried removing the graphs per the recommendation given here also and see if that help with the issue.

See error below:

[crossrulz]

3 major issues I see with the code.

1. The top loop is using a channel wire to control its state.  That is adding a ton of overhead when it should just be a simple Shift Register.

2. The cascading Split 1D Array in the bottom loop can be replaced with a single Index Array.  That will eliminate several additional arrays being created.

3. You are constantly adding to your queue, but only occasionally taking data off of the queue.  This is causing multiple issues.  a) The queue is ever increasing in memory usage (the main issue you noticed) and b) you are dequeuing old data.  You should move your timing to the producer loop so that it only enqueues the data every X amount of time.  The consumer can then be a very simple loop where it just has the dequeue and the processing.

I implemented the first two for you.  I don't have the time to fix #3 for you.

[GRCK5000]

@crossrulz Thanks. You are the man! I learned something, and how you cleaned up the code would put a smile on any labVIEW programmers. Simple code and easy to understand.