Buffer Overflow

[CCHUA1994]

Good day all,
Is there any solution to prevent the ERR-200279 a.k.a buffer overflow from happening? 
Basically reducing the sampling rate and number of samples is impossible due to research purpose need. 

So here the problem, is there any other method to prevent this error from happening? For example like increase the read rate? Or keep the data outside the while loop until the read inside the loop is completed and ready to read second round of data? 

[Kevin_Price]

I can't open the code here where I'm at LV 2016, but Yes, there are (usually) ways to prevent buffer overflow errors.

1. Yes, increasing the read rate will help.  More generally, it'll pay to think of read rate as the product (reads / sec) * (samples / read).  To avoid overflow, you'll need this sample retrieval rate to keep up with the device and driver that are delivering data to your task buffer according to you task's sample rate.

2. You can search for examples of the "producer - consumer" design pattern using queues (or channel wires).  This helps to make your read loop lean & mean for the fastest possible (reads / sec) iteration rate.  Here's one example.

3. Using a USB device will be more limiting than using a desktop or PXI device.  I don't have exact figures, but this won't usually be a problem unless your bandwidth is up into the MB/sec realm.

4. It may be safest to use DAQmx's built-in TDMS logging capability and then post-process the data after capture is done.

5. What device are you using?  What sample rate do you need?  How much total data will you be collecting?

-Kevin P

[CCHUA1994]

Good day Kevin, im currently using NI9234 with cDAQ 9171, sample rate is 51.2kS/s due to my frequency of interest is up to 10kHz. As the resolution of my frequency is 4Hz, so I set the number of samples at 12800 samples. 

I had attached a older version of VI.

[cbutcher]

Hi CCHUA,

Your requirements don't seem so onerous, and the hardware shouldn't have a problem (afaik).

I'd suggest you take a look at the Producer/Consumer architecture that Kevin already mentioned, but beyond that I think you might have some problems with your DAQmx configuration.

In the first version, you have a specified "number of samples per channel" but the control default is -1. Perhaps you set this to a different value, but be aware that the Read will return instantly, perhaps with an empty array (?) if there are no samples. This will have an effect on your downstream processing (spectra etc). In the second copy, you've removed this, so it will always be "-1".

You also don't set the number of samples, despite the default for your sampling mode control being finite. Again, perhaps you set this to continuous, or perhaps this is an issue with your VI.

You give a "sampling time" which you calculate from rate and number of samples, but you're not using this to control your Reads. Perhaps you want to setup finite sampling, and calculate instead the number of samples (per channel) required for a given time (N = rate * duration) then set this for your finite acquisition at DAQmx Timing. That will allow you to put an Is Task Done? node in your While loop to detect the end of the acquisition.

[Kevin_Price]

Excellent comments from cbutcher.  Adding a few words:

- in Finite sampling mode, the default value -1 for # samples to read causes a special behavior.  The call will block and wait for the entire buffer of finite samples to be filled before it returns with all the data at once.  You can see how this is used in the shipping example for finite voltage acquisition.

   This is *entirely* different from the behavior when the task is in Continuous sampling mode.  I've seen this cause an awful lot of confusion over the years and finally posted a little cry for help to the Data Acq Idea Exchange.

- When you configure for Finite sampling and don't wire a # samples, a default value of 1000 will be assumed.

-Kevin P

[EDIT:  I fully expect your system to be capable of keeping up with your 51.2 kHz sample rate.  Probably indefinitely, but surely over a time frame of seconds or less.]

[CCHUA1994]

Hi cbutcher,

I will try to describe my problem based on my understanding from the comment by Kevin and sir, please check if my understanding is correct. 

For DAQTiming, my sampling method will always be continuous sampling method as I want the DAQ to completely acquire the signal capture by the microphone, with the sampling rate at 51.2KS/s which is at the limit of NI9234. So since my sampling method is Continuous Mode, so I will not wire the samples per channel as it will be ignored at this mode, which is mentioned by Kevin. 

For DAQmxRead, i would required my DAQ to acquire number of samples per channel to be 51200 samples as I need at high frequency resolution to prove my calculation. 

So based on this configuration, DAQRead suppose will acquire a total of 51200 samples.

But here comes the problem, configuration like this will result the err-200279 in Labview as attached. 
Even if i reduce the number of samples in DAQRead to 12800, it still faced the same error. So do when i set the number of samples to be -1.

[CCHUA1994]

Hi, cbutcher

I will try to describe my issues based on my understanding after read Kevin post and your comment. 

For DAQ Timing

For my case, I used continuous sampling method with 51.2KS/s, which means the sample per channel is                insignificant as mentioned by Kevin in his post. So i will not wired the sample per channel. 

For DAQRead
As for DAQRead, I set the number of sample per channel to be 51200 Sample, this means that DAQ will attempt to read 51200 samples right? But here's the issues, the err-200279 will popped out. Even if I set the number of samples to be -1 to read available samples, it still give the same error. Or reduce the number of sample. 

I had attached an image for reference.

Regards,

CHAN

[cbutcher · Accepted solution]

You need to remove the Highlight Execution option.

Using Highlight Execution makes your program run much slower (of course) because you're trying to follow the wires etc - if it ran at normal speed, you'd usually not be able to see the values before they updated many many times.

However, for a Data Acquisition application, it will destroy your DAQmx loop timing, meaning you'll only read the buffer every <long time here> instead of frequently.

This can easily overflow the buffer.

As a simple example, I tested this with a simulated device and a simple VI:

If I leave Highlight Execution (lightbulb) off, then this will run happily forever.

Within a couple of iterations of having it turned on, I get the error you describe.

If I turn it off, it returned to normal (because here I used tunnels, not shift registers, for my error condition).

[Kevin_Price]

I think cbutcher just solved the main reason for your buffer overflow error.  Once you turn off execution highlighting, I expect your code to run without generating that error any more.

But there are a couple other basic concepts you'll probably still need to address. 

1. Spectral Plot - you don't plot your spectral results until after the While loop ends.  (Dataflow!)

2. Loop tunnel - the spectral data is delivered via a tunnel which will get overwritten every iteration of the loop.  Only the very last value will get delivered to the plot.

3. Loop termination - your loop terminates either on DAQmx error or when a particular output from a spectral analysis function is True.  In the case of a DAQmx error, you'll probably be delivering empty data to your plot when the loop terminates.

   <time passes>.  Ok, I looked up that function and the boolean in question is a "averaging done?" status.  That makes some sense as a loop terminator.  However, one of the default values of the "averaging parameters" input to that function is to specify "no averaging".  Perhaps when you ask for no averaging, you never get a True out of the "averaging done?" output?

4. Wait 0 - not needed, delete it.  The call to DAQmx Read will limit your iteration rate according to (# samples to read) / (sample rate).   It's generally a bad idea to try to pace your loop with more than 1 single method.

5. # samples to Read - This value *can* be mostly independent of the sample rate and the buffer size.  When doing spectral analysis, you should choose a # samples to Read according to the frequency resolution you want.  So as you mentioned in the original post, if you Read 12800 samples, that represents 1/4 second at the 51200 Hz sample rate.  Thus your freq resolution would be 4 Hz.

    Your latest postings are reading 51200 samples at a time, 1 second worth, thus a freq resolution of 1 Hz.  If you wanted super-fine freq resolution, you'd have to read more samples at a time - but then you might start getting into buffer size issues.

    DAQmx buffer auto-sizing is presently giving you a 100000 sample buffer, not quite 2 seconds worth.  That's what you'll get if you don't wire a value into "# samples per channel" when calling DAQmx Timing and configuring for Continuous sampling.  You'll also get it if you wire in a smaller value.  However, you *can* wire in a larger value and DAQmx will honor that request and give you a bigger buffer.  Then you can read more samples at once without an overflow.  (There would be other ways to accumulate larger chunks over the course of multiple reads, but the simplest approach is to make a bigger task buffer.)

-Kevin P

[Kevin_Price]

Brief followup: I still had the help page up for the spectrum function.  Scanning down the page a bit further than I did before, it appears the "averaging done?" output will always be True when specifying "no averaging" as an input.

-Kevin P