PXI 6602 - Error -200141 - I can read for a second or two before receiving a data over written fault but my buffers don't appear to be overflowing

[Matt_AM]

Hello Fancy Folk,

I am hoping someone can shed some light as to why I am receiving an error -200141 when performing a counter in angle position continuous read.  I believe I have everything wired properly, the data acq card should be fast enough to perform the samples, the clock generated looks stable on an o-scope, and the code is reading the task fast enough to not cause a buffer overflow.

Test Setup:

I want to read the A and B lines of an encoder on a shaft that spins up to 2000 RPM.  We have a hall effect encoder at a fixed position from a magnet which is mounted in the chuck of a lathe.  The lathe is speed controlled via a 10-turn pot and we have an encoder on the back end of the shaft of the lathe to monitor position and thus derive speed.  I understand this isn’t the ideal situation for my test, but I wasn’t the one that came up with it, I was the one that was given parts and told to color. 

NI Max setup:

I have 3 tasks that I am running, 1 counter in for angular position, 1 analog in for A and B lines, and 1 counter out to derive my clock for the other two tasks.  The counter in and counter out tasks are running on a PXI 6602 and the analog in task is running on a PXI 6259.  The encoder is a 4096 pulse per revolution, so I set my counter out task (clock) to 500k so I can have multiple samples per A and B line while also staying under the aggregate sampling rate of the 6259 (4096 pulse/rev * 2000 rev/min * 1min/60sec = 136533 pulses/s, so 500k is about 3.6x the rate of 136k).  I have set the clock for both the encoder in and analog in tasks to the PFI 8 (counter 7 out) on the 6602.  From what I remember, DAQmx is supposed to handle the routing and everything to make sure the proper signal goes to the proper pin and I have done 0 manual DAQmx routing in my code.

Code setup:

Before running my actual test, I wanted to verify that the tasks work as I expect.  Below is an image of my code that is stripped down to just the analog read and counter read loops while also starting the counter out clock task.  This will be mostly identical to my actual test with main the difference being instead of the read going to indicators, they will go to a queue to follow a consumer-producer design to make sure the read loop has as little hang time as possible. Additionally, once I figure out the number of samples and wait time for the loop, I will also update those variables as well.

I was trying three different methods to perform the read to not get the -200141 error and I was too lazy to update the analog read loop logic since that one isn’t having issues, hence why the top and bottom loop look different.  Digressing, method 1 is set the wait time to 0 and read however many samples are there (currently shown in the upper loop).  Typically, I am seeing 75 samples per iteration through the loop.  The second method (seen in the bottom loop) is where I set the wait time to 0 and read 100 samples.  Based on a wait time of 0 and reading 75 samples per iteration, the 100 samples “should be slower” so the DAQmx read will act as my timing source.  The third method (not shown in either loop) was where I set the wait time to 1 ms and read all samples at the card, which was approximately 500.  This makes sense since it is close to the timing source of my clock (500k).  All three of these methods have yielded a -200141 error stating data was overwritten before it could be read by the system.  If the data transfer mechanism is interrupts, try using DMA or USB Bulk.  Otherwise, divide the unput signal before taking the measurement.  However, all 3 methods will run for a second or two before faulting out.

I am assuming that the 6602 should be able to read at 500k since I am successfully doing that for a little bit and don’t see my number of samples when set to -1 grow exponentially.  However, googling says that the max sampling rate for the 6602 is system dependent (yay…), so I can’t verify via NI that the card can sample that fast for that long. 

Additional troubleshooting:

I hooked up an o-scope on the PFI8 pin to see the 6602’s counter out clock task.  It looks like a clean pulse.  I have also tried using a waveform generator set to 500k with a 5 Vp-p square wave, verified on the o-scope, to go to PFI8 (trying a second clock to verify the problem is not the 6602 clock out).  It likewise causes the fault after a few seconds.

Reiteration of question:

I was wondering if anyone could shed some light onto why I am getting the -200141 error.  My previous theory was that the 6602 was outputting a noisy clock which caused the card to sample faster than expected thus causing the error.  However, after using the o scope and a secondary clock, I don’t believe that is a problem anymore.

My current work around will be to sample the position waaaay less fast than the analog in sample since the encoder on the shaft is to verify speed and I don’t think I’d need to sample at the 500k rate compared to my analog in task.  However, I am still curious as to why this is an issue.

Thanks,

Matt

[mcduff]

I don't have much experience with Counters, hopefully @Kevin_Price will chime in.

A few tips that helped me avoid that error for AI tasks:

1. From the diagram it looks like you are displaying data in the same loop as acquisition, that will slow down your loop if you are downloading a good number of points. Display the data in another loop.
2. No need for the Waits, the read should control loop timing.
3. Grab about 100 ms of data for each loop, anything faster is not needed and adds overhead.
4. This works for acquisitions above MSa/s, which you don't have, but couldn't hurt. Make the number of samples acquired an even multiple of the disk sector size.
5. You can increase the buffer size, in the DAQmx Advanced Task Options Palette. Set it to 4-8 times the sampling rate.

[Kevin_Price]

I suggest focusing first on mcduff's tip #5, then #s 3 & 2.  (I see that you're already intending #1 in your real app.  Nothing wrong with #4 either, but that's not where your problem is at present.)

Your screenshot doesn't show any timing or buffer size configuration for your counter task.  Try setting your counter task up with the same timing/buffer params as your AI task (500 kHz sample rate, 1 million samples per channel for minimum buffer size).  Then request 50k samples per iteration and remove the msec wait timer.   Let us know how it goes.

-Kevin P

[Matt_AM]

---

@mcduff wrote:

I don't have much experience with Counters, hopefully @Kevin_Price will chime in.

A few tips that helped me avoid that error for AI tasks:

1. From the diagram it looks like you are displaying data in the same loop as acquisition, that will slow down your loop if you are downloading a good number of points. Display the data in another loop.
2. No need for the Waits, the read should control loop timing.
3. Grab about 100 ms of data for each loop, anything faster is not needed and adds overhead.
4. This works for acquisitions above MSa/s, which you don't have, but couldn't hurt. Make the number of samples acquired an even multiple of the disk sector size.
5. You can increase the buffer size, in the DAQmx Advanced Task Options Palette. Set it to 4-8 times the sampling rate.

---

1) Yeah, in my main code I wont display the data.  I understand that can cause issues with the speed of the loop.  My general assumption was that since I am reading it as fast as possible and my number of points read was ~75, then it wouldn't matter since I'm not seeing a lot of samples being read

3) I don't understand how reducing my "data transfer rate" (not sampling rate) will help with my problem. Since my error seems to be related to a buffer over ride, wouldn't I want to make sure my buffers are as empty as possible and thus request the data from the card quicker?  As far as over head, I am assuming that when I tell LV to get the samples acquired from the 6602, there is overhead to initiate the transfer, then the transfer occurs, then overhead to verify the transfer happened.  I have 0 clue as to how the transfer mechanism actually works and I am just going off CAN's transport protocol logic and assuming that is similar enough for my PC to request it from the card.  Digressing, I get that requesting data less frequently means that I have fewer "header and footer" messages thus improving the transfer rate over longer periods of time.  

Also, you got any white papers on this?  Not trying to be a "pics or it didn't happen" guy.  DAQmx timing has always been a weak point for me and I would like to know more about proper sampling methodologies.  I took the DAQmx online training during 2020 and felt like it did a good job at explaining the "how the measurement is taken" in hardware but I didn't feel confident that I could set up the timing properly to sync a bunch of measurements together or how to handle fast sampling. 

5) In NI MAX, I set the sampling rate and number of points to acquire to 500k.  Is it safe to assume that LV would create a "good enough" buffer for the 500k samples or should I specifically define a buffer?

Thank you for your response,

Matt

[Matt_AM]

---

@Kevin_Price wrote:

Your screenshot doesn't show any timing or buffer size configuration for your counter task.  Try setting your counter task up with the same timing/buffer params as your AI task (500 kHz sample rate, 1 million samples per channel for minimum buffer size).  Then request 50k samples per iteration and remove the msec wait timer.   Let us know how it goes.

---

Yeah, I forgot to add that into the NI MAX setup, I have set the counter in task to sample at 500k Hz and read 500k Samples for both the analog in and counter in task.  Am I safe to assume that Labview will create a good enough buffer or should I specifically define my input buffer?  

I'll edit this info into the original post

Thanks for your response,

Matt

Edit: Looks like I cant edit my original post.  I believe you can only edit a post for 10 minutes after creating said post.

[mcduff]

---

@Matt_AM wrote:

---

Also, you got any white papers on this?  Not trying to be a "pics or it didn't happen" guy.  DAQmx timing has always been a weak point for me and I would like to know more about proper sampling methodologies.  I took the DAQmx online training during 2020 and felt like it did a good job at explaining the "how the measurement is taken" in hardware but I didn't feel confident that I could set up the timing properly to sync a bunch of measurements together or how to handle fast sampling. 

---

Original PPTX attached

[Matt_AM]

@mcduff and @Kevin_Price

I reworked my code to the recommendations with the back panel seen below and I am still running into the -200141 error.

Note, I have specified the sampling clock for the Ai and Ci to be identical then set the Ci input buffer to 4M (8*500000).  I also am requesting to read 50000 points (100 ms/.002 ms) and removed the wait blocks from the loops.  Additionally I have removed the FP array indicators but still have the size of the data array to verify that I am reading the desired number of points (I am assuming this is a lot less overhead than updating 50,000 points in an array).  

Unfortunately, I am still receiving the -200141 error only for the Ci loop.  I will say that the error is happenings 4-6 seconds after starting the code compared to the original 1-2 seconds after starting the code.  

I tried adding in a buffer property node to read the onboard buffer size but that was erroring out (-200452, specific property not supported by this device or is not applicable to this task).

As a heads up, I'm just going with my work around to sample my Ci much slower (~1 Khz) than my Ai since the Ci is more of a verification of shaft speed in the main test compared to the actual data (Ai) I care about.

Matt

[Kevin_Price]

So sorry, my memory failed me.  I looked up the error text for -200141 and thought I recognized it as the sorta-classic "task buffer overflow" error, which is *actually* error # -200279.  In fact, -200141 is an overflow of the device's internal hardware FIFO buffer, which represents a more fundamental issue related to PXI bus sharing and the DAQmx driver.

There are generally at least 2 buffers involved in hardware-clocked DAQmx tasks.  One is what I call the "task buffer" which resides in system RAM, and can be made almost arbitrarily large.  This is the one that can be influenced by the 'samples per channel' input to DAQmx Timing or explicitly set by a call to DAQmx Configure Input Buffer.

    The other is the device's onboard buffer, sometimes referred to as an onboard FIFO to emphasize that only sequential access is allowed.

    The DAQmx driver does most of the dirty work in the background to manage the transfer of data from the device's FIFO up to the task buffer.  Its job is to move data out of the device's FIFO and into system RAM before new samples overflow the FIFO itself.

And now here's where the problem lies.  The 6602 has a itty bitty little FIFO.  Only 2 samples per channel.  Further, the PXI bus is a shared resource where your 6602 and your 6259 have to negotiate access to the bus in order to deliver data up to system RAM.  When you try to sample count values at 500 kHz, you'll need access to the PXI bus every 4 microsec or less to avoid FIFO overflow.  This helps explain why the counter task can work all by itself but starts to error out in the presence of an AI task on another device needing access to the same PXI bus.

See also pg 2-8 of the user manual, where benchmarks (on admittedly older, slower systems) topped out at < 250 kiloSamples/sec transfer rate.  You can apparently get > 500 kHz for the counter task by itself on a newer system, but not when it has to share PXI bus access with an analog task.

The reason you don't get a similar error from the AI task is because it has a much larger onboard FIFO of 4095 samples.  Note that there's no relief available from the counters on the 6259 -- they're also stuck with the same itty bitty 2-sample FIFO.

Finally, if you can drop your counter sample rate down to ~1 kHz, things will likely work out fine.  I would probably set up a counter output task that would take the 500 kHz clock and divide it down such that low ticks + high ticks = 500.  Then make sure to this divider task *before* starting the 500 kHz output task (as you've already done for the AI and encoder tasks).

-Kevin P

[mcduff]

---

@Kevin_Price wrote:

So sorry, my memory failed me.  I looked up the error text for -200141 and thought I recognized it as the sorta-classic "task buffer overflow" error, which is *actually* error # -200279.  In fact, -200141 is an overflow of the device's internal hardware FIFO buffer, which represents a more fundamental issue related to PXI bus sharing and the DAQmx driver.

-Kevin P

---

Thought it was -200279 also. Counters are interesting, will need to program them in the future.

Far out solution: How big is the chassis? If it big enough it may have a PCI bridge, if so, would putting cards on either side of the bridge help?

[Matt_AM]

---

@Kevin_Price wrote:

There are generally at least 2 buffers involved in hardware-clocked DAQmx tasks.  One is what I call the "task buffer" which resides in system RAM, and can be made almost arbitrarily large.  This is the one that can be influenced by the 'samples per channel' input to DAQmx Timing or explicitly set by a call to DAQmx Configure Input Buffer.

    The other is the device's onboard buffer, sometimes referred to as an onboard FIFO to emphasize that only sequential access is allowed.

    The DAQmx driver does most of the dirty work in the background to manage the transfer of data from the device's FIFO up to the task buffer.  Its job is to move data out of the device's FIFO and into system RAM before new samples overflow the FIFO itself.

Gotcha. So basically the on board buffer is like the cache (my understanding of CPUs is limited, so I think its the cache that is the "CPU RAM") and the task buffer is like the RAM.  

---

Note that there's no relief available from the counters on the 6259 -- they're also stuck with the same itty bitty 2-sample FIFO.

---

Great minds think alike, as I was reading I was curious if the 6259 had a larger counter buffer.  Bummer that it doesnt.

---

Finally, if you can drop your counter sample rate down to ~1 kHz, things will likely work out fine.

---

Ends up that I misunderstood the lines I am supposed to color in. Apparently the encoder is meant to be sampled at the same rate as my analog samples. 

@mcduff

I have a PXI 1044 chassis if that helps at all. 

I have a kind of sketchy work around that I wanted to get another set of eyes to try and "interleave samples".  Theoretically, if I routed my encoder to 2 counter ins, set the sampling rate to say 200 kHz, have one trigger off the rising edge of the clock and the other off of the falling edge of the same clock, would that in essence give me a 400 kHz signal?  I understand there is A LOT of "if the stars align" with this idea.  Worst case scenario, my idea fails and I at least have 1 set of data that is good.  "Best case" scenario would be me sampling the data twice as fast and then figure out how to realign the two signals to create a single signal, which... "best case"...

Matt