DMA FIFO from FPGA to RT host gets full

[Cashany]

I'm transferring data through DMA FIFO from FPGA to RT host. 

DMA FIFO gets full, I have tried every thing that I know: 

- increased the size of the DMA FIFO to the maximum, on the FPGA side

- configured the depth of the DMA FIFO to be 100000000

- increased the amount of reading from DMA FIFO in every iteration of the loop

- used a timed-loop with a frequency of 1MHz instead of a normal while loop

Please find attached my project file, FPGA code and RT code.

[Nick-C]

There is alot going on in the loop where you're reading from the DMA on the host side. Have you benchmarked the time it's actually taking to execute the loop? Even if you put a timed loop there on a real-time target, it doesn't guarantee that the code will execute that quickly, it simply places the tasks inside the loop at a higher priority for the processor.

[Cashany]

Do you have any option in mind to prevent DMA FIFO from getting full?

[Cashany]

I'm not at my desk now, I can't check how much time the loop takes, but I'll check later.

[The_Seeker]

Expanding on Nick's comment, you need to be sure that your FIFO servicing loop (on the RT side) can keep up with how fast you are loading the FIFO (on the FPGA side).

Keep in mind that cRIO RT is not going to be as capable as a high-end, multicore desktop PC.

Options:

1) Reduce the number (or optimize) the operations to reduce the processing load on the RT.

2) Upgrade to a more capable cRIO controller (if you are not already using the top-end unit)

3) Offload some of the processing to a Host PC -- if this makes sense for your application.

4) Pre-process on FPGA -- if this makes sense for your application, this one could potentially reduce both the volume of data in the FIFO, and the RT processing load.

First, benchmark your RT loop to determine if this is indeed the bottleneck. If so, in many cases, code optimization and data reduction is possible. These are probably your best places to start.  

Good luck with your application!

-- Dave 

[Cashany]

Good suggestions. I'll work on them and I'll post the results afterwards.

[Cashany · Accepted solution]

I solved my problem. 

Below you can find my FPGA code before solving the problem and after solving the problem. 

Solution: I just added a wait function of 5000 milliseconds (5 seconds) before getting the samples from the analogue input (AI) nodes. 

Before: 

After: 

[CoastalLabVIEWER]

I think I stumbled on something similar today with Digital I/O on a 7853R.

On the host VI side I have code to start the FIFO then immediately Read for 0 elements, get the remaining elements, then do another read for these elements.  I do this before a loop as a way to "flush" my buffer.  However, the "flushing" action didn't seem to work and just about every time I'd get nearly full buffers read back on the first iteration of my while loop.

I added a 100ms wait between starting the FIFO and the first read and this seems to have solved what looks like some kind of odd race condition.

I didn't try reducing the wait time down and for now I've left my application at 500ms wait but I fear, and until I can do quite a bit of testing, that occasionally even 500ms won't be enough.

Has NI seen anything similar to this?

I can probably attach the host and fpga VIs if it helps.

[CoastalLabVIEWER]

I'm still seeing behavior that I can't explain.

Basically, I open a reference to an FPGA VI that contains a DMA FIFO of 32767 U64 elements.  To this FIFO I am storing a time stamp and a position reading from an SSI probe at about 1khz.

This reference gets opened but I may not actually read from these FIFOs for seconds to minutes later.  My hope is to flush the FIFO first and  then begin reading from it.  However, I can't quite explain what I am seeing.

Generally, if I open the reference and then wait enough time to guarantee I have enough samples to overflow the buffer typically my first Read from the FIFO will show a small number of samples and then if I wait at least ~5ms the next read almost always shows 32767 items whether I read out that initial small number or not.  I can't explain this and it's causing me fits because for any given entry into my Host side VI  that is meant to collect samples I can't (easily) tell if I'm flushing out dummy values or if I'm actually collecting "good" data and am thus subject to having to wait until the buffer fills up (up to 32seconds at my scan rate).

I could probably reduce my FIFO size down to a small number (the default 1024 might be ok) but I really want to understand what appears to me at the moment to be buggy behavior that happens at FIFO Start in the host VI in cases where the FPGA VI FIFO has been running for long enough to overflow the requested size of the FIFO.

[Cashany]

I hope NI engineers will answer your question.