sbRIO9637 data corruption at higher loop speeds

[JesseBuck]

Hello, I am sending a waveform out AO0 and AO1 using a FIFO, snippet of loop is attached.  If the loop timer is 1 msec then the output is clean.  However when I take the loop timer down to 50 usec (20 kHz) then I get occasional glitches in the output, a scope trace of one such event is attached.  If I take the loop timer down to 20 usec, then the overall duration of the output doesn't get much shorter but the waveform is severely corrupted.  I would like to understand how the corruption occurs and how to get reliable output at 50 kHz update rate.  I am guessing the fixed point divide could be using too much CPU, but I don't see how to put an I16 value into the 16 bit DAC, it expects a voltage of +/- 10.

thanx,

JEB

[JustJohn]

Hello JesseBuck,

A few quick questions for you. What type of FIFO is that? Is it just an RT FIFO or is it a DMA FIFO that is communicating betweent the FPGA and the RTOS? Also I am not entirely sure I understand your question about the dividing issue. Is the output very large and you have to divide it down to be somewhere betweeen -10 and 10V? What values are being written to the FIFO that you have in your code? Could you post that side of the code so that we could see what is actually being written to the Analog Out ports?

[JesseBuck]

Hi John,

Thanks for the feedback, this FIFO is Host to Target DMA.  The requested number of elements was 200 and I just tried changing that to 5000 since my data set is 2000 points, but it seemed to make no difference.

The timeout value is set to 0 for both read and write FIFO Invoke Methods.  I was having trouble with the program hanging up using -1 timeouts.

I found a possible mistake in the code that I posted, it was also writing voltage values to indicators inside the loop.  However, when I unwired the indicators I still see data corruption at moderate update rates.  This data corruption is my main issue, the program is not reporting any errors and yet I'm getting junk on the outputs when I look with a scope.

On the dividing issue, the idea is that my data source is an I16 number with range  -32,768 to 32,767.  The numbers come in from a data file created by a separate program.  I was planning to just poke that value directly into the 16 bit DAC, traditionally I do this sort of thing with integer math only.  As shown, I can divide my value by 3276.8 to scale it to +/- 10 but I was still hoping to command the DAC in bits instead of volts and bypass the fixed point math.

Thanks,

JEB

[JustJohn]

Hello JesseBuck,

What would happen if you did the divide down before you sent the value into the FIFO and just simply read out the value on the FPGA side? Also could you implement one DMA FIFO for each value that you are trying to read? Instead of having to index the small array, you could just write the output element directly to the true or false selector and wouldn't have to worry about the dividing or the extra time needed to parse the array.

[JesseBuck]

Hi John,

Before responding to your questions I wanted to provide some more information.  I attached the code that writes to the FIFO as you suggested before.  I got a sizable increase in speed before corruption sets in by unwiring the indicators inside the loop that writes the FIFO.  Now I can set the loop time down to 25 usec before the data gets corrupted.  This seems to indicate that the FIFO write is involved with the corruption??

I'm starting now with sets of 2 elements per step but later it will be up to 6 elements per step.  It seemed like array auto-indexing is a clean way to handle various numbers of elements, the read node that you commented on is hardwired for 2 elements just for initial testing, if I used 1 FIFO per element it seems like it would be harder to move to the higher number of elements cases.  The write node included here is set up for different array sizes already.

Your different implementation suggestions make sense for improving performance, but before I change the approach I prefer to understand where and what is going wrong with my current code.  If I can identify the error condition that makes the output turn into junk that would greatly increase my confidence going forward.  For example, the FIFO write has an error terminal and I never see an error from that, but the FIFO read has no error terminal, so how do I figure out where the error is?  It seems like the software would know there is an error somewhere if I'm getting garbage out of the analog outs, but the only indication I have right now is seeing it on the scope after the fact.  For example, the FIFO write has an error out terminal but I have not seen an error, while the FIFO read is not equipped with an error terminal 

Can you assist me to locate the specific cause of the corruption?  I attached a trace showing the behavior I get now with a 20 usec loop timer.  This is typical and not the wildest waveform I could find.

Thanks,

JEB

[JustJohn]

Hello JesseBuck,

Thank you for the detailed information. If you plan to expand this setup, I believe that it would be a good idea to keep the single DMA FIFO instead of using one per output. I assume that the portion of code that is writing to the DMA is on the RTOS of the sbRIO. My thought would be to change the type of while loop being used to a Single Cycle Timed Loop to see if this affects the glitches at all. Based on the picture that you sent, it either looks like the RT side isn't writing values at times causing these dips down to zero, or the FPGA side is not reading the values correctly which might result in it just reading zero as opposed to the correct value. If this were the case, there wouldn't be any visible errors because technically the software structures are working as they are expected to work.

I would check the values that are being written in (by possibly writing them to file to a file or something else to verify that they are correct) and then check that they are correct on the FPGA side by using the same method. We need to determine where the degradation is occuring. Is the DMA working properly but having zero values written to it? Or is the FPGA reading the data wrong that is writtent correctly to the DMA FIFO.

[JesseBuck]

Hi John,

I will soon follow up on your suggestion to probe the FIFO data and narrow down where the bogus data comes from.

Quick question on your performance improvement suggestion, it seemed like you were saying to use a single cycle timed loop to write the data into the FIFO, but since the RTOS is writing the data and only the FPGA has single cycle timed loops, I seem to have something confused.  Please expand a little on the while loop / timed loop suggestion.

Thanks,

JEB

[JesseBuck]

OK, I found that the zero values were extra points inserted by the FIFO Read when I gave the Read a positive timeout value. Now I have -1 timeout applied to Read and Write FIFO nodes and the zeros are not a problem. However there are still sometimes delays in the middle of clocking out the data, scope picture attached. I have a DIO line toggling every 2 data points shown in yellow. The waveform output is subject to random pauses it seems, as seen in the flat line parts yellow DIO trace. With a 25 usec clock time I usually see a clean trace, but sometimes I randomly get junk as in the picture. It suggests to me sporadic overloading of the RT processor but I didn't think I was telling it to do all that much, the data file has already been read and the array is just waiting to be poured into the FIFO. One other thing, I changed the fixed point scaling step in the read loop from a divide to a multiply to reduce loading. I'm happy that the performance is now high enough that I can get some things done but I need to understand what the limits are that I am running into!

Thanks

Jesse

[JustJohn]

Hello JesseBuck!

So it appears that by changing the timeout to -1 your FPGA VI will no longer read zero values because it has to wait until something is available to read. The issue I think is the actual rate at which the FIFO is being written to. The FPGA is reading the data so fast that it actually is having to wait for the RT VI to write more elements for it to read which might be why we are seeing the delays on the Oscope. If you look closely it doesn't look like the values are wrong in the Y-axis, just the X-axis which might be caused by the FPGA having to wait. The Timed Loop suggestion was mainly about the RT VI, I wrote "Single Cycle Timed Loop" but you are correct, SCTLs only apply to FPGA VIs. The "Timed Loop" can be used in the RT VI and the priority of the Timed Loop can be set so that the RTOS writes the values properly at a rate that you set. One other suggestion would be to take out any non related processes like the "Get File" case structure in the top right of the RT VI. This can be implemented in a different while loop that won't alter the performance of the process that is writing to the FIFO. Also is it possible to speed up the rate of the RT VI loop? You would be able to set this when using a Timed Loop but I want to see if writing to the FIFO faster eliminates the strange pausing and junk data that we are seeing on the Oscope.

[JesseBuck · Accepted solution]

Hi John,

Good job steering me to the Write execution- the big problem was adding a FOR loop and placing the FIFO_Write inside the loop.  That means I said to queue 3 elements, a thousand times in a row.  The idea was using the FOR autoindexing to change the 2D array into a 1D accepted by the FIFO Write.  But as I periodically learn in LabVIEW, never do something with a FOR loop if there is a VI somewhere that will do the same thing in a single bite!.  In this case Reshape Array turns the data straight into a 1D array containing all the points (shown in snippet), and I ask for all the points at once.

With this change it tops out at 320 Ksamples/sec net analog out.  That's close to the max rating,so results are now considered very good!

(problem solved but I don't see the button for that)

thanx

JEB