Issue with sampling from the 9212 using the FPGA

[cirrusio]

Hello all,

I seem to be having an issue with sampling from the TC-9212 using the cRIO.  The issue I believe is on the client side.  I am sampling from the 9212 and am trying to take measurements at 1 Hz using the following code in the image:

Simple enough.  But, for whatever reason, attempting to read the data from these FPGA controls is taking over 1.5 s.  Does anyone have any thought as to why this might happen?

[crossrulz]

I don't see anything glaring.  But if all you are doing with those temperature sensors in the FPGA is pass them up to the RT, then move that module out of the FPGA and use the Scan Engine to read those.  You will have a nice temperature measurement and you will not have to do any of the calculations.

[GerdW]

Hi cirrusio,

which of these timing modes do you use?

How do you read all those channels in the FPGA?

[cirrusio]

Thanks, guys.  I am using the high speed conversion time on the TC module - but this really should not be the issue as I am just reading the front panel values.  On the FPGA, the temperatures are just read off of a single node at 2 Hz.

@crossrulz - I can not (and likely would not even if I could) the scan engine.  I am using a lot of resources on the cRIO that are not scan based.  And beside, why would you use the scan engine when you have an FPGA ;)?

[GerdW]

Hi cirrusio,

I am using the high speed conversion time on the TC module - but this really should not be the issue as I am just reading the front panel values.  On the FPGA, the temperatures are just read off of a single node at 2 Hz.

Then you should show more than just an image of your host VI block diagram!

Strip down your project to contain:

- the FPGA VI with mainly the 9212 IO nodes

- the host VI reading the 9212 values from your FPGA

- the project file with all needed definitions and additional VIs

Then (and only then!) we can start to debug…

After stripping down your project you should also compile your FPGA VI again and test your host/FPGA combination. Maybe your assumption is wrong and it's not the 9212 who limits performance!?

[cirrusio]

> Maybe your assumption is wrong and it's not the 9212 who limits performance!?

Likely. But, I am going to go ahead and once again try to reiterate why I think "post the code" is a poor request in cases such as this.  When I post the code, I am have to assume that others have

• the FPGA and RT modules
• the DAQ module in question
• a cRIO of similar computational power
• the inclination and time to actually compile and work through the example I provide (working with the FPGA is not something you simply run on your desktop as you likely know)

Further, I have to assume that the problem is not caused by something else that is going on in the project.  I will be clear - at no point have I made an assumption that there is not something else going on.  I am simply putting out there what I know - my simple VI (though part of a considerably larger application) of which the primary function is to read those temperatures is getting bound up in that one block. 

By posting this I am asking two things - 1) has anyone seen something similar when working with this module and 2) if not does anyone have a possible alternative explanation.  In the interest of transparency I will go ahead and posit a possible alternative explanation - the FPGA resource is getting bound up in another process and this VI is waiting for access.  I don't know if this is reasonable (and having worked with the FPGA for almost 10 years now I have never seen anything such as this), but it is a suggestion of something to look at.

However, if you want to see "stripped down" code, then you simply have to look up the example that is shipped with LabVIEW.  I can provide no more basic example then this.  Feel free to compile this code and test the timing for reading to see if you get a similar issue.  Here is a snapshot of the FPGA VI which actually probes the TC module for reference.  The loop timer runs at 500 us.  The VI which contains the read on the RT side is essentially what is shown above.

As GerdW has pointed out, this is very likely something else.  I will post when I find out if and what that is.

[GerdW]

Hi cirrusio,

When I post the code, I am have to assume that others have…

If you would post code AND clearly state "the code is about cRIO and FPGA" then you would be sure you get responses from people being able to look at the code and (maybe) even to compile it…

if not does anyone have a possible alternative explanation.  In the interest of transparency I will go ahead and posit a possible alternative explanation - the FPGA resource is getting bound up in another process and this VI is waiting for access.  I don't know if this is reasonable

I don't think your assumption is reasonable: the FPGA ensures fixed execution times as the code is executed by hardware (fabric and gates)!

My explanation: you have one big loop doing all different kind of HW access, including reading measurement signals AND executing property nodes of the VERY SAME module. I guess this will lead to "stalls" in the execution, even when there is just "gates and fabric"!

- I use to put seperate "tasks" into separate loops on the FPGA: do DIO stuff in one loop, do AI stuff in one loop, to AO/Pt100/thermocouple stuff in their own loops. Especially do property nodes stuff in separate loops - only on request, not with each iteration!

- I would also stuff all/most of those boolean controls into just one U16 or U32 value: much less communication overhead between FPGA and RT host…

- On the RT host side I would also break up reading FPGA values into separate loops when I know the FPGA handles those different values in different loops at different rates…

[cirrusio]

> If you would post code AND clearly state "the code is about cRIO and FPGA" then you would be sure you get responses from people being able to look at the code and (maybe) even to compile it…

OK, so here is the deal - this is a large AF application using the FPGA on a real time controller with web services.  Creating a separate application to have others debug would likely require hours of my time for something that is not clear to me would be even useful given the reasons above (I have posted the code for the FPGA and the relevant portion of the code which reads the TC module data on the RT side).  I post on here to get my brain turning to see if others have suggestions while I try to push other items forward (this problem is not a show stopper yet as the rest of the application seems to be humming along fine and there are other cherries to pick).  So, the gist of this is that I will not be posting code.  If this is a problem, I completely understand not engaging. 

> My explanation: you have one big loop doing all different kind of HW access, including reading measurement signals AND executing property nodes of the VERY SAME module. I guess this will lead to "stalls" in the execution, even when there is just "gates and fabric"!

What is this comment referring to?  Execution of what? The FPGA?  If so, please look at the diagram and tell me where the "one big loop" is that you are referring to?  This application uses the native clock which is executing at 40 MHz.  Each of the operations within that loop should be able to execute within a single cycle.  The loop timer for that large loop is set to 2 Hz!  The reason for that "big loop" is that all of those operations are required to execute at very low rates (they are general housekeeping) and I have never had an issue with loop execution not completing within the requested period for these kind of operations.  If this is a problem, then it is related to the TC module specifically.  But, laying all of that aside, the FPGA execution rate on the loop should have no bearing on the execution on the RT side!  The RT side is simply reading latest values and does not have to wait for anything from the FPGA.  That being said, I am adding a timer on that loop to test this.  And, if there is something about how the TC module read is handled that should prevent it from returning at no slower than 1 Hz, please let me know.

> I use to put seperate "tasks" into separate loops on the FPGA: do DIO stuff in one loop, do AI stuff in one loop, to AO/Pt100/thermocouple stuff in their own loops. Especially do property nodes stuff in separate loops - only on request, not with each iteration!

We have a different philosophy about this - it is the FPGA.  Yes -a n IO node can take more than 1 cycle to execute, but if you are operating under the assumption that your required execution is low frequency, then this is never a problem.  If I have 40 million cycles to work with, I don't really care if something takes 2 or 5 or 10 cycles to execute. In fact, everything on that diagram can take 20 million cycles to execute - for these operations I don't need the execution speed so I have cycles to spare.

> I would also stuff all/most of those boolean controls into just one U16 or U32 value: much less communication overhead between FPGA and RT host…

Once again - different philosophy.  I used to be of the same mind, but the truth is this is really just obfuscation and makes the code less clear to read.  You have to ask yourself what you really gain out of this.  The way you are suggesting, I have to first convert an array of booleans to an int and then pass it along.  On the RT side, you have to grab that int, convert it to a boolean array and then figure out which boolean is what.  If you are toggling booleans in multiple places (i.e. your application is distributed) you will have to first convert that int into a bool, then replace an array subset and then convert that array once again to an int for passing back to the FPGA module.  I will go ahead and put forth that, when you have the space, pushing individual named bools in and out is significantly easier to understand than having to assemble and disassemble arrays of bools.  Much easier on the maintainer (even if it is just you a year later).

> On the RT host side I would also break up reading FPGA values into separate loops when I know the FPGA handles those different values in different loops at different rates…

This is a distributed application using the AF so this occurs naturally.  What I am suggesting above that could be happening is a possible consequence of this architecture.  There are two subVIs on the FPGA diagram, each of which is handling high speed interactions.  Large amounts of data are pushed into FIFOs that are picked up by the DMA on the RT side.  I am retrieving data from those FIFOs at 1 Hz which requires the application to sit on the FIFO reads while it waits for new data.  What is not clear to me is whether something funny could be happening where these reads are blocking access to the FPGA ref (similar to if you had an application trying to access the same VISA session in two places simultaneously).  Given that these reads are executing at 1 Hz, I would expect the ref to become available at at least 1 Hz.  But what if it is not?

But like I said, this is just one possible hypothesis.  Thanks for your suggestions thus far.  I will examine all of this in so far as I can and let you know what happens.  Cheers.

[GerdW]

Hi cirrusio,

Creating a separate application to have others debug would likely require hours of my time for something that is not clear to me would be even useful given the reasons above (I have posted the code for the FPGA and the relevant portion of the code which reads the TC module data on the RT side).

Just two notes:

- How much time would it require to create a simple FPGA VI containing mostly the 9212 code (as you suspected it to be the cause for the problem) and a simple RT host VI to read/write stuff to/from the FPGA? Hours?

- You did NOT post code, but pictures of code. Ask Bob about this topic, he writes nice comments about "code images"…

Beside all this: holiday season starts and I wish you a merry XMas and a happy New Year!

[cirrusio]

> How much time would it require to create a simple FPGA VI containing mostly the 9212 code (as you suspected it to be the cause for the problem) and a simple RT host VI to read/write stuff to/from the FPGA? Hours?

That would take me approximately 0 minutes - it is already in the examples (look up 9212).  Is that what you are looking for?  Would it be better to package the example up to post here? I don't understand - if you were interested in that, I pointed to that example which is shipped with all LabVIEW FPGA modules a day ago.  Now, if you are interested in the context, then that would be more significant.  And probably a complete waste of time given that I have pointed to this simple example and not one person has compiled it on their hardware and reported the performance of doing a simple read of all channels simultaneously.  Nor have I even had anyone respond with their experience with reading all channels off of the FPGA from the 9212.

So, I have no idea who Bob is (Schor maybe?), but at this point I don't really care to continue this line of conversation as it is just a distraction from input germane to the original post.  If you happen to have the TC module and a cRIO and you want to look at this with me, then please compile the example and just run a simple timing test on the RT side to see how long it takes to retrieve temperatures from the module.  If you don't, not a problem.  This will eventually get worked out either way.

Now, onto a real finding

I dropped the number of temperatures that I am reading on the RT side from the 8 channels + the CJC temperatures (2) down to 4 + 1 CJC and found that the the time spent on this FPGA IO node dropped by half (from over 1.5 s to 850 ms).  This timing is specific to that one block as can be seen in the initial image.