Real-Time Application doesn't run; source code works fine

[Andy_Humphrey]

The short version is I'm programming a cRIO and apparently the RT code isn't running after being deployed and I can't figure out why. This is further complicated that I'm doing all this remotely and I don't have direct access to the unit since I'm 500 miles away. I'm working through a couple of other guys who know some LabVIEW, but neither works at the site so they have to explicitely travel out there every time I have a bright idea.

I was out there a few weeks ago. During this time I created a simple cRIO code, since I'm new to cRIO, that allowed the user to move a control and change a graph. It worked fine, but I should note it did not have an FPGA component. After that I worked on the real code, which reads some sensors, displays the results on a UI and logs the results. This did have FPGA. I used it in the LabVIEW environment and it worked fine, but I ran out of time before I could complete a release version and deploy the RT as a compiled application. I sent them the release version later, my contact deployed it but got network stream errors when running the UI.

After hours of looking at network problems and sending over debug versions, I tried creating a log on the RT level so I could see what was going on. The log doesn't even open, even if it's the first command in the code. I pored through the forums and found http://forums.ni.com/t5/LabVIEW/cRIO-Troubleshooting-creation-and-deployment-of-startup/td-p/1956475... which took me in a new direction.

I had my contact use the RT debug console and when he pulls up the RT front panel, it shows a broken run arrow. He clicks it and nothing happens -- no running, no bug list. If he pulls up the bug list manually, it's empty. Again the RT works fine if you run it through LabVIEW and not as a compiled real-time application. He also noticed that the Open FPGA VI was grayed out on the block diagram. No other icons are.

So the problem appears to be that the compiled RT application is getting some kind of error but not telling me what it is, and it seems to be related to opening the FPGA. I've recompiled the FPGA and RT. I've had him recompile the RT himself, but not the FPGA because it would take hours. He's downloading everything correctly to the cRIO. The RT is set to run automatically. He's rebooting the cRIO every time he deploys the RT. They have LabVIEW on a computer there but it doesn't have the right drivers to run the code from the LV environment. I'm resisting having them install the dirvers because downloading big files is complicated there due to security restrictions and a lousy network connection at a remote site. Besides that doesn't solve the problem of the RT executable not running the same as the source code, which according to the thread above appears to be a thing.

The latest thing I'm trying is that I sent him instructions for how to build a source distribution from the project I sent and try deploying that to the cRIO. Even if that works I'm not sure that's an acceptable solution because I assume running the VI rather than the EXE is slower, and they need speed on this project.

I simply have no idea where to go from here. I probably need to get direct acess to the cRIO and I might be able to convince them to ship it to me so I can figure this out, but I'm not sure where I'd even start other than the standard voodoo debugging of "try stuff at random until something works". I'm open to suggestions if anyone has managed to solve this before.

Code snippet of the first part of the project is attached, though I'm not sure how much good it will do. I'm really stumped and the client is getting frustrated with how much of the budget is going to fix this.

[nathand]

Have you checked the cRIO error log? Usually I'd access it through the LabVIEW project (right-click on the target, don't remember the exact menu options and I don't have the RT toolkit installed on this machine to check), but it must be stored somewhere on the cRIO as well, although I don't know if it's in a human-readable format.

Which cRIO are you using? What exactly do you mean by "debug console"? (This may be related to the cRIO - the newer ones have video out, although I don't know if that's what you're referring to.) With a broken run arrow, you won't get an error list unless you're running in the development environment.

Have you confirmed that the software installed on the cRIO matches the version you're using for development, including patch level? Get someone to connect to the cRIO with Measurement and Automation Explorer, and get a list of the software installed on the cRIO.

Sounds like the ability to connect with a remote debugger would be helpful here, if you can get the right drivers installed on the machine with LabVIEW that's connected to the cRIO. Make sure all driver versions match what you're using. Any chance you could then do a remote desktop connection from your work site to the remote LabVIEW machine?

[Andy_Humphrey]

Thanks for the input, nathand. It's a cRIO 9082. I'll have my client check the error log to see if there is anything there. As I said, I'm new to cRIO development so I didn't even realize that was there.

The debug console -- at least that's what I call it -- is what you get when you click on "Operate > Debug Application or Shared Library..." from the project screen (assuming the RT has been compiled with the debug option). It lets you get the front panel and block diagram of a compiled real-time application so you can place probes.

I'm fairly sure the software is the same on the cRIO as my development computer but I'll double check with the client.

And a remote connection to the cRIO from here would be great, but network security is super high (national laboratory) so that's not going to happen.

[nathand]

I don't have any way to test this, so this is purely a guess, but I think Open FPGA VI may be greyed-out in the block diagram if the FPGA drivers aren't installed on the machine that's being used for remote debugging (which I think you said is the case). In that case, it might be misleading as the source of the problem. I hope someone who can actually test (or knows, from previous experience) will confirm or deny this.

As for the log file your code tries to write, are you sure you're writing to a valid path? The format for paths varies from platform to platform, and the snippet you posted doesn't show the path where you're trying to write the file.

[Andy_Humphrey]

The client looked at the cRIO error logs and there is nothing in the Real Time section of any of the logs. As for my own log, yup it's a valid path. I've checked it about a hundred times.

Hmm, in fact as I type that I realize the log used to work several versions ago while I was debugging the original connection problem. I wonder if while trying to fix the connection issue I broke something else and it manifested itself as a connection issue!

I'm still looking into the software installation on both machines since that could still be the problem.

[Sam_Sharp]

Just as a really unlikely possible cause, you're not using network shared variables in your project are you? I found a bug once in LV2012 where having a network shared variable based on a type definition would cause the RT target to not run my rtexe - no errors, no indication of anything going wrong and even just simplifying it to trying to flash the user1 LED didn't work. It ran fine from source but never as a rtexe.

The solution was disconnecting the type definition on the shared variable - I could then deploy and run the code on the cRIO without problems.

Apparently the issue was fixed in LV2013.

[Andy_Humphrey]

Interesting. I am using network shared variables based on typedefs, but I'm using LV 2013 so that's probably not it. Thanks for the suggestion though.

[MichaelBalzer]

As soon as I saw the message title my first thought was typedef'd shared variables. I've run into this issue more times than I care to count, and I'm sure I saw it as recently as LV2013 SP1.

Did you try disconnecting the typedefs as Sam_Sharp suggested? There's a disconnect typedef option in the RT build spec you can check to remove all typedefs from the build. I check this option out of habit now to avoid just this issue.

[Andy_Humphrey]

Yeah, I'm going to try that the next time I get my hands on it. I talked the client into shipping the cRIO out to me so that is going to make debugging much easier. I have a handful of ideas to try including the network variables. I'll let you folks know what I figure out.

ETA: And I just reread your post and saw the bit about the RT build option. That will be much easier than messing with all that manually. Thanks!

[natasftw]

Try this: abstract the FPGA out of the problem for a moment.  Create a simple project that uses the scan engine to read a single sensor.  Does that give you the same problem?

What are you doing with the inputs that makes the FPGA more beneficial to you than the scan engine?