How to debug LabVIEW crash on roboRIO Error (Hex 0x661)

[WillToth]

Hi,

This is our first year trying Command and Control for the roboRIO and we have a problem that we have not seen before. The behavior that we noticed this was with compiled code deployed as startup and either attempting to deploy a new build or running from the LabVIEW IDE. While deploying the driver station CPU percent suddenly drops, and all the PWM references say they have been been closed and are invalid, and the deployment process stops and says its waiting on the target. This completely prevents us from pushing new code to the bot.

We also saw a sudden disconnect from both the driver station and the development computer at one point while running the RIO from the IDE, which may also have been a similar issue.

We are using the latest image V17 on the bot as well as the CTRE CAN library. 

I've attached the error log which includes the following message in several cases:

02/24/2018 21:11:59 [error] LabVIEW:  (Hex 0x661) The LabVIEW Real-Time process encountered an unexpected error and restarted automatically. LabVIEW Real-Time process restarted

My question is, what are some things we can do to properly debug this?

Thanks

[BoKnows]

I'd start with a base project and see if the error reproduces there.  With that, we can determine if there's something corrupt or something specific to the code.

Are the CAN drivers the only piece you added?  A lot of teams are using the NavX, for example, that add more external calls as well.

I'd personally try to use a binary type troubleshooting.  Remove a subsystem from the project and use that to narrow down where things start to go awry.  At least, that helps point us down the path of the crash to better determine what exactly is causing it.

[WillToth]

Hi BoKnows,

Thanks for the feedback, that is a decent place to start, and we'll look at that today. 

The CTRE library for CAN is the only new thing we added other than a whole lot of LabVIEW code that we have developed for our subsystems. When you say 'external calls' do you mean external library function calls?

[BoKnows]

By external calls, I mean I haven't dug into the CTRE library or the NavX, as a popular API, to determine if they're using things provided by LabVIEW or if they're making external library calls.  At a generic level, I just mean code that isn't built by your team or included in the WPI libraries. 

[WillToth]

We are starting to suspect the CTRE library is causing these issues. We noticed there are two debugging messages that appear on the driver station when everything is working properly:

Warning  0  [Phoenix] Initialization is complete.   

 After pushing code and waiting for it to transfer to the RIO we get

Warning  0  [Phoenix] Shutdown cleanly  

I also noticed that when we try to run code from the IDE, we get the message Waiting for Real-Time target (Target) to respond... but this goes away and the code pushes successfully. The message goes away only after the second warning above is seen on the driver station.

When we encounter the issue where we cannot push, the second warning message ([Phoenix] Shutdown cleanly) is not seen on the driver station at all.

Both of those above behaviors have been observed multiple times.

I also don't suspect and loops hogging the CPU, as the roboRIO reports plenty of CPU head room when it is 'stuck'. I've attached two logs, one showing the 'good' behavior and one showing the 'bad' behavior.

Does the deploy have to wait for the CTRE library to complete before pushing new code?

We haven't been able to test it by removing a subsystem at a time yet as the proposed method is crazy time consuming and we have to prepare for our regional in 2 days. Its tough and time consuming because we would have to remove subsystem --> Build --> Deploy (if we can) --> Try to reproduce (takes some time) --> Repeat. 

[WillToth]

We are starting to suspect the CTRE library is causing these issues. We noticed there are two debugging messages that appear on the driver station when everything is working properly:

Warning  0  [Phoenix] Initialization is complete.   

 After pushing code we get

Warning  0  [Phoenix] Shutdown cleanly  

I also noticed that when we try to run code from the IDE, we get the message Waiting for Real-Time target (Target) to respond... but this goes away and the code pushes successfully. The message goes away only after the second warning above is seen on the driver station.

When we encounter the issue where we cannot push, the second warning message ([Phoenix] Shutdown cleanly) is not seen on the driver station at all.

Both of those above behaviors have been observed multiple times.

I also don't suspect and loops hogging the CPU, as the roboRIO reports plenty of CPU head room when it is 'stuck'. I've attached two logs, one showing the 'good' behavior and one showing the 'bad' behavior.

Does the deploy have to wait for the CTRE library to complete before pushing new code?

We haven't been able to test it by removing a subsystem at a time yet as the proposed method is crazy time consuming and we have to prepare for our regional in 2 days. Its tough and time consuming because we would have to remove subsystem --> Build --> Deploy (if we can) --> Try to reproduce (takes some time) --> Repeat. 

[nestor-sg]

Read this document from CTRE, it might solve your issue:

https://github.com/CrossTheRoadElec/Phoenix-Documentation/blob/master/LabVIEW%20Deploy%20Issue.md

Our team (4343) has been seeing the exact same issues... and just this last weekend we updated to the newer libraries as specified there. Too early to say that actually fixed it... but we made changes in the code in which we serialize/sequence CTRE calls and we stop seeing the issue.

[WillToth]

We had inadvertently taken the steps outlined in that document and we completed our first regional event in Dallas and I can report that we did not see this issue at any point (different roboRIO and different TalonSRXs). One thing we do still see is it hangs for a few seconds until the message from Phoenix shows up, but we never saw it fail.