cRIO application works in interactive mode but broken VIs reported in error log when run built application on target

[MarkCG]

I have been getting more than my fair share of grey hairs the past few days because of this. Over the past few months this almost never happened. Lately, after a seemingly minor change to some serial communications code, hardly a single build works without throwing errors like this:

#OSVers: 6.3
#OSBuild: Jun 6 2014, 09:14:16
#AppName: /c/ni-rt/system/lvrt.out
#Version: 14.0
#AppKind: AppLib
#AppModDate:

InitExecSystem() call to GetCurrProcessNumProcessors() reports: 1 processors
InitExecSystem() call to GetNumProcessors() reports: 1 processors
InitExecSystem() will use: 1 processors
starting LabVIEW Execution System 306708506 Thread 0 , capacity: 1 at [3507679466.00933790, (19:24:26.009337710 2015:02:24)]
starting LabVIEW Execution System 306708507 Thread 0 , capacity: 1 at [3507679470.59560204, (19:24:30.595601929 2015:02:24)]
starting LabVIEW Execution System 2 Thread 0 , capacity: 24 at [3507679481.25993776, (19:24:41.259937648 2015:02:24)]
starting LabVIEW Execution System 2 Thread 1 , capacity: 24 at [3507679481.25993776, (19:24:41.259937648 2015:02:24)]
starting LabVIEW Execution System 2 Thread 2 , capacity: 24 at [3507679481.25993776, (19:24:41.259937648 2015:02:24)]
starting LabVIEW Execution System 2 Thread 3 , capacity: 24 at [3507679481.25993776, (19:24:41.259937648 2015:02:24)]
Thread consumption suspected: 3 Try starting 4 threads
starting LabVIEW Execution System 2 Thread 4 , capacity: 24 at [3507679515.59324312, (19:25:15.593243004 2015:02:24)]
starting LabVIEW Execution System 2 Thread 5 , capacity: 24 at [3507679515.59324312, (19:25:15.593243004 2015:02:24)]
starting LabVIEW Execution System 2 Thread 6 , capacity: 24 at [3507679515.59324312, (19:25:15.593243004 2015:02:24)]
starting LabVIEW Execution System 2 Thread 7 , capacity: 24 at [3507679515.59324312, (19:25:15.593243004 2015:02:24)]
Thread consumption suspected: 8 Try starting 1 threads
starting LabVIEW Execution System 2 Thread 8 , capacity: 24 at [3507679515.73854351, (19:25:15.738543576 2015:02:24)]

and 

####
#Date: TUE, FEB 24, 2015 04:47:22 PM
#OSName: VxWorks
#OSVers: 6.3
#OSBuild: Jun 6 2014, 09:14:16
#AppName: /c/ni-rt/system/lvrt.out
#Version: 14.0
#AppKind: AppLib
#AppModDate:

InitExecSystem() call to GetCurrProcessNumProcessors() reports: 1 processors
InitExecSystem() call to GetNumProcessors() reports: 1 processors
InitExecSystem() will use: 1 processors
starting LabVIEW Execution System 306708506 Thread 0 , capacity: 1 at [3507670087.84041262, (16:48:07.840412710 2015:02:24)]
starting LabVIEW Execution System 306708507 Thread 0 , capacity: 1 at [3507670092.45153570, (16:48:12.451535843 2015:02:24)]
starting LabVIEW Execution System 2 Thread 0 , capacity: 24 at [3507670103.11840487, (16:48:23.118404925 2015:02:24)]
starting LabVIEW Execution System 2 Thread 1 , capacity: 24 at [3507670103.11840487, (16:48:23.118404925 2015:02:24)]
starting LabVIEW Execution System 2 Thread 2 , capacity: 24 at [3507670103.11840487, (16:48:23.118404925 2015:02:24)]
starting LabVIEW Execution System 2 Thread 3 , capacity: 24 at [3507670103.11840487, (16:48:23.118404925 2015:02:24)]
VI_BROKEN (0): [VI "NC serial comms protocol.lvlib:Set Lower Temperature Limit.vi" (0x04667310)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NC serial comms protocol.lvlib:Set Lower Temperature Limit.vi" (0x04667310)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "Heater Control HAL.lvclass:Write Heater Control Active.vi" (0x058c4e98)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "Heater Control HAL.lvclass:Write Heater Control Active.vi" (0x058c4e98)]
this->flags=50340352, compilerError=6
VI_BROKEN (0): [VI "NI_LVConfig.lvlib:Save Config File.vi" (0x047b7e90)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_LVConfig.lvlib:Save Config File.vi" (0x047b7e90)]
this->flags=33563136, compilerError=6

and so on.

This is using a cRIO 9074 running CompactRIO recommended software set  14.0.1 .

These errors happen right at startup. Memory use is steady in interactive mode, about 20 MB contiguous memory is unallocated out of 128 MB. I have taken pains to prevent allocation and follow cRIO besta practices non concatenated strings array building in loops etc). CPU is 50-60% when running interactive mode.

Really at my whit's end here, I have tried the various combination of enabling / disabling debugging in the RTEXE, disconnecting typedefs etc. I have not noticed a pattern. 

Regards,

MarkCG

[j_bou]

Hello MarkCG,

What minor changes did you make to the serial communication code? Posting the VI or a screenshot of the VI with the modifications which caused the RTEXE to stop functioning could be useful to troubleshoot the issue. 

Regards,

J_Bou

[AbandoningCausality]

I don't have a solution but I'm also seeing the same sort of "Bad build" issues on a 9066. Have you had any successs?

####
#Date: Tue, Apr 05, 2016 11:57:35 AM
#OSName: Linux
#OSVers: 3.14.46-rt46-ni-3.5.0f0
#OSBuild: 200238
#AppName: lvrt
#Version: 15.0.1
#AppKind: AppLib
#AppModDate: 


InitExecSystem() call to GetCurrProcessNumProcessors() reports: 2 processors
InitExecSystem() call to GetNumProcessors()            reports: 2 processors
InitExecSystem()                                      will use: 2 processors
starting LV_ESys2_Thr0 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr1 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr2 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr3 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr4 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr5 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr6 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
starting LV_ESys2_Thr7 , capacity: 24 at [3542702258.77927494, (11:57:38.779275000 2016:04:05)]
VI_BROKEN (0): [VI "nisyscfg.lvlib:Close (System).vi" (0x00059588)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "nisyscfg.lvlib:Close (System).vi" (0x00059588)]
this->flags=50340480, compilerError=6
VI_BROKEN (0): [VI "nisyscfg.lvlib:Error Cluster From Error Code (nisyscfg).vi" (0x00317cf8)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "nisyscfg.lvlib:Error Cluster From Error Code (nisyscfg).vi" (0x00317cf8)]
this->flags=50340480, compilerError=6
VI_BROKEN (0): [VI "nisyscfg.lvlib:Initialize (Helper).vi" (0x002f7d88)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "nisyscfg.lvlib:Initialize (Helper).vi" (0x002f7d88)]
this->flags=50340480, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:Get Numeric Information.vi" (0x002356a8)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:Get Numeric Information.vi" (0x002356a8)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "NI_Real-Time Target Support.lvlib:RT Get CPU Loads.vi" (0x002e3a70)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Real-Time Target Support.lvlib:RT Get CPU Loads.vi" (0x002e3a70)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "AbstractHMIRecipient.lvclass:Send error to HMI.vi" (0x0051e748)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "AbstractHMIRecipient.lvclass:Send error to HMI.vi" (0x0051e748)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "NI_Real-Time Target Support.lvlib:RT CPU Load.ctl" (0x002f1480)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Real-Time Target Support.lvlib:RT CPU Load.ctl" (0x002f1480)]
this->flags=34611712, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:Unit Info.ctl" (0x0024b360)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:Unit Info.ctl" (0x0024b360)]
this->flags=34611712, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:Unit Type.ctl" (0x0024a428)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:Unit Type.ctl" (0x0024a428)]
this->flags=34611712, compilerError=6
VI_BROKEN (0): [VI "nisyscfg.lvlib:Initialize Session.vi" (0x002fbc08)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "nisyscfg.lvlib:Initialize Session.vi" (0x002fbc08)]
this->flags=50340480, compilerError=6
VI_BROKEN (0): [VI "nisyscfg.lvlib:Language.ctl" (0x0031d090)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "nisyscfg.lvlib:Language.ctl" (0x0031d090)]
this->flags=34611712, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Advanced (DBL Array).vi" (0x00535fe0)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Advanced (DBL Array).vi" (0x00535fe0)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Derivative Action Buffered (DBL Array).vi" (0x00529598)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Derivative Action Buffered (DBL Array).vi" (0x00529598)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "NI_Real-Time Target Support.lvlib:RT Get Number of CPUs.vi" (0x002f0ed0)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Real-Time Target Support.lvlib:RT Get Number of CPUs.vi" (0x002f0ed0)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Get Advanced Error (DBL Array).vi" (0x0051f300)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Get Advanced Error (DBL Array).vi" (0x0051f300)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Resize 1D Array.vi" (0x0048eed0)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Resize 1D Array.vi" (0x0048eed0)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "NI_AALBase.lvlib:Mean (DBL).vi" (0x0044a2c8)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_AALBase.lvlib:Mean (DBL).vi" (0x0044a2c8)]
this->flags=50340352, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Get Advanced Error (DBL).vi" (0x0051f838)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Get Advanced Error (DBL).vi" (0x0051f838)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Integral Action Buffered (DBL Array).vi" (0x00508568)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Integral Action Buffered (DBL Array).vi" (0x00508568)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "NI_PID_pid.lvlib:PID Parameter Manager (DBL Array).vi" (0x00527a78)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_PID_pid.lvlib:PID Parameter Manager (DBL Array).vi" (0x00527a78)]
this->flags=50471424, compilerError=6
VI_BROKEN (0): [VI "AbstractHMIRecipient.lvclass:AbstractHMIRecipient.ctl" (0x0051e588)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "AbstractHMIRecipient.lvclass:AbstractHMIRecipient.ctl" (0x0051e588)]
this->flags=36708864, compilerError=6

[j_bou]

Hello AbandoningCaus…, 

Did these built issues start happening fairly recently? Did the issue start happening after a code change (similar to MarkCG)? Is the build error random or consistent?   

As quick and easy troubleshooting step, I would recommend creating a fresh LabVIEW project and importing all your code to this new project. Then, attempt to compile for your target.

Best Regards, 

j_bou

[MarkCG]

Hi J_Bou,

I don't think now that it's serial code that was the problem. Because the problem was intermittent- some builds worked and others didn't-- I latched on to that as the cause at the time because it was the latest change.

I tracked down ONE cause of these errors, and with NI reliably reproduced the bug and had NI issue a CAR number in response.

That cause was the following: using User Defined Variables in absolute reference mode. Changing every single shared variable node in my code to relative reference mode was the response and that seemed to clear this problem.

I also stopped using "separate compiled code from source". Doing that may not be necessary.

Regards,

Mark Garnett

[AbandoningCausality]

Yes, this is a fairly recent issue. The build change involved recompiling the FPGA and some changes in my FPGA-RT communication.

I do have code separated.

I also have over 100 IO node variables, so that's interesting. I would not want to try and switch them to "relative". Can you send me the CAR number so I can look it up.

Most of the time, I see this issue after the project reboots the target after deploying the EXE. Often I can hard power cycle the target after than and the exe runs normally.

I will try a fresh project next time I see this issue. It's not all that easy to go fully fresh because I have extensive FPGA config (FIFOs, Memory, etc)

[MarkCG]

"CAR#571106: rtexe loads broken on target due to absolute reference mode for UDV".  I don't believe you need to do a fresh project. 

[railang]

This also happens on my sbRIO 9607. Looks like things go wrong when I load the FPGA.

The FPGA VI is empty. The RTS VIs send syslog trace to loghost kdmws04.

Wenn running this VI fom dev environment I get these logs:

Time	Priority	Hostname	Message
23:03:56	User.Info	192.168.3.172	0.256 kdmws09:RTS Untitled 2 Closed FPGA
23:03:56	User.Info	192.168.3.172	0.254 kdmws09:RTS Untitled 2 After loading FPGA ref is 0xB5202E80
23:03:55	User.Info	192.168.3.172	0.000 kdmws09:RTS Untitled 2 Before loading FPGA to device RIO0

As an rtexe I get this on RIO reboot:

Time	Priority	Hostname	Message
23:06:11	User.Info	192.168.3.172	0.000 kdmws09:RTS Untitled 2 Before loading FPGA to device RIO0

From the RIO error log:

VI_BROKEN (0): [VI "KDM_Trace.lvlib:Syslog Device Send.vi" (0x0011fa38)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "KDM_Trace.lvlib:Syslog Device Send.vi" (0x0011fa38)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:Data Type.ctl" (0x00188458)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:Data Type.ctl" (0x00188458)]
this->flags=101720576, compilerError=6
VI_BROKEN (0): [VI "TACOS_RTS.lvlib:Trace.vi" (0x00180e00)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "TACOS_RTS.lvlib:Trace.vi" (0x00180e00)]
this->flags=50340352, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:MDTFlavorToTypeEnum.vi" (0x00189a28)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:MDTFlavorToTypeEnum.vi" (0x00189a28)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:Type Definition Info.ctl" (0x0018e348)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:Type Definition Info.ctl" (0x0018e348)]
this->flags=34611712, compilerError=6
VI_BROKEN (0): [VI "NI_Data Type.lvlib:Get Type Information.vi" (0x0016e958)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "NI_Data Type.lvlib:Get Type Information.vi" (0x0016e958)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "KDM_Trace.lvlib:Variant to string.vi" (0x00153a28)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "KDM_Trace.lvlib:Variant to string.vi" (0x00153a28)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "KDM_Trace.lvlib:Buffer.vi" (0x0011d418)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "KDM_Trace.lvlib:Buffer.vi" (0x0011d418)]
this->flags=33563136, compilerError=6
VI_BROKEN (0): [VI "KDM_Trace.lvlib:Build Info.vi" (0x00140a78)]
VirtualInstrument::SetOrClearBadVILibrary - now VI is bad on [VI "KDM_Trace.lvlib:Build Info.vi" (0x00140a78)]
this->flags=33563136, compilerError=6

This points to the VI vi.lib\Utility\Data Type\Get Type Information.vi, inside the RTS-Trace.vi.

While reducing the application to this code I often need to build and deploy the rtexe. There is one observation I can see at 10 % of the builds: When the rtexe build is finished, i get an popup the the dev environment has lost the connection to RIO. I'm talking about the build, not the deployment. The RIO seems to reboot and I get correct logs once. The next RIO reboot fails again.

Any suggestions?

Thanks

Rainer Langlitz

[TheEngineeringBear]

Hey @railang,

Is this error happening with just your code?  Try building an example and deploy it to the sbRIO.

Are you using the always include feature for the files necessary for your deployment in the build specifications?  I would make sure that you have all the files you need on the sbRIO and the code is set to find them not on the host computer but on the sbRIO.  It sounds like that is the issue you are seeing.  Try setting this in Build Specifications.  Also use the "Always include" feature.

As for the reboot, it's a common error the sbRIO needs a reboot randomly after building and deploying.   

-Ben

[railang]

Hi Ben.

This also happens with an example implementation from scratch and with a second RIO hardware. Meanwhile I have found out: The FPGA loading blocks the ethernet interface for about 3 seconds. Because of this the logs get lost. I don't know how this is related to the broken code error report. But it seems not to be the same root cause like the CAR mentioned in this thread.

My actual workaround is: Wait three seconds after FPGA loading.

Regards

/rainer