Appear to lose connection with the Shared Variable Engine

[thisisnotadream]

Have been through the boards, but with no success regarding the specifics of my problem.

I am running a real-time application on a PXI-8108.  The host software is taking care of user interface, etc.  Information exchange is handled using Unbound Network Shared Variables (located on the host, though problems occur no matter where the variables are located).  Everything deploys fine.  Software runs swimmingly.  I have backed off the DAQ/communication processes until the RT cores are at <10% busy.  However, occassionally while running, I get a popup "Waiting for Real-Time Target xxx to respond..."  If this message stays up for long enough, then the target's attempts to read the Shared Variables start erroring out with an error -1950679035.  As the variables have already been deployed and used prior to this instant, I find this curious.  If I filter out the error, eventually the real-time software comes back up and everything progresses as though nothing happened.  However, this is a potential deal-breaker for the users who can be impatient with having to wait a few seconds for the software to respond.

Any idea what is happening?  I'm not pumping huge amounts of data over the network, no threads should be starving.  Kinda confused.

Dan M. 

[Will_S.]

Hi Dan,

It does still sound like threads are starving. How have you architected your RT application? Are you using While Loops or Timed Loops? Network communication works on lower priority threads and can be ignored causing network connectivity issues if there is not enough time for the CPU to complete all tasks. Are both cores at <10% when you receive this error?  Just in case, you may want to add a wait function in your While Loop or extend the loop period of your Timed Loop.

Error -1950679035 can stand for a couple different things, what text description do you get with this error code?

Are you using the DSC module?

[thisisnotadream]

The shared variables are being used much like how a queue would, directing each loop to perform an action should the host require it.  If no action is required, then the loops default to check their respective bus or DAQ.

Currently, I am running 13 timed loops (whose periods I have extended signficantly during debug to no detrimental effect of my software, but to no improvement in the issue in question):

• RS-232/J1708 converter loop (50ms period)
• CAN loop (50ms)
• DMM loop (250ms)
• Oscilloscope loop (250ms)
• DAQ loop (50ms, includes 1 AI task, 4 DI tasks, and 6 CTR in tasks)
• Switch loop (50ms)
• RS-232 loops (50ms for the main, 1 additional loop at 1Hz)
• RS-485 loops (50ms for the main, 4 additional loops at 1Hz)

Both cores appear to share the load, usually the aggregate load is under 15%.  When the popup appears telling me that the host is waiting for a reply from the target, the PXI controller monitor output still updates the CPU loads, which do not show anything unusual, such as a spike or a freeze in loading.  The target loading does spike up to about 30% total after the communication is reestablished, but that appears to be the software attempting to "catch up" with the events that occurred during the "black out". 

The error reads like this:

Shared Variable in Main Target Application.vi

This error or warning occurred while reading the following Shared Variable:

\\My Computer\Host-Target Interface\RS-485 Messages to Target    [<--insert other variables here]

\\192.168.1.122\Host-Target Interface\RS-485 Messages to Target

Unable to locate variable in the Shared Variable Engine.  Deployment of this variable may have failed.

Are there any other drivers that would be starving the Shared Variable Engine?  For example, if a VISA or NI-DAQmx driver goes to sleep, could it hold up the SVE?  What execution system are they running under? 

Thanks for your help!

Dan M.

[thisisnotadream]

PS--NOT using DCS module.

[thisisnotadream]

Another few possibly interesting things to note. 

• The target does not throw the error until after the host returns to its senses, ie--the screen returns to user control and the "waiting" window disappears (implying a return of communications)
• Even more curious:  I noticed today that, even if the target software is not running, I still get the occassional "Waiting for Real-Time target to respond..." message on the host.

[Will_S.]

Could you please post some basic code that exhibits this issue so we can reproduce it?

[thisisnotadream]

I'm not sure that I can post as much as would be needed as it would reveal a good deal of proprietary information.

However, I have not been just sitting on my hands.  The problem appears to be on the host side, not the target side, as the host seems to be indicating.  The problem still arose after I had removed all target code but one loop running at 50ms period and relatively low priority.  So I turned my attention to the host.  It appears that a portion of my host code (which performs a peripheral, rarely-used function) seems to cause the problems.  Disabling that portion of code removed the problem.  I tried running the problem loops slower (from 50ms to 200ms) and that seemed to help.  It appears that these modules were starving the network thread.  I may have to play around with priorities and threads to completely mitigate the issue.

Does this make sense to you, that the host would be the culprit?

Dan

[Will_S.]

It could occur if the code is used to directly affect the communication between host and target. What happens in this function? Does it communicate with or modify the shared variables in any way? How often is it used? Perhaps you can set flags in your program to turn on when the function is called to see if it coincides with your loss of communication.

[rex1030]

Also, if some VIs are writing to the shared variables faster than the subscribers (readers) are reading from it, you may be getting buffer overflows.

PS- I am totally stealing your strongbad icon.

[thisisnotadream]

I'm not in front of my code right now.  The shared variables are basically queues between the host and target VIs.  One variable per direction per code module.  Most of them are single writer variables, and I'm writing to them relatively slowly, about 50ms.  One question, the elements in this pseudo-queue are usually on the order of a handful of bytes of string data.  I came across the Flush Data.vi.  Is it generally considered a good idea to use this VI to post the data as soon as possible?  I know it would take some time to reach 8kB of data for some of these...maybe not 10ms, but I don't know.

Dan

PS--Be my guest!  Strongbad rulz!