Large data in network shared variables - Error 180121605

[IanDeaville(ZF)]

Hi,

My main RT LabVIEW system keeps finding new ways to puzzle me.  I have a PXI-8106 controller linked via private network to the host.  The PXI runs the machine & collects data from short-run tests.  When the data is ready, the PXI packages it into a custom cluster as a network shared variable.  It sets a Boolean NSV flag to indicate that there is new data for the host to read.  Fine.

Some of the tests are now running longer, with data sets of 10-15MB.  Occasionally one or more data sets in a series fails to be seen by the host, although the PXI claims to have processed them as normal.  One of the few clues I have is Error 180121605 - 'OS and Network Services: (Hex 0xABC7005) Variable data is being written faster than it can be transmitted. This may result in data loss.'.  I am losing data.  

I can see evidence of other parts of the system slowing down as the data is being handled, but I can accept this, as it occurs as a predictable time in the overall sequence.  What can I do to improve the reliability of data transmission here?  

NSVs suit the use case - low priority, occasional use only.  I don't want / need to stream data to the host, nor do I want to develop my own protocol.  Any other suggestions?

Individual data points are queued at test run-time.  At the end of the test the queue is flushed, and the data packed into the custom cluster.  This is then passed into a queue within a functional global variable.  Elsewhere a low-priority loop monitors this FGV and passes the cluster to the NSV.  Running LabVIEW 2018, but had similar behaviour on 2014 also.  Processors are quite busy - 80-85% loading.  Plenty of memory available and the network is quiet.  

Thanks all,

Ian

[Ben · Accepted solution]

---

@IanDeaville(ZF) wrote:

...

NSVs suit the use case - low priority, occasional use only.  I don't want / need to stream data to the host, nor do I want to develop my own protocol.  Any other suggestions?

...

  Processors are quite busy - 80-85% loading.  Plenty of memory available and the network is quiet.  

Thanks all,

Ian

---

I can not speak to that issues since I do develop my own protocols.

Can you break up the data transfer into smaller parts?

I have read that NSVs are questionable under heavy CPU loads.

Ben

[Mark_Yedinak · Accepted solution]

You can try using the network queues to transfer your data. Use them like you would a LabVIEW queue. I haven't done a lot with data the size you mention but have transferred messages that were about 1MB. Our use of these queues is more ina publish/subscribe messaging system where we have small messages but lots of them.

[IanDeaville(ZF)]

Ben, Mark,

Thanks both for your input.  It seems that I have reached the limit of reliability for NSVs, so I need to find another way.  I will look at options to divide the data into chunks - ideally as individual channels (waveform), with either a modified NSV or Network Queue process to pass them from PXI to Host.  

Reducing CPU load would also be useful...I need to manage DAQ options differently for that...

Regards,

Ian