STM vs shared variable

[jyang72211]

From what I have seen on the forum so far, it seems to me that compared to shared variable, STM is better with different data type and bigger data size. The only thing that makes shared variable slightly better than STM is being a bit more simple.  If that's the case, why use shared variable at all?  Are there aplication where shared variable is a lot better than STM?  

[soupy · Accepted solution]

we've been using shared variables/datasocketfor the past 5 years and they work well. In the simplest form, they're extremely easy to use. I highly recommend wrapping them in some logic to give you the ability to detect connection status and re-establish connection if it is lost (The simple implementation will throw an error and not reconnect)

The biggest risk in using this protocol is that NI holds all of the cards. You have no guarantee that they won't introduce critical bugs in the next upgrade, or phase it out entirely (aka datasocket). Also, when you identify obscure bugs/features with the protocol, NI will be responsive, but, chances are, you won't be able to get a concrete timeline on the fix. This is not very cool if this is the backbone of your system.

Because of the risks above, i've started looking into using TCP/IP primitives to do all of our network communication. I have a theory that they're super-stable and somewhat future-proof (except maybe for ipv6). Can anyone back me up on this?

[soupy]

oh, one other thing. the HUGE difference between the two systems is that STM is a "messaging" system and shared variable is a "variable" system. in SV, once you publish a variable to the SV server, it will always be there for all clients to use with no further action required by the publisher. in STM, you'll only be able to get data from the host if it sends it to you. Also, SV inherently supports multiple subscribers, while STM only supports a single connection.

If you are publishing variables via STM, you would have to set up your own query/response protocol to emulate what SV does. 

If you are publishing messages via SV, you'll have to set up buffered reading/writing and accept the fact that it will drop messages without warning if the buffers overflow. (I think that network streams are better than SV in this case)

[Ben]

Adding to rockethacker's most excellent reply...

SV's are the heart of the DSC add-on that has historical logging trending alarming SQL DB and some other features.

Ben

[nathand]

---

@rockethacker wrote:

Also, SV inherently supports multiple subscribers, while STM only supports a single connection.

---

 STM can support multiple connections, although you'll need to write some code to make it happen.  It would be very similar to writing code to support multiple connections with the TCP primitives.

---

rockethacker wrote:

Because of the risks above, i've started looking into using TCP/IP primitives to do all of our network communication. I have a theory that they're super-stable and somewhat future-proof (except maybe for ipv6). Can anyone back me up on this?

---

 Well, the STM stuff is all built on the TCP primitives, so it should share those same advantages.  There's a converse argument to be made that with shared variables, if the transport layer does change, NI can make those changes "under the hood" and you won't need to modify your code.  The TCP primitives are wrappers around calls to the operating system and the shared variable server must use those same operating system calls, so at a low level there's not a big difference.  The TCP primitives give you a lot of control over the network connection, but making it reliable (able to handle timeouts, multiple connections, dropped connections) is not simple.  If you don't want to do all that, and you trust the work that NI's engineers have already done for you, then stick with shared variables.

[soupy]

a few more notes:

• setting up firewall rules for shared variables are little more tricky than STM http://zone.ni.com/reference/en-XX/help/371361D-01/lvhowto/psp_firewall/ This can be an issue if your IT department is uber-paranoid.
• Yup, you can set up multiple connections with STM, but it does not come with the stock examples. You'll have to do quite a bit of development to make it happen. This is the implementation i plan to go with, and the work involved is what is preventing me from going this direction.
• Yup, at low level, they are both TCP. It is the stuff stacked on top of it that i'm worried about.
• We're still investigating this, but we believe that we've found a rare case where certain network events can cause applcations reading shared variables to crash and/or leak memory. Again, we're still digging into this to fully understand it. Instances like this is why we are looking more into STM over SV. We found similar bugs/issues when were using datasocket and they were never fully resolved. Eventually NI's solution for us was to move to shared variables. We moved and those issues were solved, but, as with all new products, other bugs were found. Since NI is continually evolving SV, I don't anticipate the bugs fully going away. I'm not knocking the NI guys, this is just the nature of developing new products.

Disclaimer: The issues we've found are very rare and usually don't effect the average user. In our case,we're testing large rockets and the network connections are the life lines to the control room. An interruption of network communication at the wrong time could place personnel and/or hardware in danger.

BTW, I just changed my screen name today from rockethacker-->soupy 

[nathand]

---

soupy wrote:

• Yup, you can set up multiple connections with STM, but it does not come with the stock examples. You'll have to do quite a bit of development to make it happen. This is the implementation i plan to go with, and the work involved is what is preventing me from going this direction.

---

Getting off-topic a bit from the original question here, but I wrote a server in LabVIEW that used the TCP primitives and handled multiple connections.  The same approach should work for the STM, using STM connection info clusters in place of connection references.  In case it's of use to you, here's what I did (unfortunately I no longer have access to that code, it was several jobs ago):

- have one loop dedicated to listening for new connections.  When one arrives, put it in a queue and go back to listening.

- in another loop, maintain an array of connection ids.  For each connection ID, also store a timestamp.  When a new connection arrives, dequeue it, bundle it with the current time, and add it to the connection ID array.

- inside the same loop, add another loop to iterate through each connection, checking for data.  If there's data, do whatever you need to with it (usually enqueue it for some other loop to handle), and update the timestamp associated with that connection ID.  Ignore error 56 (timeout) since any connection that doesn't have data will report this error.  For any error that indicates the connection is closed or no longer available, remove that connection from the array.  If the difference between the timestamp for a connection and the current time is greater than some amount, close the connection and remove it from the array (this is optional, but prevents you from leaving connections open that haven't been cleanly closed).

If you're using a command-response scheme, and you pass the data off to another loop to handle, you'll need to track the connection with which it is associated so that the response can be sent to the right place.  I was using "Variant to Flattened String" and "Flattened String to Variant" for data passing over TCP, so I just tacked on the connection information as a variant attribute, but there are many other ways to handle this.

[soupy]

Nice design! Especially the variant attribute trick!

In the design i'm contemplating, i'm using STM-UDP to receive connection requests. so i can then compare them to an access list and then respond back with the TCP port that i would like them to use, or an "access denied" message. The reasoning behind this is to separate out the connection requests so they don't impede the TCP traffic.

[nathand]

---

@soupy wrote:

In the design i'm contemplating, i'm using STM-UDP to receive connection requests. so i can then compare them to an access list and then respond back with the TCP port that i would like them to use, or an "access denied" message. The reasoning behind this is to separate out the connection requests so they don't impede the TCP traffic.

---

That sounds excessively complicated to me, but maybe I don't understand your goals.  Why do you want each client to use a separate TCP port?  The connection requests won't interfere with TCP traffic, and dealing with multiple ports means having multiple listeners and added compilcations.  In the approach I described, waiting for new connections is handled in a separate loop, but it waits for TCP connections on a single port and hands off each connection to the communication loop once established.  It wouldn't be hard to add access list checking in that loop, prior to handing off the connection id.

[jyang72211]

I have learned a lot from the discussion.  Thanks!