AWS IOT on cRIO - dependency version clash with OpenSSL?

[User002]

I'm trying to get a fleet of cRIOs to run in concert with AWS' IoT "framework", for lack of a better word.  Being a framework of sorts, it brings a bunch of tooling for running a fleet of IoT devices according to best practices (security, etc.) and is practically infinitely scalable. 

I've examined the LabVIEW Cloud Toolkit for AWS by NI ( https://sine.ni.com/nips/cds/view/p/lang/en/nid/215508  ), and while the toolkit works for communicating with certain AWS interfaces in general (S3, SQS, and a couple others over HTTP), it does not work with the IoT framework.

The main reason is that devices are meant to communicate over MQTT.  What's more, the TLS session is mutually authenticated by both the client and the server.  TLS is barely available on some platforms in LabVIEW, so no MQTT library has this even close to implemented (I've checked).  Luckily, these sorts of security-related items and more are already implemented for us in AWS' collection of C libraries:

https://github.com/aws/aws-iot-device-sdk-embedded-C

As simple as build and install?  Almost!  But alas, no.

The one dependency I can't satisfy is OpenSSL, which needs to be 1.1.0 or greater.  Everything else works (I've tried it) NI Linux RT only has 1.0.2 in the repos.  Thus my questions:

• Is there a way I can use OpenSSL 1.1.0 for this?  I suspect if I try to compile it from source, I'll run into terrible problems.  I know some key libraries in NI Linux RT are chosen for compatibility with things such as glibc - and know at some point I'm more likely to have success rebuilding the whole OS from bitbake recipes.  That's a very, very deep rabbit whole that I don't know if I'm ready for (or that the client can afford!).
• Alternatively, does anyone know of an another way to communicate as a "Thing" with the AWS IoT framework?  It appears that even their http calls are mutually authenticated.
• I did find that earlier versions of the AWS IoT SDK relied not on OpenSSL but on mbedTLS.  It looks like I could get that working relatively easily... but I don't know how long a runway that has.  You generally have to keep up with AWS because they don't keep old versions of APIs around forever.

I suppose my alternative is to construct the entire application in Python.  There's the NI libraries for FPGA (DAQmx doesn't work on RT), and there's the AWS libraries in Python.  As a side note there, the Python base that NI provides is nearly deprecated and I'm hoping there's an upgrade path before too long there too.

[Taggart]

Don't know much about the AWS IOT framework, but you mentioned it has a python library. You can call python from LabVIEW, so maybe just do that? You could also just have a seperate python app that communicates to the AWS MQTT server and relays those messages over localhost to your LabVIEW program.

[User002]

Thanks for the reply, Sam.

This is on a real-time cRIO.  I know there's at least two great ways to call python on Windows (Enthought library and now the built-in nodes).  But if there's a way to do it on RT, please let me know.  I could only find suggestions in the idea forum for it.  The built-in nodes break if you move them under an RT target, too.

I could probably call multiple scripts to pass info to python, but that gets ugly fast since I'm hoping to pass a lot of data and each sysexec call spins up a new bash shell session.  The other way is to start a python script and pass everything over local networking (or unix pipes).  That path is clunky in its own way, because now I have my application split across two very different toolchains and long-term maintenance and debugging gets to be ugly. 

I'm a bit disappointed that RT doesn't have a way to call directly from LabVIEW to python.  Being built on Linux it should be almost painfully easy. (I say "should" - but of course that's expectation compared to everything else on Linux rather than technical lift to get LabVIEW there).

Please correct me if I'm wrong - I would LOVE to be right now!

Thanks,

[Taggart]

Andrew, 

                  I don't have a secret on how to call Python in LabVIEW RT. I thought there was some trickery involving ssh or something. Part of the problem is that LabVIEW RT runs in a chroot.  I think the solution is to run sshd outside the jail and ssh into the main linux instance and then run your python commands. Some googling might reveal a post or two on the forums about that. I remember seeing it somewhere.

I don't know enough about pipes to know if pipes work across a chroot, but I imagine localhost would.

I agree that adds an extra layer, I guess you could spin it as a feature. It's more modular or an abstraction layer or something.

Sam

[User002]

Sam,

Good point about the chroot.  I knew it LV code ran as lvuser, but didn't realize it was a chroot.  I guess that may make it harder to run the main python script from there.  It's easy enough to run it with a startup script and connect over local TCP, but still a bit of a hurdle.

However, I'm beginning to think that it might be easier to run without LabVIEW at all.  If we can't get the C AWS libraries working, I'm beginning to think the total effort would be far less if we were to simply use python.  Python to call the AWS libraries, and to call the FPGA API (since DAQmx doesn't work on cRIO, but the FPGA VI does).  90% of what we want to do is encompassed in acquiring a waveform and sending it in a (properly authenticated) MQTT message. 

My worry with python on cRIOs these days is that the version in the repos is nearly deprecated.  I don't have it in front of me, but I think it was python 3.5.5, from around 2018.  When I ran pip update, it gave me a deprecation warning.  With the way AWS keeps up to date, that may not have a very long runway unless NI updates the package repos soon.

Thanks,

Andrew

[Taggart]

Hmm.. As far as python versions, I know Ubuntu is very far behind, but there is deadsnake?  PPA you can add and pull the latest version from there.  I know the ni linux rt stuff uses opkg, maybe there is an opkg repository you can pull the latest version from?

If not, I guess you would have to build the latest version?  That sounds like a pain. I would ask around and see if someone else has already put in the leg work.

[User002]

Thanks for the thoughts, Sam.

To close the loop on SSL, NI support got back to me.  No luck - it's bound to dependencies that can't change.  That C code won't run on the cRIO.  I imagine the only possible way to do that would be to rebuild the entire OS with a custom version - and very likely navigate a whole different world of dependencies (that likely wouldn't work with the NI hardware drivers).

So one way or another, it's looking like the python route for us.

It's disappointing how hard it was to integrate such great hardware as the cRIO into something so ubiquitous as AWS.

Thanks,

Andrew

[rtollert]

Hey, I'm the schmuck you can blame for 1.0.2 still being on the target. I don't have the time to poke my head around here that often but I do what I can.

I took a look at the AWS IoT SDK you mentioned, and srsly? The OpenSSL dependency you cite was added seven months ago. (And no TLS support even appears to have existed prior to that commit.) This is an extremely new library. Don't get me wrong, I want this situation to change, and soon. I strongly appreciate that OpenSSL 1.0.2 is no longer supported upstream. And you are not even the fifth person to ask about this. But super-hard dependencies on OpenSSL 1.1 remain a little uncommon. (Obviously that will change quickly.)

The moment that you start doing C development, you are, as far as NI support is concerned, at least somewhat off the reservation. If SSP included OS support and/or C development support for NI Linux RT, you would probably be paying us a lot more money. 😉

You probably stopped too soon getting this working. Just because NILRT installs OpenSSL 1.0.2 system-wide, and will probably not appreciate an upgrade to 1.1, ought not to prevent you from building OpenSSL 1.1 yourself and statically linking it into your shared lib.

Now, that said... that MQTT so closely relies on client certificates (and that this would impact LabVIEW developers this quickly) is something of a surprise to me. (In the wider HTTPS world, it's very hard to find great examples of client certificate deployment outside of slightly esoteric enterprise environments.)

If the following request sounds incredibly dumb, that's because it is incredibly dumb, but... even though LabVIEW 2020 TLS support doesn't include client certificate support... could you still try it anyway? The feature got kicked out of the release, and nobody's attempted it yet, but I'm staring at the code right now, and I'm not seeing any obvious reason why it wouldn't work. The TLS functions support both client and server operation, and while adding a private key is primarily meant for server use, all of that happens before the code even knows if it's about to act as a client vs server.

"TLS is barely available on some platforms in LabVIEW." Insofar as you're talking about NI Linux RT, I don't think I understand this comment.

[User002]

Thank you for taking the time to comment.

---

I took a look at the AWS IoT SDK you mentioned, and srsly? The OpenSSL dependency you cite was added seven months ago. (And no TLS support even appears to have existed prior to that commit.) This is an extremely new library. Don't get me wrong, I want this situation to change, and soon. I strongly appreciate that OpenSSL 1.0.2 is no longer supported upstream. And you are not even the fifth person to ask about this. But super-hard dependencies on OpenSSL 1.1 remain a little uncommon. (Obviously that will change quickly.)

It is a new requirement in this library, yes, but it looks like (if I remember correctly) that OpenSSL 1.1 was released in something like 2018.

---

You probably stopped too soon getting this working. Just because NILRT installs OpenSSL 1.0.2 system-wide, and will probably not appreciate an upgrade to 1.1, ought not to prevent you from building OpenSSL 1.1 yourself and statically linking it into your shared lib.

Interesting. I assumed that because it and a few other key libraries like glibc were pinned, there were compatibility reasons for holding them steady.  From that, I figured that such a static linking was doomed to fail because of those other compatibility issues.  I'm still a bit reluctant to try it unless you feel pretty sure it's probably ok.  But if it is likely to succeed, that would be great news!

---

Now, that said... that MQTT so closely relies on client certificates (and that this would impact LabVIEW developers this quickly) is something of a surprise to me. (In the wider HTTPS world, it's very hard to find great examples of client certificate deployment outside of slightly esoteric enterprise environments.)

Yes, was new to me.  But from what I can tell, it also appears to be becoming best practice in the IoT space - making the other NI-supplied AWS IoT library too deprecated for this use case. ☹️

---

If the following request sounds incredibly dumb, that's because it is incredibly dumb, but... even though LabVIEW 2020 TLS support doesn't include client certificate support... could you still try it anyway? The feature got kicked out of the release, and nobody's attempted it yet, but I'm staring at the code right now, and I'm not seeing any obvious reason why it wouldn't work. The TLS functions support both client and server operation, and while adding a private key is primarily meant for server use, all of that happens before the code even knows if it's about to act as a client vs server.

---

Fascinating.  Looking at it again, it does make sense.  I might give this a try - hmm but to use it, I'd need to write such a module into whatever LabVIEW-native MQTT library used. 

As to why my frustration may have leaked out in my post - the cRIO's software compatibility is really close to having this be an easy project, but the ecosystem falls short in a bunch of little ways.  None of them is a single major flaw, but they seem to line up perfectly with each other and combine to nearly block me completely.  All the project really needs is an MQTT library that has good send queuing (because the link out there at the IoT edge is sometimes spotty), but none of the native LabVIEW libraries I could find supported that.  Well, and of course the mutual TLS authentication.  After spending years in systems engineering at NI and recommending cRIOs to our top customers and seeing the amazing applications they excel at (and being a huge fan of cRIO in general), this felt a bit like a letdown.

If ut if TLS might be made to work, then I just need a good MQTT library.  Hmm... still lots of work to integrate that - doing this in python still might be the way to go.

Might you have any insight into more recent python libraries on RT?  (I'm fine moving to the newest (say, LV RT 2021) if I need to do that to get it).

Thanks,

[User002]

---

"TLS is barely available on some platforms in LabVIEW." Insofar as you're talking about NI Linux RT, I don't think I understand this comment.

---

Apologies if this was unclear - I meant barely in time rather than barely in functionality.  As if to say that since it took however many years to get TLS in LabVIEW, that it might take many more to get support for whatever comes next.

Forgive my indulgence in this next statement because it is not entirely justified because this case alone, but rather as a conclusion from a much broader set of very similar experiences.  So often NI and LabVIEW appear to have a need to re-implement things themselves rather than pull in compatibility to a 3rd party library.  From where I sit, it seems that if NI took a fraction of the cost they spend on rebuilding tools within LabVIEW and instead spent it on improving certain kinds of compatibility, the ecosystem would be very much better off.

Thanks,