Protocol Buffers (ProtoBuf) structured data serialization for use in communications protocols

[TroyK]

Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data. Like XML or JSON but smaller, faster, and simpler.

LabVIEW already has XML & JSON functions, why not ProtoBuf?  It is the default mechanism for serializing structured data used by gRPC.

If LabVIEW users want to create gRPC microservices, then LabVIEW needs to support ProtoBufs.

[thols]

Interesting, but wouldn't this be more suited as an add-on? Does it really have to be an integrated part of LabVIEW? 

[TroyK]

---

@thols wrote:

wouldn't this be more suited as an add-on?

---

Maybe? But XML <> string and JSON <> string functions are built-in functions.

[AristosQueue (NI)]

@TroyK:

We needed a new JSON library because the built-in support frankly sucks. In LabVIEW 2019, we added a new entry to the palettes -- one that installs a third-party JSON library when you click on it. We did that instead of adding more built-in support for JSON. LV has grown as a language such that we can do most work in VIs just as efficiently as we could in the built-in functions. Going forward, I suspect the majority of requests like this would be add-on toolkits.

Now, NI might still be the one to develop that library, so this idea stays open, but @thols' point stands.

-- AQ

[rd-car-o]

I have looked at this a bit and found a couple problems with trying to make this an add-on, as opposed to a built-in feature.

1. Protobufs is not a serialization technique. It's a memory-sharing technique which happens to happen through a network. The power is in the fact that you don't need to marshall/unmarshall data. Just wholesale replace your object with the new one from the network buffer.
2. Given that, in LabVIEW it would be quite difficult to do that without copying data quite a few times, thereby undermining the efficiency gains.
   1. Copy 1 - From the network read itself you get a string which I am guessing is actually, itself, copied from the NIC buffer.
   2. Copy 2 - There is no way to 'wholesale replace' an object in memory with some other data in G. (Feel free to shoot that down)

It's worth mentioning at this point, Un/Flatten To String:

Flatten to string, which NI invented a long-long time ago, is actually very similar to protobufs with the exception that it isn't defined by some external file. While there is documentation about the method they used, users cannot change it and it is therefore largely unable to be shared. The technique, though, is fantastic and actually between 13-25% more dense than protobufs because NI can make certain optimizations which protobufs cannot. To use it, though, users must copy data at least once but probably twice before using it in G.

So might we do?

In order to truly make this protobufs-idiomatic, we would have to do something like this:

Make a variation of the Network Read and Un/Flatten to String which can accept a "pointer" to a G object. This new primitive's job is to take data off the NIC buffer and stuff it under this G Object pointer. One Copy. Couple that with a .proto file which is parsed at compile-time to reconfigure this magical Primitive and we have ourselves true protobufs integration.

This would be quite challenging in G as-is, for a number of reasons. I tried this, sort-of, by creating a .proto which matched NI's flatten-to-string paradigm and used flatten/unflatten to/from string on an object and things lined up quite closely... One major shortcoming was in my inability to leverage type-propagation in my own VIs. Hence, difficult to deliver as an add-on. Maybe now with Malleable VIs? Also without the recompile-ability it's not very attractive to have an "ni-specific" proto file, kinda undermines the technique.

[PhillipBrooks]

A recently announced roadmap (https://forums.ni.com/t5/LabVIEW/LabVIEW-Roadmap-2022/m-p/4218319/highlight/true#M1223209) indicates that Native gRPC server/client interfaces will be added to LabVIEW.

By design, gRPC uses protocol buffers.  Will this idea be changed to "in development"?

[Christina_R]

The LabVIEW gRPC work is at https://github.com/ni/grpc-labview. I don't believe that this project "satisfies" the request for this Idea, but if I'm wrong please let me know!

[Manucla]

Is the grpc-labview github code tested in LabVIEW for Ubuntu ?

[sujain]

The gRPC LabVIEW git hub project (https://github.com/ni/grpc-labview) does support Linux. The code is distributed as a VIPM package. Some unit tests(https://github.com/ni/grpc-labview/tree/master/tests) are already there but there are no end to end tests as of now.