Allow users to define string lengths and array dimension sizes in Type Descriptor when unflattening variant from string

[m3chtroid]

Flattened String To Variant is so close to letting us define the layout of arbitrary external data.  However, labview's insistence on all strings and arrays being prepended with their lengths in the data string means when working with binary data from non-labview sources, the array or string lengths might have to be calculated or are somewhere not immediately adjacent to the data.  I understand it's far too late to modify the base unflatten from string or read from file as requested in this idea, but this feature would be incredibly useful for reading binary data directly into clusters without requiring either multiple copies of incredibly large (100,000+) arrays floating around, or incredibly tedious and complex sequences of Read from Binary File nodes.

[AristosQueue (NI)]

If the length of the string isn't given, how does the original file intend for you to compute the length of the string to allocate memory? Is it NULL terminated data? That works for strings... for arrays, what does the sentinel look like?

Maybe I am misunderstanding your request, but it seems like what you're asking for is impossible. The "incredibly tedious and complex sequences of Read from Binary File nodes" is what's necessary to specify an arbitrary data format... I don't know of any way that LabVIEW could just intuit that if you have a cluster with arbitrary string data embedded in it.

Can you provide an example format that we can discuss further?

[m3chtroid]

Oh, the length of the strings and arrays are given, but they're not immediately preceding the data like labview assumes.  Take one of the simpler chunk definitions shown below.  As soon as I have the chunk tag, I know exactly which offsets to read to get the size of the chunk and the lengths of the name and property strings.  With my suggestion, I could combine http://zone.ni.com/reference/en-XX/help/371361B-01/lvconcepts/type_descriptors/ and the known lengths into a type descriptor array that would allow me to use the Flattened String To Variant vi to unflatten the entire chunk of binary as a cluster of labview objects, allowing me to peruse them at my leisure.  Right now, I have to advance slowly through the file, reading a handful of bytes at a time and then throw everything into a cluster at the end.

 (As to be more specific about my change, I'd like to be able to encode the lengths of things like strings and array dimensions in the type info instead of being forced to use FFFF FFFF and reading the element length from the data string, as pointed out here in the above type descriptor link:

)

[m3chtroid]

To add a bit more, this would be the uint16 Type Descriptor array that would define the 8 element cluster of a property chunk, described below:

[0050, 0008, 0004, 0008, 0004, 0008, 0004, 0008, 0004, 0007, 0004, 0007, 0004, 0007, 0008, 0030, XXXX, XXXX, 0008, 0030, XXXX, XXXX]

0050 0008 -- Cluster, 8 elements
0004 0008 -- 64 bit int (actually an 8 char string, but the length is constatn, so *shrug*)
0004 0008 -- uint64_t cbChunk
0004 0008 -- uint64_t objectId
0004 0007 -- uint32_t DATATYPE (actually an enum, but for the sake of space I won't include the entire enum definition)
0004 0007 -- uint32_t cbName
0004 0007 -- uint32_t cbProperty
0008 0030 -- String Name
XXXX XXXX -- length of Name, X, where X would be filled in programmatically. (Currently I have to define the length as FFFF FFFF and read it from the first 4 bytes preceding the string itself in the data string. Which doesn't match the type defintion.)
0008 0030 -- String Property
XXXX XXXX -- length of Property, X, where X would be filled in programmatically. (see above)

You can see how it'd be used with the proposed changes in the chunk parser vi snippet. (I appear to be unable to add an attachment?)

[AristosQueue (NI)]

Ok, m3chtroid, now I see what you're asking for.

The request makes sense. I doubt we act on it -- the type strings are problematic for us because the int16 format cannot handle a fair number of type descriptors (the first int16 is the number of bytes, and there are enums and deep clusters that don't fit in that space). Doing work to extend something that we've removed from all code except those two primitives seems unlikely.

But that doesn't mean your request couldn't be handled in another way. Several of us have been toying with other ways of representing type on the diagram. We should consider this use case in those discussions. I don't know if it'll go anywhere, but thanks for the feedback.

[m3chtroid]

Thanks, AristosQueue.  It doesn't have to be as general as your Character Lineator, but something philosophically similar that would allow us to define "This is how a labview class/cluster/structure should map to and from a series of bytes" has always been a bit of a holy grail for me, and the type strings came so close to it that I got excited.  Thanks again, and enjoy the rest of your day.

[AristosQueue (NI)]

You know... you can solve the problem by changing your data type... instead of an array, declare a cluster of N elements... the format of a cluster of N homogenous elements and an array of N elements is identical except for the leading size. Similarly, a cluster of N unsigned bytes is the same as a string except for the leading size. If you know the size in advance, you could read them in with the fixed data structures. Converting the cluster of N elements to an array is easy (using the Cluster To Array node). Converting the cluster of bytes to a string can be done with Type Cast. See picture.

[AristosQueue (NI)]

If you want the size(s) to be dynamic at run time, that's a type descriptor you could build with today's Int16 array by duplicating entries... no need for any changes from NI. Worth a try maybe?

[m3chtroid]

Yeah, that's really promising, actually! Especially because the extremely large data chunks are being filled in later with a preallocated read from a binary file.  For the small stuff this should be far more straightforward.  Thanks!

[AristosQueue (NI)]

Providing a workaround like this gives me mixed feelings. Sure, I helped a customer out today, and that's happy-making, but at the same time, I feel like I just dropped a load of my dirty laundry on someone's doorstep, told them they should clean it, and then had them thank me for the opportunity.

Some days, mine is a weird job. 🙂