Nugget - Using control references

[Ben]

“Adaptive Save Restore”

"A Case Study of Exploiting Control References to Reduce Development Time"

By
Ben Rayner, Data Science Automation
And
Ray Robichaud, Drawbridge Technologies, Inc

Introduction

A frequently requested operation in applications is the ability to save and restore configurations when an application is shutdown and later restarted.  If you are working in the LabVIEW development environment, this can easily be accomplished using the “Edit >>> Make Current Values Default” operation, see Figure 1.

 

Figure_1_Save_as_Default.PNG: Useful to save defaults in simple programs.

While this operation is useful, it is limited in a number of ways. This functionality does not allow various sets or “versions” to be saved.  When the current values are set as default, the previous default settings are lost.  Attempts to perform this same operation fail if the VI is running (see Figure 2) furthermore, it is not available in any form if the VI is intended for deployment as an executable.
 
Figure_2_Not_Available_at_Run_Time.PNG: Can not save defaults at run time.

The generally suggested solution to these restrictions has been to use or develop code that saves the control and system information to file such that it can be restored at a later time.  If the file storing the data does not have to be in an easily readable format, then the OpenG solutions are worth considering.  For instance, refer to http://jkisoft.com/vipm/

If the information has to be saved in an easily human readable form, then consider the code provided by David Moore from www.mooregoodideas.com

Both of these approaches use variant data to one extent or another. While the use of variants is acceptable, we must carefully observe that the format of the variant data is within the National Instruments domain and they can (and will) change its format or representation in future releases of the development platform. In this Nugget we attempt to present a novel approach that is independent of the variant data format. It illustrates another method of saving and restoring data that relies on control references. This method exploits the characteristics of control references. It automatically adapts to changes to data structures (as is often the case with Type Definitions).

The ability to work with any data structure allows all of the sub-VI’s to be re-usable. The sub-VIs will not have to change if the data type changes. It also allows support for multiple data structures with a single VI devoted to each data type and sharing all other sub-VIs.

Concepts Used

How to read from arbitrary object?
Property >>> value
The value of a control can be written or read using a “property >>> value” node explicitly linked to a control or a “property >>> value” node linked from a control reference (see Write_Control.vi) as shown in Figure 4.

 
Figure_3_Write_Value_with_Property_Node.PNG: Writing to a control at its most basic.

Please note that the data type of the property node is determined by the control reference. This becomes an issue when control references of different types are stored in an array. All references in an array must be of the same type.

How can I determine the data type a reference?
Use “To More Specific Class” and check if it works***
The “To More Specific Class” node allows you to take a generic control reference and cast it as any appropriate type you choose (see Figure 4). The casting to a more specific type will only work if the target type is compatible with the type of the control reference.

 
Figure_4_To_More_Specific.PNG: The data type of the target class dictates the data type of the “property >>> value” node.

The “Digital (strict) reference” defines the data type and representation of the numeric control. The Digital reference identifies the control type as numeric without reference to its representation. Similarly for the Boolean and String versions of the “To More Specific Class” operation. This method will be used to set and get the values of atomic data types (not complex control but rather numeric’s, strings, Booleans, etc).

As mentioned earlier, the “To More Specific Class” returns an error code if the typecast attempt failed. This behavior allows the “To More Specific Class” operation to be used to test the control reference in order to determine if the type specified by the “Target Class” input is appropriate for the reference that is being tested. This behavior will be exploiting this behavior to determine the data type that will be supported. I should mention that thought was given to using the “ClassID” and or “ClassName” properties, but because they are both subject to change by NI, the trial and error method was used.

[Ben]

How do I get access the object within a cluster?
Cluster >>> Controls[]
Clusters present our first challenge in our attempt to interact with generic data types and avoiding strict references to data types. Variant operators are very effective if you specify the data type using constants or strict references. The strict reference would demand a VI for each cluster type that you plan to deal with and does not lend itself to adaptability (see Figure 5).
 
Figure_5_Variant_Data.PNG: To use “property >>> value” nodes of clusters, strict references of the cluster are required.

Another method for accessing the data in a cluster is to use the “Controls[]” property that is available for cluster references (see Figure 6). Since there can be mixed data types in a cluster, the Controls[] are all generic control references. Generic control references can be type cast, as mentioned earlier. Successful typecasting indicates the data type is compatible. A simple example of how this can be done is shown in Figure 6.

 
Figure_6_Manipulate_Cluster_Elements.PNG: Trial and end casting and writing stops when no error is detected.

Clusters of Clusters
As illustrated above, clusters can be read and written using references to the controls inside the cluster. This applies to nested clusters as well. The sub-cluster can be manipulated using the elements contained in the sub-cluster.

How to interact with Arrays of Clusters?
Use “ArrElem”
Arrays in general present the biggest challenge when attempting to develop code that can support many data types.

These challenges are:
Unknown data type,
Unknown size and,
Only one reference to all of the elements of the array.

The unknown data type challenge can be handled in a fashion similar to the trial and error illustrated above using the “ArrElem” reference and type casting as the appropriate type as shown in Figure 7.

 
Figure_7_Manipulate_Array_Element.PNG: Elements of an array can be manipulated if we know the data type of array element.

But that is not enough to be able to handle accessing array data because it only addresses a single element of the array and does not specify which of the elements of the array is being accessed.

How to index the array?
IndexVals – well almost
If you add a dimension to the example I provided and change the index settings and running it you will find some interesting behavior. The value that gets updated is dependent on the index settings AND the number of elements that are viewable. (Please experiment for yourself).  If, however, the arrays viewable size is set to 1 X 1 (this can be controlled using the “Number of Columns” and “Number of Rows” properties), then the index does control which element gets updated or read.

How do I determine the array size?
“Keep stuffing it until it gets bigger” – well, it’s one way…
The array size operator simply will not accept a variant data type, so the operator we normally reach for is not available.  Reading from an ArrElem referenced node will always return a value even if the indexed value is larger than the size of the array in the control, therefore test reads will not help us determine array size.  Pulling the data out of the variant data is possible but puts the application in jeopardy of upgrade complications.

 These obstacles can be over-come by checking the size of the variant (flattened to a string) before and after a test read-write of increasingly larger indexes (see Figure 8).  If the flattened variant has changed size as a result of the test-write, we have stepped past the previous size of the array.

 
Figure_8_Determine_Size.PNG: An unorthodox approach to determining the size of arrays on unknown data types uses a “Rube Goldberg” code construct to “probe” array elements.

How can a variety of data structures be supported by one set of VI’s?
Use a “VI Server”

The final major challenge was allowing for arbitrary data structures.  Arrays of clusters require interacting with multiple references with multiple indexes, while clusters of arrays is another approach.  Traditional approaches to developing code to save a cluster to an ini file generally manifest themselves as code structured in a manner synonymous to the data being saved. For example, to save a cluster that contains an array, the array could be unbundled and passed to a For Loop so each element could be saved.  If an array of clusters had to be saved, the structure would have to be turned inside out.  VI Server calls allow reentrant calls of VI’s from within reentrant VI’s. 

This Nugget is continued in the next post

[Ben]

Strategy:
While developing this approach, an attempt was made to limit support to only those data types that are used regularly (e.g. float, strings, paths, etc) while allowing for easy adaptation for other data types. Another goal was to keep the instances of the supported data type to a single VI. This goal allowed us to keep the rest of the functionality completely disconnected from the data type definition. This in turn allowed for easy cloning of the top-level save-restore functionality to support other data structures (see Figure 9).
 

Figure_9_Data_Type_Usage.PNG: Only the top-level VI is depends on the data type that will be saved or restored from file.

Arrays and Clusters are saved and restored using reentrant VI calls to allow for nesting to any degree. Only two-dimensional arrays were implemented. The work was broken up into three phases. The three phases are Init, Save, and Restore. During Init, the data structure is “learned” so that the save and restore operations would only use the appropriate technique required to either read or write the control.

Figure 10 shows the test VI for this example. It invokes the Action Engine four times and allows the Save-Restore VI to be quickly tested. The first call of the Save-Restore VI initializes it for use later to either save or restore information from file.

 
Figure_10_Save_Restore_Tester.PNG: After learning about the data type, the data that had previously been written to disk are restore. New values are then written and verified.

Save-Restore Overview:
There are three actions implemented in the Save-Restore VI, Init, Save, and Restore. The Init action prepares the VI for later use by analyzing the data structure that will be written to or read from disk. During the analysis, a set of arrays is developed that contain references to all of the elements within the data structure. It also builds arrays of data type identifiers, parent references, and strings that will used to develop the Key names used in the ini file. The analysis phase results will drive the Save and Restore operations. The Save and Restore operations use the arrays developed during analysis to process each control reference. Save and Restore will be discussed later.

Init:
The analysis code is shown in Figure 11 and illustrates how a typical element is tested for its type, in this case, Digital. This figure also illustrates how each element of the “Control Info” is populated. For each iteration of the outer While Loop, a control reference is removed from the “head” of an array in the shift register. The shift register stores all of the references that need to be identified. This shift register is initialized with the reference to the control that will contain the input or output data.
 
Figure_11_Init_analysis_phase.PNG:  Determines type of each element and takes notes on how to structure calls required to save or restore.

When the array of references “to be processed” is emptied, the identification process will terminate returning five arrays that identify:
Control references for input control
Control references for output control
Enum identifying the data type of the control
Name of the control as used in constructing the key name
Index of the parent of each sub-control

The values contained in the enum labeled “Types” will act as a selector when it’s time to save or restore.  As each of the control references are indexed, the corresponding Type will invoke the proper type casting for the data type (assuming that the input type matches output type*).  The indexes stored in the “Parents” array allow children to identify their parents and a parent to identify its children.

 
Figure_12_Init–analysis_Cluster.PNG: The elements of the cluster are queued-up for reiterative processing.

When a cluster is found in the “to be identified” array, all of the controls within the cluster are prefixed to the “to be identified” array.  Similarly, for the output control references.  The array of names and parents also get entries that will establish the relationship of the sub-element to the cluster**.

[Ben]

 
Figure_13_Init–analysis_Array.PNG: The reference to the element in arrays is used to determine how to interact with the array elements.

Arrays differ from cluster in that they only have a single element. At the end of the analysis phase we have all of the information we need to process the data structure.

 
Figure_14_Analysis_Results.PNG: When the analysis is done we have arrays that describe  the data type, its components and their relationships to each other.

Save:
Saving data is actually more complicated than restoring data because we have to answer the question, “how big is the array?” but before we get into that detail lets step back and look at what the inputs and outputs of the “Save” operation are (see Figure 15).

 
Figure_15_Ins_n_Outs.PNG: The  data structures require flattening to be written to file and un-flattened to when restoring.

Inputs and Outputs:
The input will be put in the control that was analyzed during the Init operation. The output of the Save operation will be an ini file that can be read back to restore the data. Since LabVIEW data types can consist of nested structures of various depths, a single section was used for all elements of the control. The key names are composed such that they describe which control is associated with which cluster within which array at which index… etc.

 
Figure_16_Types_Control_Processing.PNG: Case structures linked to the “Types” enum call the appropriate code to interact with the data types.

Saving is done by indexing through the list of control references and based on data types, either process the data element or call a VI for that data type as illustrated in Figure 17.

 
Figure_17_Save_Array.PNG: Saving array elements are done one element at a time.

The save operation starts by determining the size of the array (as discussed earlier). The visible size of the array is set down to 1X1 before the indexes are manipulated in a loop. As the indexes are manipulated, the elements of the array are read as dictated by their data type.  Clusters require cycling through all its sub-elements so a sub-VI is called to handle clusters inside an array.
 .
Figure_18_Save_Cluster.PNG: Clusters can have sub-cluster so recursive calls are structured as required by the data structures being saved or restored.

When saving a cluster, we must process each element of the cluster. This is shown in the insert of Figure 18. If a cluster is located inside of a cluster, the VI calls itself recursively to process all of the elements of the sub-cluster. If an array is found inside the cluster, the array save operation we examined previously is called recursively to handle the sub-array.

Restore:
The restore operation is exactly the same as the Save except it reads from the ini file and writes to the property nodes. It also determines array sizes by looking for keys in the in the ini file.

Conclusion:
A generic control reference can be probed to determine its data type. Although the data type of cluster and arrays require explicit knowledge of the data type at development time, the elements of arrays and cluster can be read-written individually. Please see “Adaptive_Save_restore_Multiple_Tester.vi” as an example that uses the above example for two different data structures in the same application at the same time.

Questions and Ideas:

OK, now we get to the part where I get to learn stuff.

Q1
The example provided only supports 2-d arrays. Do you think that a 2-d array has a significant number of dimensions to satisfy real world needs? If not, can you please provide a possible use case that would require 3-d?  And how would the suggested example be modified to support N-d?

Q2
Enums are not supported by the provided example. How would these be implemented?

Q3
Numerics are all saved as a fixed number of significant figures. How to track and specify the number if sig figs?

Q4
Unused keys are not deleted.  When the number of elements in an array is reduced, the key entries still exist in the file. Can you offer any suggestions on how to how to keep track of what keys need deleting?

Q5
The example provided presents an effective but inefficient technique for determining array dimensions. Can you suggest a better method that can survive LabVIEW updates?

Q6
What other tips suggestions or feedback can you offer for the above example?

Footnotes:

* Another application of the techniques illustrated in this Nugget would allow re-packing cluster of one cluster order into a cluster with a different order provided names do not change. This could be useful with sorting algorithms to allow sorting on different criteria.

** A hierarchal naming convention within a single section was chosen simply as a brute force approach at keeping the names unique and descriptive. I would like to hear about how to handle the key names and section better.

*** I learned of this method from an example posted by Jean-Pierre Drolet.

For alist of past Nuggets see here.

If you want to write your own Nugget see here.

[Ben]

Examples in LV 8.2.1 and word form of Nugget

Ben

Message Edited by Ben on 08-26-2007 09:54 AM

[tst]

If nothing else, the amount of work that went both into the code and into the nugget are appreciated. Thank you, Ben and JLV.

Now, to some comments. I'm afraid they will be mostly "anti", but they should be taken in good spirit.
Also, this is just an initial impression. I will be lying if I said I understood everything you did. There is simply a lot of it to take in.
Additionally, there is probably some stuff which I will forget.

• Goes through the UI thread.
• Requires duplicating the AE for each typedef you want to save (although maybe Stephen's Swiss-army-knife LV2OO Glob could help with that).
• Relies on "tricks" (e.g. resizing the array control, checking the size of the variant to determine the dimension size, which I didn't really understand, but might change in future versions. In 8.x you should be able to get rid of it by using the VIs in vi.lib\Utility\VariantDataType).
• How well tested is this?
• How about speed?
• What happens with unknown types? OpenG flattens to a string.
• How well does it survive name changes in the typedef\file?
• Would the parent indexing hold well in changes to the typedef?

The example provided only supports 2-d arrays. Do you think that a 2-d array has a significant number of dimensions to satisfy real world needs?

It probably does, until someone decides they want to save a 3D array.

Enums are not supported by the provided example. How would these be implemented?

That depends on the usage of the enum. If the users are likely to add elements to the middle of the enum, you should save and load by the enum text.

If the users are likely to change the names of the enum elements, you should save the enum value.

The problem is that there is no way to tell what the users would do.

What you can try is this - save both the string and the number. At first, try to load both. If there is no match, try the string and if there is no match to the string, use the value.

Numerics are all saved as a fixed number of significant figures. How to track and specify the number if sig figs?

In the OpenG code, this is done by supplying an input. You could try adding metadata to these keys which would have this data and get the data from the properties.

Unused keys are not deleted.  When the number of elements in an array is reduced, the key entries still exist in the file. Can you offer any suggestions on how to how to keep track of what keys need deleting?

I brought up this topic with Jim once. I believe what he implemented in the end was deleting all the keys for the array before writing it again.

The example provided presents an effective but inefficient technique for determining array dimensions. Can you suggest a better method that can survive LabVIEW updates?

As mentioned, there built-in VIs in 8.x.

[shoneill]

I've been down this "universal reference" system before.  I'm fairly familiar with all the points Ben has mentioned here.

If I'm not mistaken, Tst's statement that a new version of the AE is required for each typedef is incorrect.  At least my version took ANY reference and was able to parse it and save / restore it.  This is the whole beauty of the approach, it's flexibility.

True, it goes trhough the UI.  However, this isn't typically an operation which is performed during HW interaction, more at program start and stop, so i I think this is acceptable.

True, it relies on tricks, but then we all use lots of tricks.  As long as they work, who cares?

Regarding testing, it's fairly robust actually.  Initially, making sure references are closed and so on can be nasty, but testing is easy with multiple reads and writes of a file to determine memory bloat.  Speed is an issue.  Stepping through an array index by index cannot be called efficient.  Here I prefer to go the Variant route.  I know people cry "But that might change".  I don't care, I'd rather cross that bridge when I come to it.  By that time there will be a better method available.  Apparently serpdrv still works with LV 8.5!

I also disagree on the "trial and error" method of getting the type of the control.  The ID value is still much more convenient, and is a lot less likely to change than the variant structure.

Overall, I want to thank Ben for a great nugget.  Even though I'm familiar with it already, reading someone else's version is very interesting, noting the subtle differences in implementation.

3d Arrays? Not until stereoscopic displays become more common!

Shane.

Ps: Tst, you say there are built-in VIs to determine srray size?  Can you enlighten us?

Message Edited by shoneill on 08-26-2007 09:03 PM

[tst]

If I'm not mistaken, Tst's statement that a new version of the AE is required for each typedef is incorrect.

I was refering to the top level VI. The typedef cluster is part of it, so it will have to be duplicated.

True, it goes trhough the UI.  However, this isn't typically an operation which is performed during HW interaction, more at program start and stop, so i I think this is acceptable.

Speed can be an issue, but all of these methods have speed problems. I believe the benchmarks MGI published for their method showed it to be much faster, but those are 8.x only.

True, it relies on tricks, but then we all use lots of tricks.  As long as they work, who cares?

Yes, "as long as they work". I wouldn't want to find that a certain trick stops working.

Regarding testing, it's fairly robust actually.

I was wondering about Ben's version specifically. I know that control refs and properties work fine, but I was wondering whether this specific implementation was tested and used in actual applications.

I also disagree on the "trial and error" method of getting the type of the control.  The ID value is still much more convenient, and is a lot less likely to change than the variant structure.

I knew I forgot something! I would agree. That's the whole point of the ClassID - to not change even if the class changes.

Ps: Tst, you say there are built-in VIs to determine srray size?  Can you enlighten us?

I did. They are in vi.lib\Utility\VariantDataType and they are what the MGI VIs use.

LV 7 also had some VIs for this (in vi.lib\Utility\GetType.llb) which used the type descriptor, but the 8.x VIs have more options (including finding the size of the array dimensions).

[unclebump]

I guess we know what Ben and JLV do in their spare time!!!

[Gabi1]

....And now that the tool is ready i can just use it ! 🙂

Very nice!  up to now, i was always creating a new load/save params vi, for the specific application, as ben depicted for arrays of cluster for example. was quite annoying. i expect it will be a world easier.

btw, for those who still like to use : make current value defaults, thats how it might be done:

For enums, i generally never let the user update them directly, but always trough an external vi, that saves both the string and the value.

Message Edited by Gabi1 on 08-26-2007 11:27 PM