App Files Saved as XML and Working with New Versions

[battler.]

We work with app data in the form of array of large clusters and use LV Anything to XML conversion VIs to save to disk.

The problem is when new app versions are developed and changes are made to the cluster format; we want the customer to be able to open the old version saved XML file format.

Usually the saved cluster (in XML) has be changed by adding/removing a field or even changing the options of a ring control for example.

The app name and version is stored in the XML.

Question: Is there a way of easily working with XML file formats with slightly different fields?

In the past we have worked with the raw XML text by searching for fields which are different and modifying them accordingly so they are accepted by the new file format.

Any tips would be greatly appreciated.

[Bob_Schor]

I'm dealing with this exact issue.  The key element is that you know the Version, so you know which Cluster definition to use.

There are several ways to do this, but this is the approach I took.  First of all, I'm assuming that you have Version 1 (the original Version), Version 2 (a changed Version), and Version 3 (the current Version).  In the Best of All Possible Worlds, Version 2 and Version 3 are the same thing, and Version 1 is just "missing" some fields.

My main program uses Cluster 3 (based on Version 3) for everything.  When I'm ready to read my XML file, I first determine which Version of the data are present.  If it is Version 1, I read into Cluster 1, "map" the result into a Cluster 3 output, and send it out as Cluster 3.  I do a similar thing for Version 2 data.

To help encapsulate and isolate this mapping, and handling the "foreign" clusters, I wrap the reading/conversion routines in a Project Library, with the inner workings kept Private, exporting only the Cluster 3 result.  This allows me to have VIs that are basically identical, except one Library has Cluster 1 "buried" inside it, the other has Cluster 2.

As it happens, I've just finished "refactoring" my original "One-Version" code in this manner, as I am currently creating Version 2.  I have not yet tested this, but I'm pretty confident that it will work.

Bob Schor

[battler.]

I have written a software name and version cluster to XML.  How do you read the version field in the XML?  Do you use string search?  Is there a better way?

I'm interested in how best to work with XML.  It seems cumbersome and prone to error to use string functions.  Is there a way or best practice of reading the XML fields directly without converting to cluster?

Can you elaborate on these "mapping" and "reading/conversion" routines you're using?  How do they work and what functions do they use?

Why do you use Private Project library?  Is your project based on OO?

[Bob_Schor]

---

@battler. wrote:

I have written a software name and version cluster to XML.  How do you read the version field in the XML?  Do you use string search?  Is there a better way?

Yes, I use String Search (because it is near the beginning of the XML string).  Depending on your XML "flavor", you should be able to uniquely-identify the Version information.

I'm interested in how best to work with XML.  It seems cumbersome and prone to error to use string functions.  Is there a way or best practice of reading the XML fields directly without converting to cluster?

XML is, after all, text with specific "tag" information, also text, identifying it.  String functions are pretty much "the way to go", but except in rare circumstances, you shouldn't necessarily "roll your own", but should try to use the utilities that come with your XML package that "know" how to do the mapping between the XML representation of a LabVIEW variable (and its value) and the Variable itself.

Can you elaborate on these "mapping" and "reading/conversion" routines you're using?  How do they work and what functions do they use?

An excellent question.  There are at least three XML packages out there for LabVIEW -- the "native" LabVIEW package that NI provides, EasyXML from JKI Software (distributed using VIPM), and GXML, also on the LabVIEW Package network.  I'm afraid that I'm slightly "abusing" XML, and have made some modifications to how I use EasyXML and GXML. 

If you simply view an XML file in a "smart" editor, you'll see a series of text lines with Begin/End tags (like HTML).  Depending on the XML "flavor", embedded between the tags are data that describe the LabVIEW variable type (e.g. Dbl, Cluster, Array, etc.) and also the value(s) of the Variable.  The "ReadXML" (or similarly-named) function takes this description and (usually) returns a Variant that you can "turn into" the chosen variable (because you know the data type).  [You may need to tell the Reader, in advance, the type of LabVIEW variable to expect so that it "knows when to stop reading"].

As to how they work, they just do String pattern matching and processing, usually of a pretty sophisticated level, but then you don't have to do the hard work yourself!

Why do you use Private Project library?  Is your project based on OO?

The reason I used a Project Library with the "messy details" in the Private parts has nothing to do with OO, but rather with "data hiding".  I actually developed this for my data processing routines.  I had a set of routines that collected and analyzed the data (about 700 VIs).  Let's call this Project "Version 1".

For Version 2, I basically added a dozen variables to the main  Data Cluster (having over a hundred elements) that characterized the response.  As this Cluster was in a Type Def, I didn't have to change my code very much to accomodate this new data format, so most of my VIs kept the same name, but were "different" because the underlying TypeDefs were different.

So I have a Version 1 program that can analyze Version 1 data, and a Version 2 program for Version 2 data.  What I want is a Version-independent program that can look at the data, and then call for Version 1 or Version 2 analysis.  By burying the Version 1 and Version 2 code inside of a Project Library, and only exposing the top-level "Analyze My Data" VI, my Analysis routine, which didn't know (and didn't care) about the two different Data TypeDefs, just figures out (from the XML) which Version of data I have and then calls either Lib1:Analyze My Data or Lib2:Analyze My Data.  If I ever change the Version again, I just need to copy the code in the Library to a new Lib3 Library, modify the TypeDef and related version-specific code as needed, and I'm done.

I may have made this sound easy -- trust me, it was a definite learning experience.  While trying this out, the first thing I did was to "break" my installations of LabVIEW 2012, 2013, and 2014 (curiously, LabVIEW 2011 wasn't broken).  It stayed "broken" for about a week, then "cured" itself (I'd done something illegal, or maybe just immoral, with my Library, and it took a while to purge the badness).

BS

---

[battler.]

Thanks.

I haven't seen that the XML toolkits you mentioned have any low level VI for extracting key values.  Please let me know which XML toolkit and specific low level VIs you're using.

I am using LV native XML VIs (and format) which I would be reluctant in changing to JKI format for example.

It seems like the old .ini config files and NI VIs were easier to work with (you could find keys names and return values) except there wasn't a VI to easily convert large clusters to .ini...

Specifically how do you "map" from the old cluster format to the new??

You read the XML and confirm the version number then (a) do you convert it to its old cluster format?; or (b) do you modify the xml string (add/remove tags using string functions) such that the string will convert directly into the new cluster format??

If you go straight to (a) above how can you "map" a cluster to a cluster??

[Bob_Schor]

Oops, I mis-spoke.  You basically have to do the conversion from one cluster (typedef) to the other "manually", which means that you need access to both Version 1 and Version 2 typedefs.  Just a sec, I'll cobble up an example ...

Suppose Version 1 used a cluster with elements Position (dbl), Velocity (dbl), and Name (string).  Version 2 added Weight (dbl) after Velocity.  Let's further assume you have figured out how to take an XML string that you know has a Version 1 cluster inside it and parse (or "extract") this cluster from the XML.

Your goal is to always work with Version 2 clusters.  If you see that the XML file contains Version 2 clusters, you are Home Free -- simply extract the Cluster from the XML string.  But what if you determine that the XML data is in Version 1 format?  You need to supply the Weight (the missing parameter), which I'm showing here as a separate Control (you can use a default Weight, or have supplemental data that supplies the Weight).

The Snippet shows what you do -- take the Version 1 Cluster you extracted from the Version 1 XML file, unbundle it by name, and bundle-by-name a Version 2 cluster, matching elements and adding in the "missing" data where appropriate.

BS

[battler.]

Thanks.

Ok this is how I propose how to open XML files of the latest format and for previous versions.

1. Take a snapshot of the previous version cluster format; save it as ctrl with all internal clusters disconnected from thier typedefs (if they exist).  Note to check that the snapshot control contains no typedefs, place it on the BD as constant and any typedefs when expanded will show the black triangle at thier corner.

2. Upon File open of XML attempt to read XML using latest cluster file format, if error then try to open using previous version(s) cluster format.  If all fails then incorrect file format and file fails to load.

3. If step 2 above has determined the XML file format is of the current cluster format then proceed to open.  If it is determined to be the file format for a previous format then map the old cluster to new format and proceed to open in new format.

[mikeporter]

I have been thinking about how to respond to this post since I first saw it. You do realize that even if you figure out how to manage this particular problem, you will have basically papered over a fundamental problem that someday will have to be fixed? Moreover, the longer it gets put off the more painful the correction is going to be.

The basic issue is that if the structure of your data can change over time, you don't want to save it in a format that cannot seamlessly track version changes. And it only makes it worse when the data is stored in a way that is inherently insecure. Remember that behind it all, XML is just a text file with all the limitations and security problems that go with text files.

What you need is a database. A database stores data in binary form. It stores data securely and in a form that is version agnostic. Databases are also searchable and a plethora of reporting tools exist that allows you to extract information from all your data. Plus, the conversion process could easily be automated.

Mike...

[battler.]

Hi Mike,

Example of such a database implementation?  An example of some reference material might help me to understand your suggestion better.

I thought it was common practice to store app data in xml files.  The changes between versions are stored in the mapping VIs and in separate design documentation.  We encrypt the xml string.

Cheers.

[mikeporter]

It may be common practice in some circles, but that doesn't mean that it's a defensible practice. Storing encrypted strings in XML may protect the data from modification, but not from destruction. A malicious person might not be able to see or modify your data, but changing a character here and there in the encrypted string will destroy it because it will no longer decrypt.

I learned a very long time ago that saving data in separate files is tantamount to throwing it away. The best you can hope for is a situation where the data is all there (probably) but you can't find the specific bit you need.

In terms of examples, what do you want to see? There are many examples on the forums of people using databases in conjunction with LabVIEW. I recently did a 3 part series of using databases with LabVIEW on my blog. The important thing is that you take the time to do a proper data modeling of your application data. Hopefully, at least some of that work was done when your XML implementation was designed.

With that data model in hand you can create the table structure to save your data.

What kind of organization are you in? If you have an IT department, they can help with the design and implementation of your database. In fact, depending on your organization, they may insist on doing it. Likewise, the goal should be to have the database located on a corporate server (no worries about backups and archiving).

Oh, and we haven't talked about whether you work in a field or business where there are governmental regulatory requirements. If so, XML datafiles are a disaster waiting to happen.

Mike...