Which file format for a complex configuration file?

[VinnyAstro]

Hi,

I am working for a small company that produces controllable devices via telecommands and telemetries.

I have developed a software to control them for internal testing purposes (Control, acquire and analyse data for PASS/FAIL criteria), but I now need to review it completely because it was not scalable: A new device with an different On-Board Software means potentially a lot of new development on my software.

In this aim, I am currently thinking of working with classes (obviously) but mainly configuration files for each devices to avoid hard coding information and have it rather loaded dynamically. This (these?) file would:

• Lists all telemetries and telecommands with their TM/TC ID various fields (length and other properties) and to which conversion parameter to refer to.
  • e.g.: SET_RATE / 0x42 / "Field1"=0..15 / "IDLE"=16..31 / "RATE_CONVERSION"
• Apply small UI Changes:
  
  • Hide/Show; Enable/Disable; Change min/max of Controls and Indicators
    • Based on the device itself (A specific command might exist for one but not for the other)
    • Based on user "clearance": Admin/Operator gives access to some commands/info
  • Change Tip Strips or a help context menu
    • Some devices share the same Telemetry labels sometimes (like Error Flags "HS2") but they might mean different things (e.g. One would be overvoltage, the other one overcurrent)
• Etc. (I think you get the idea)

The point being to have a file that is readable/editable both by humans and by my software. And I'm not sure which file format is the most adapted for that.

I would be the one preparing such a file (and how to manage it) but I wouldn't be the one filling it in, which is my tough part.

As I see it, I have roughly 2 solutions:

1. Use 1 Excel workbook per device with multiple sheets (Telecommands/Telemetries/Conversions/etc) that then get exported into a .csv to be imported in my software
   • Easily done on the Edit part, easily readable by people
   • Pretty horrible to work with .csv in that matter
   • We already have that implemented for another specific product built in coop with another company
2. Use another file format that is not (so) readable by humans, but much easier to read/edit from labview
   • Requires to create a configuration manager software on top
   • A file like XML/JSON ..... ?
3. (Bonus) Change job and/or field of work

I am fairly new to this configurations ideas since my last application was not scalable in that manner 🙂 So any advices or just stuff to read are very welcome.

I have seen this post about a Configuration Editor Framework, it looks interesting and I will give it more thoughts, but I wanted to have some more options from people that most likely already had this issue before me 🙂
https://forums.ni.com/t5/Example-Code/Configuration-Editor-Framework-CEF/ta-p/3984276

Thanks a lot for your help.

Vincent.

[crossrulz]

I am personally a fan of JSON, especially if using the JSONtext library from JDP Science.  It is not that bad to read in a text editor as long as you do proper indentation.  I find it a lot easier to read than XML.

[Kyle97330]

If this was a configuration file that would only be saved/edited using an interface of some kind I would agree with crossrulz about using JSON with that toolkit.

However, if a configuration file is meant to be edited by humans, I would recommend instead the standard "Config file" format, i.e. the format used by most *.INI files.  JSON can be great for importing fields that aren't flat (like an array of clusters containing another array of clusters) but for anything that could just be called "flat" or easily made into something "flat" then INI files are probably best to hand-edit as it supports inserting comments in the file itself, has a built-in set of LabVIEW VIs for dealing with it, and is less dependent on having just the right setup of brackets and quotes to be computer-readable (since hand-edits could easily break that sort of thing).

[rolfk]

---

@crossrulz wrote:

I am personally a fan of JSON, especially if using the JSONtext library from JDP Science.  It is not that bad to read in a text editor as long as you do proper indentation.  I find it a lot easier to read than XML.

---

Second that!

There is a joke about XML adapted from a different one about regular expressions.

I had a problem to store hierarchical data in a file and choose XML.

Now I have at least two problems!!

[santo_13]

Recently, I have grown fond of the yaml format predominantly due to how easy it is to use in Python. If similar intuitiveness is available in LabVIEW, I will vouch for Yaml.

https://yaml.org/

[VinnyAstro]

Hi, 

Thank you for your answers.

I have never worked so far neither with JSON nor with yaml, I'm done to try any, just need to make sure that this will work for my colleagues too.

We're talking about 100s of Telecommands and telemetries, together with specific parameters and conversion factors (that all might or might not be shared from one device to another.) I'm afraid that a JSON might be quickly impossible to skim through in a text format and do specific changes, no?

In that case I would need to have a configuration file manager, which I am ok with, but it simply takes time to develop.

But before starting anything I need to be sure of my interfaces 🙂

[drjdpowell]

If you go with JSON, I recomend using Notepad++, as it gives syntax highlighting (does XML and YAML cas well).  Here is a small example JSON produced by the Config-file example in JSONtext:

[BertMcMahan]

Yet another recommendation for JSON here as well, especially with drjdpowell's absolutely excellent JSONtext library. I've switched away from ini files completely as json is just SO MUCH more flexible. I'd suggest looking into YAML as well, just because it might fit your needs better, but I can say XML is MUCH harder to use and INI is (usually) much too simple for anything complicated.

If you literally only have key/value pairs, INI is fine, but the instant you need things like unique sections, or repeated key names, or arrays, etc, you'll wish you had something more flexible.

And another vote for Notepad++ for viewing it. Enormous JSON files aren't exactly "simple" to look at, but neither will a giant INI file. YAML might be a little easier on the eyes but I haven't used it for anything "real" before so I'm not certain how the flexibility works.

For my use case, I have one big JSON file that controls all of my "stuff" in my test program. It has different subsections for different things, and I've written viewers that let me edit just the important stuff for the section I'm working on. I can do this manually, but when you have a 2000+ line JSON file you'll want something more "automated" than trying to only ever do things by hand. For example, section C depends on items from sections A and B being correct for it to work, since it refers back to them. My custom editors can run a quick check so when I edit section C it'll tell me right away if I've done it right, or if I've tried to refer to something undefined in another section. That would be a giant pain to do manually in a text editor, and would cause annoying bugs that also take a long time to track down in the actual test routine software.

tl;dr: Use JSON because it's much more flexible than ini, much more readable than xml, and has a great parser already available (though I admit there may be a great YAML parser that I've simply never heard of).

[RTSLVU]

I prefer XML and if you use the free XML Notepad program XML files are very easy to create and edit.

The LabVIEW XML vi's are fairly straight forward to use to parse XML files.

What the raw XML file looks like

[crossrulz]

---

@RTSLVU wrote:

I prefer XML and if you use the free XML Notepad program XML files are very easy to create and edit.

The LabVIEW XML vi's are fairly straight forward to use to parse XML files.

---

I did the same thing for a little while.  But I found the JSON libraries in LabVIEW A LOT easier to debug than XML.  I have recently ripped almost all XML capability out of my HAL since I decided to stick with JSON as my standard.