I would like to suggest that LabVIEW support HDF5 for data format, file storage and data transfer. I would to suggest the support include maintenance to coincide with new and future versions of HDF5.
I know there is some limited native support in LabVIEW for HDF5 with NI-HWS but this is dated and very, very incomplete. I also am aware of 3rd party tools but of course native support is much preferred. Also, while TDMS is much improved since it's first release it simply cannot handle all that HDF5 can handle. In addition, HDF5 has more acceptance and support with other software programs (including free java viewers) making it far easier to share data with collaborators. The more I sit and think about it the more I am really bewildered by why LabVIEW doesn't already fully support HDF5.
Unfortunately I am skeptical this idea will receive much support, not because it's undeserving but rather most LabVIEW users are happy with the data formats available. HDF5 tends to be a data format you find when all your other options have failed you. Once you find HDF5 though you realize how helpful it would have been to you in the past.
Aside from providing compatibility with other applications that support HDF5, what are the significant advantages that would make you choose HDF5 over other file formats in LabVIEW? What tasks do you need to accomplish that our currently supported file formats are not well suited for?
I think the reasons that people seek out HDF5 are varied. I think you will find that most LabVIEW users (at least those that post on the forums) start using HDF5 due to it's ability to handle large data sets. HDF5's power I feel comes from being able to handle n-dimensional arrays of complex data types. Also there isn't a limit to the hierarchical structure of HDF5, you can go as deep as you like (unlike say TDMS). Also as mentioned before it's really important in my particular situation to be able to share the data files with others that use analysis software packages such as Matlab, IGOR Pro, IDL etc.
thanks for explaining this in more detail. I definitely see your points regarding:
- compatibility with other applications
- native support for n-dimensional arrays
- more flexible hierarchical structure.
Regarding large data sets, it might be worth pointing out that both TDMS and HDF5 can handle data sets way beyond what currently available storage hardware can do (where, as far as our benchmarks go, TDMS consistently performs better than HDF5).
With all due respect, and I really mean that, I don't find it surprising that a proprietary data format created by NI for NI (TDMS) performs better in LabVIEW than an open source data format that does not enjoy full native support in LabVIEW and was created by the NCSA, NASA and DOE national labs free for the whole world. We also have to keep in mind that TDMS feature set is limited when compared to HDF5 so again I wouldn't be surprised by some performance gains. As a matter of fact, I expect better performance and ease of use in exchange for a reduced feature set.
BTW, I am not advocating replacing TDMS with HDF5. I wholeheartedly believe there is a place for both TDMS and HDF5 in LabVIEW. I just wished NI could come to the same conclusion and fully embrace HDF5.
the benchmarks were run in LabVIEW, but they were set up in a way that wouldn't hand any sort of advantage to either format. For problem areas we identfied, we actually wrote command-line C applications that we discussed with the HDF5 development team, so we're pretty sure that the comparison is valid. We were testing across a range of reference use cases for Test & Measurement. I can certainly not rule out that there are use cases we didn't at least conceptually cover in our benchmarks up to this point. I'd be glad to learn more about the characteristics of these use cases.
As I mentioned earlier, I didn't intend to argue against your points regarding the limited feature set of TDMS compared to HDF5. I also didn't mean to imply that performance could make up for a lack of features. I was simply responding to the statement that users typically turn to HDF5, because it handles large data sets. If that was a valid reason to turn away from TDMS, we would definitely have some (more) homework to do. If you come across a use case where HDF5 outperforms TDMS, I would very much appreciate if you could let me know.
The actual point of my questions however was to get a feel for how much of what users appreciate about HDF5 could be added to TDMS without causing much of a negative impact. Given that compatibility is a major point for HDF5, it is obvious that we cannot cover 100% of that. But the (ultimate) goal of TDMS development is to provide users with everything they need for file I/O in LabVIEW, as far as that is feasible. We have come a long way, but that goal is still pretty far ahead (and by the time we get where it is now, it'll have moved). It strikes me as a good goal to thrive for, though 🙂
On a side note, I'd like to point out that the specification for TDMS is published and that we do work with 3rd parties to support TDMS in applications unrelated to NI products. So, in a sense that NI owns the specification, the statement that TDMS is proprietary would be correct. In a sense of NI restricting access to TDMS support (comparable e.g. to how file formats in office applications used to be protected in the past) however it would be incorrect.
Thanks again for submitting your idea and for engaging in the discussion.
Some, if not all, of this post may be a repeat of what has been said and is also contained in the "Why HDF5" link previously provided. I have polled some users here to try and determine why HDF5 is preferred over other formats. Here is a bulleted summary:
> suitable for very large or complex data sets
> “easy” to share data across platforms or third-party apps
> open source nature of HDF5 lends the format to a more inclusive community to turn to for advice
> hierarchical data objects
Granted, other formats offer similar advantages. One of my goals within my organization here is to improve/streamline the exchange of data (large amounts of data) collected with LabVIEW to MATLAB for post processing.
Here is a specific example of HDF5 use with LabVIEW:
We started using HDF5 because it is compatible with MATLAB, which uses HDF5 as the file format for ".mat" files in newer versions. By setting some dataset attributes appropriately and using appropriate file header information, it is possible to create a ".mat" file on your own which can be easily loaded in MATLAB with the "load" command.
Here's an article just published in IEEE Computing in Science and Engineering which outlines some more reasons for using HDF. In terms of using in LabVIEW, it would be great to have an API which allowed the choice of either TDMS (for speed) or HDF (for compatibility) with an essentially identical interface, perhaps LVOOP-based.
We just launched a new branch of the NI Idea Exchange to foster
ideas for LabVIEW Add-ons. The intent was to create a place for LabVIEW Add-on
developers (both NI R&D and developers in the community) to find ideas that
matter to the community.
We’ve identified your particular idea as one that could
potentially be made into an Add-on and would like to move it to this new Idea
Exchange so that Add-on developers can easily find it. Would this be okay with you?
