Experiences with HDF5 for high-speed streaming

[cwierzynski]

Dear LabView Community,

We are building an acquisition system that will need to stream 100-150Mbit/sec to disk as a collection of heterogeneous data sets that will be large (~100-1000GB). So we are considering using the NCSA HD5 file format for our data.

I was wondering if any of you has had any experience with HDF5 for streaming a lot of data. I am particularly interested in:

-- speeds people have achieved
-- NI's level of committment to the HDF5 format, and whether they will upgrade the libraries from 1.4.4 to 1.6 at some point
-- any bugs or gotcha's I should know about!

Any advice would be gratefully appreciated.

Thanks!
Cas

[cous]

Hi Cas,

I understand that you'd like to know three things:

1. What speeds to expect writing data to HDF5 files
2. If there are plans to upgrade the libraries to support HDF5 version 1.6
3. Relevant bugs and information relating to data streaming with HDF5

I am in the process of researching the issue, and I will let you know more information as it becomes available to me. In the mean time, I can tell you that there are no explicit bugs or issues with data streaming that we are aware of. If you haven't already seen it, there is general information HDF5 on our website in a knowledgebase document entitled "Can I Edit and Create Hierarchical Data Format (HDF5) files in LabVIEW?". You can view the document at this link:

http://digital.ni.com/public.nsf/webs
earch/CB8A7F8090900F6C86256CC700033816?OpenDocument

Thanks for contacting NI, and I hope this helps.

Best Regards,
Marcus G.
Applications Engineer
National Instruments

[cous]

Hi Cas,

National Instruments currently has plans to update our components to HDF5 v1.6 at some point in the future. I can't say when that will be exactly, but it's probably in the year or more timeframe. If you have any further questions, please let us know.

Best Regards,
Marcus G.
Applications Engineer
National Instruments