Storing Data

[RL]

Storing data in PDF files is no longer satisfactory. Spreadsheets are bogging down because of data volume.
This leaves database which can be modified, searched and readily accessed by Diadem. CONFUSION reigns! Database Connectivity is a possible module and so is DSC (Dataloging and Supervisory Control). Is there a document which gives the Pros and Cons? I mostly need to log and analyse sensor output provide for a string that describes the experiment and control relays. Is there a Wizard that helps to make such a basic choice?

Regards,

[Jeremy Braden]

There is not one document that describes the pros and cons of the different methods.

Spreadsheets--cheap but usually limted in size if you intend to use with excel. No limit with diadem.
Database connetivity toolkit--used with your own database. (it is the pros of the database but with the toolkits ease of use)
DSC--slow (because it typically handles large volumes of data) but can automatically record thousands of datapoints.

Your best bet is to give NI sales a call and have them help with the decision making.

Based on your sentence description, I think you would be best off with creating a text file for your data. (Write strings to file). If you are worried about space, you can do a binary file instead.

[Herbert Engels]

1) NI is working on a white paper that walks you through the decision of which file format / database is best in your use case. I cannot tell for sure when it is going to be available on ni.com.

2) DIAdem can readily access the "Citadel" databases coming with LabVIEW DSC or VI Logger. Citadel is a streaming database designed for data logging with high data throughput and low disk footprint. It is not as flexible as a commercial SQL database when it comes to storing complex descriptive data like testcell setups, test step definitions etc. It doesn't seem that you are storing such complex data, so from what I can tell from your description, Citadel should fit your needs. In case you need SQL connectivity, NI provides an ODBC driver for Citadel data.

3) DIAdem has a native command set for SQL/ODBC data access and supports ADO via VBScript - both of which need writing DIAdem autosequences. DIAdem also comes with an SQL table reader that allows you to interactively import complete database columns into DIAdem channels (no queries). The database connectivity toolkit provides programmatic access to databases from LabVIEW, but it doesn't match the comprehensive feature set (data viewer etc.) you find in LabVIEW DSC.

4) Storing measured values to SQL databases requires some design consideration. Unless you are dealing with very small amounts of measured data, you will probably need to separate the "meta"-data you use for searching (test description, input values etc) from the actual measured data values. You then store the meta data to database cells you can query via SQL. The measured data values are stored as binary blocks to "BLOB"-cells (binary large objects). This is done for performance and disk-space reasons. Only if you need to perform SQL queries on all of your actual measured values, it makes sense to store them in single database cells. Another way of storing measured data is to write the data to binary files, and copy the meta-data from the files into a database for searching. That allows you to use a low-end database for huge amounts of data - but you need to take care of keeping your database in sync with your files. There is also a couple of higher-end solutions around (ASAM-ODS standard, VARRAYs on ORACLE systems ...) that can be accessed with NI software, but it sounds like you can get the job done with something more basic.

[RL]

Thank you Jeremy,

Typically I log 500K to 1M points per measurement. Which makes Excel a moot point. If I understand correctly, saving to Diadem is essentially saving to Citadel which is facilitated by the DSC module. When you remark that it is slow are you inferring that it is unable to capture at higher speeds, that it requires a large buffer, or that the operator has to wait longer between measurements while writing to disk. It was my impression that I should write directly to Citadel with DSC package instead of using the Database Connectivity module to write to MySQL then access that information from Diadem. For clarification purposes I have the required modules but would like to figure out what are the implications of various methods. Sales or suppo
rt have not clarified this for me but then perhaps I was asking the wrong questions.

Regards,

[RL]

Thank you Herbert,
By the way I already have the addon modules for Labview.
I wish to store in separate fields:
-time stamp
-string that describes the experiment (if this
string can be searched or queried by the
database so much the better)
-4 channels of data with 100K to 300K points

I am a database newbie and thought that it was bad enough learning Citadel for Diadem than to learn that plus something like MySQL.

As mentionned in a previous post I am not getting the information from NI that makes me comfortable about the choice of which way to go. Perhaps I have not asked the right questions.

Regards,

[Herbert Engels]

In my previous answer, I assumed that what you want is a database. That would mean, the idea of having a single consistent data store with comprehensive searching capabilities is a core requirement.
If your main requirement is data exchange between LabVIEW and DIAdem, I would highly recommend looking into the Storage VIs in LabVIEW 7.1. They are fairly easy to use and they write the native DIAdem file format, which consists of an XML-based file for descriptive information (channel names, units etc.) and a binary file for the data arrays. There is also a query-like function that allows you to load e.g. all channels the maximum of which exceeds a given number, or all files the author of which starts with "NI". This function makes it pretty st
raightforward to implement a query mechanism similar to a database. Searching will be slower (obviously) but the whole approach is very light-weight compared to a database.

[RL]

Essentially my acquisition vi will gather raw data which will be identified by a Time Stamp and a descriptive string. I would like to then use a
Reader that will do a FFT and other analysis on records that match a search string. If I only have a data repository for the sake of Diadem I fear that I will not have the option to "data mine" in the future. Now if I can transfer that data to MySQL or another capable database this is a moot concern. Diadem will allow a more complex presentation than simple graphs.

In the mean time I will install 7.1 and have a look at the Storage VIs.

Thanks again.

[RL]

Perhaps the Database Connectivity module is more appropriate to capturing 4 signals at 200k/sec than DSC. In addition there is a Time-Stamp and a descriptive string which explains the experimental setup.

Can anyone comment about the Database Connectivity module?

Regards,

[rolfk]

RL wrote:

> Perhaps the Database Connectivity module is more appropriate to
> capturing 4 signals at 200k/sec than DSC. In addition there is a
> Time-Stamp and a descriptive string which explains the experimental
> setup.
>
> Can anyone comment about the Database Connectivity module?

Storing that much data is simply out of question other than as binary
data directly into a file, or you would need some highly specialized
database system with hardware and software characteristics which only a
military project could possibly come up with enough funds for.

If you are thinking about storing that as a BLOB (binary large object)
you could VERY VERY maybe get there, but in general databases are not
meant to deal with su
ch large data and are not optimized at all for
that. Your database would bulge out of its seams very fast.

The DSC Toolkit won't be able to do that either as its historical
logging database is simply not meant for that either. Unless you are
going for binary data files directly written from your application, your
options are probably quite exhausted.

Rolf Kalbermatter

[RL]

Thanks Rolf,

The frequency was included only to give information regarding speed. The measurements are rather short and consist of 500k to 1M points with only about 10 to 20 measurements/day.

Am I back into database territory?

Regards,