Saving Raw Sweep Data and Summary Parametric Data in Same file

[Erickey]

For sometime I have saved raw sweep(array) data into a .csv file then another file with calculations from that sweep data. It always bothers me to have 2 separate files. I've saved the parametric data as columns appended to the raw data before,  but I think its really ineffective(in terms of file size) as the calculation result is simply repeated for every iteration of my array.

So I'm curious if anyone out there has done something similar. I used to work for a large company with IT support and had databases but now am working for a startup and am seeking a simpler approach. I was considering a multi-tab excel file as well.

Ideas?

[Gregory]

Hi Erickey,

You can write your calculated values back to the same file as the raw data. How large are your files and how many files are you generating where size is an issue?

[Erickey]

yeah I've done this. If I have a sweep that contains an array of 80 items, the calculations are repeated 80 times. Basically the same number 80 times(no blanks). While file size doesn't kill me, repeating a number 80 times seems inefficient and I was curious about what others thought.

[mcduff]

You could 2 things, that I can think of:

1. Keep your text file, (.csv), but write a header to the file before the data. The header may contain, Start Time, Instrument, User, Calculation result, etc. This way you have all of the information in the file, but not a column of repeating values.
2. Store your data in a file that allows metadata for each column (dataspace). TDMS, H5, FITS, are a few file formats that allow metadata to be integrated into the file. Obviously, LabVIEW has a lot of tools for TDMS. Your colleagues can download the free Excel plugin that will allow them to read TDMS files and their associated metadata in Excel.

mcduff

[Bob_Schor]

Or use the Report Generation Toolkit to write an Excel Workbook with two WorkSheets, "Raw Data" and "Calculated Values" (or whatever is appropriate).

I'm not sure why you are doing 80 repeats.  If you have "an array of data" and also "a small set of parameters for these data", you could create a "formatted CSV" file by doing the following:

• Write the Array of Data, possibly with a Header Row, to a CSV.
• Calculate your small set of parameters.
• Append/write (again, possibly with a Header Row) the parameters to the CSV.
• Write a spacer (a blank line, perhaps) to the CSV.
• Repeat as needed.

This works "best" when the number of rows in the Array is fixed, as it simplifies reading the file (you read the Array, then the parameters, then skip to the next Array), but you could be "clever" and embed the number of rows as part of the header.  It just depends on your intended use and whether a goal is "human-readable" or "optimized for computer analysis" (and, yes, you can do both at the same time ...).

Bob Schor

[Gregory]

---

@Erickey wrote:

yeah I've done this. If I have a sweep that contains an array of 80 items, the calculations are repeated 80 times. Basically the same number 80 times(no blanks). While file size doesn't kill me, repeating a number 80 times seems inefficient and I was curious about what others thought.

---

Are you saying you perform a single calculation on the entire array? Then yes, only write a single value to your CSV file. For some test data I start the file off with a parameter called "InfoRows". That way, I know where my sweep data starts. You could hardcode the number of InfoRows as well, but I often find that I need to add a row later, and this way earlier test data is just as compatible with your file reader as new test data, just missing whatever new parameters you added. It might look something like this in Excel, where "Average" on row 5 could be your calculated value.

[Bob_Schor]

.. and if you really want to do this as a Delimited Spreadsheet, you can do that, too (you just have to work a little harder).  I'm attaching some code that generates gregoryj's example, with some names changed to protect the guilty.  Note I also used the default <tab> delimiter, which explains the spacing of my output file.

Bob Schor

[Erickey]

Bob, really interesting idea. Thank you so very much.

May I ask you to please repost your code using LV 2015(pretty please)

[Erickey]

Greg, 

very creative idea, thank you. 

[mcduff]

Attached is Bob's code in 2015 version, without using write spreadsheet vi, (I never liked that VI).

I would recommend writing lines to the file using low level VIs and if you have a big array of data, write it in chunks.

mcduff