LabVIEW
Saving Data - optimising '.csv'
Solved!
Introduction
I have collected a significant amount of data using my LabVIEW programme which collects 2 channels of accelerometer data. Since collecting the data I have updated some calculations in VI, therefore, the data needs to be reprocessed which would be a tedious manual task. To automatically reprocess the '.xlsx' files and save both the raw data and new outputs I have created the attached VI which I intend to apply to my existing VI. In this case the output parameters will be simplified to represent the maximum voltages from each channel. I have successfully created a VI which automatically loops through the contents of a folder containing '.xlsx' files and save the data, however, I have expereinced some issues and would be greatful if you could recommend any suggestions to optimise this saving process.
Issue 1:
Although there were no warnings while executing the VI this warning appeared when attemping to open the files. This was attributed to the filenames containing '.xlsx' when trying to save as a '.csv' - illustrated in the attched photo. When the filenames were manually inserted as '.csv' they would save and could be opened without any issues, however, this would require manual input making the automatic task redundant.
Question 1:
Can anyone suggest how remove the '.xlsx' from the string so that I can save the files as '.csv'?
Issue 2:
The data was recorded at 10kHz for 3s, therefore, there are 30,000 raw data points. When the '.csv' is opened the remaining 29,999 cells below the single output values are populated with zeros which signficiantly increases the size of the '.csv'.
Question 2:
Is there any way of preventing the '.csv' from populating with thousands of zeros to minimise the data size of the file?
Issue 3:
Previously I was using the Express VI write to measurment file which is why the files were initially saved as '.xlsx' - I was advised not to use this as the express VI's have a bad reputation. Since this is the first time using Write Delimited Spreadsheet VI I'm unfamilair with giving the columns titles
Question 3:
What is the best way for giving each column a heading in the '.csv' files?
@Max2796 wrote:
Introduction
I have collected a significant amount of data using my LabVIEW programme which collects 2 channels of accelerometer data. Since collecting the data I have updated some calculations in VI, therefore, the data needs to be reprocessed which would be a tedious manual task. To automatically reprocess the '.xlsx' files and save both the raw data and new outputs I have created the attached VI which I intend to apply to my existing VI. In this case the output parameters will be simplified to represent the maximum voltages from each channel. I have successfully created a VI which automatically loops through the contents of a folder containing '.xlsx' files and save the data, however, I have expereinced some issues and would be greatful if you could recommend any suggestions to optimise this saving process.
Issue 1:
Although there were no warnings while executing the VI this warning appeared when attemping to open the files. This was attributed to the filenames containing '.xlsx' when trying to save as a '.csv' - illustrated in the attched photo. When the filenames were manually inserted as '.csv' they would save and could be opened without any issues, however, this would require manual input making the automatic task redundant.
Question 1:
Can anyone suggest how remove the '.xlsx' from the string so that I can save the files as '.csv'?
Issue 2:
The data was recorded at 10kHz for 3s, therefore, there are 30,000 raw data points. When the '.csv' is opened the remaining 29,999 cells below the single output values are populated with zeros which signficiantly increases the size of the '.csv'.
Question 2:
Is there any way of preventing the '.csv' from populating with thousands of zeros to minimise the data size of the file?
Issue 3:
Previously I was using the Express VI write to measurment file which is why the files were initially saved as '.xlsx' - I was advised not to use this as the express VI's have a bad reputation. Since this is the first time using Write Delimited Spreadsheet VI I'm unfamilair with giving the columns titles
Question 3:
What is the best way for giving each column a heading in the '.csv' files?
1. Use String Functions to Find.xlsx from the file name and replace with .csv.
2. Log only meaningful values (Check before logging)
3. Write File Row value as header followed by data.
Hi Max,
@Max2796 wrote:
Issue 2:
The data was recorded at 10kHz for 3s, therefore, there are 30,000 raw data points. When the '.csv' is opened the remaining 29,999 cells below the single output values are populated with zeros which signficiantly increases the size of the '.csv'.
Question 2:
Is there any way of preventing the '.csv' from populating with thousands of zeros to minimise the data size of the file?
You convert your 2D array containing (atleast) two channels of 30k samples into a DDT wire (which is in the same way bad as using ExpressVIs!), then you index the data of those two channels into their own 2D arrays (wouldn't a 1D array be sufficient here?), then you calc the max value (scalar value!) for each channel and add them as two new channels to your 2D array.
Now you wonder why the CSV file contains two more channels with only one useful sample?
Do you know 2D arrays in LabVIEW are ALWAYS "rectangular" meaning each column has the same number of rows???
The resulting CSV file contains your rectangular 2D array. "Missing" elements are filled with the default value for the used datatype, which is a zero for numeric datatypes!
Suggestion: when you REALLY want to add two more columns with only one sample next to your two channels of 30k samples you could convert all data into a 2D array of strings: the 2D array will still be rectangular, but the default value for strings is EMPTY…
