Optimised text file read & write

[Trevor Mcalister]

Hi

I require assistance in optimise my vi for writing to a text file.
I am having to convert large amounts of saved data from binary format to text file (csv) so that the data can be processed by Mat Lab at a later stage.

I am using binary file for all my near real-time data saving, but I need to increase the speed at which I can save data in text format.
I can achieve over 400Kbytes/second with binary but I am still restricted to about 35Kbytes/second when writing to text file.
The file sizes I am dealing with is 200 - 400MB, so you can see why I am after something that tacks less the 30 minutes even on a fast CPU.

Also dose any one know a way of getting out the number of individual strings (lines) in a text file rather than the file size in bytes, which isn’t much help when you want to read a file in line chunks and the number of digits/ characters in a individual string (line) vary.

Thanks for your help
Read & write vi's attached

Trevor

[chutla]

Hi Trevor:

    I don't see how you can improve a whole lot on your vi's; they seem to be pretty lean and mean as they are.   I think you're just bumping up against processing speed.   My real question is, why do you need to convert to ASCII to work with Matlab in the first place, since matlab does such a marvelous job with binary? 

Eric

[Trevor Mcalister]

Hi

My main reasons for being able to convert to Text file (csv) was to link in with existing tools that are only csv file based, without needing another little application to convert between the data formats.

If this is the best it can get I could get our numerical analyst to do me up a Mat Lab script to convert the larger files.

Do you know a way of getting out the number of individual strings (lines) in a text file rather than the file size in bytes, this is so I can read in existing small csv files.

Trevor

[RobSims]

Hi Trevor,

Thanks for posting on the NI Forums.

I have taken a quick look at your code and I aggree with Eric, it does indeed seem pretty much as 'lean and mean'' as it gets.

With regard readinf out the number of lines within the file you can open the file, read the data out and use the Match Pattern VI (in the string function pallette) to search through the file for 'Line Feed' ACSI characters. By scanning through your file in this way you will see one line feed for each line of data in your file.

Give this a go and let me know how you get along. If you get stuck then just shout, I'll knock some example code together for ya!

All the best,

Rob

Message Edited by RobS (AE) on 07-19-2006 02:06 PM

[DFGray]

You need to double buffer your data.  The optimum chunk size to read from / write to disk (for WIndows OSes, anyway) is about 65,000 bytes.  The farther you get away from this, the worse your performance will be.  You can get order of magnitude improvements, depending upon where you are now.  It is unclear from your code exactly what your read/write sizes are.  Burrow into the LVM express VIs to get an example of this for text files.

You should be able to get at least 10MBytes/sec for binary files.  Text file speed should also improve.

Good luck.  Let us know if you need more help.

[chutla]

Good point, Mr. Gray.   This can also be tweaked somewhat by formatting the hard drive with custom block sizes.  Some systems allow you to use much larger than default block sizes.  You might be able to gain some performance by going to an older FAT file system than NTFS...it has much less overhead.   As you said, it depends on where you are NOW.

Eric