Here is my VI. It was written to read and decode text files in the specific format generated by my SPICE program (an electrical circuit simulation program). This is a subVI to a larger program which does a lot of display and data manipulations.
The largest file I had readily available was 191.9 MB on disk. It takes about 15 seconds to read, parse and convert that file to a 2D array of Extended precision floating point numerics. The array size is 126 x 68063 which occupies about 86 MB (depending on platform, as Extended datatype representation varies). The Profile Performance and memory tool shows the VI using 257.48 MB. According to the OS LV occupies about 1.4 GB while running this program. The main program probably makes a couple of extra copies.
The Open state uses a file dialog to select the file to be read and reads the first 7 lines, which contain part of the header. The Get # Variables state parses the header to determine the size of the rest of the header and the size of the data array. Get Variable Names reads the reaminder of the header. Init Data Array initializes one of two data arrays depending on whether the data is real or complex. Read Data Chunk reads the rest of the file 1000 lines at a time, converts the strings to numerics, and places the data in the appropriate array. The Close state closes the file and the Error state handles the EOF error which I used as a normal termination.
This VI will not work directly on your files because the format is different but this should give you an idea of how to approach managing the larger files.
Lynn
