Read required rows from a large text file

[inana]

Hello!

I wonder if I could get your help on extracting data from text file.

I have a text file separated by rows which looks a bit like that:

0.0000012345      4

0.0000012355      4

0.0000012456      6

0.0000012569      4

0.0000012577      7

0.0000012621      4

0.0000012742      6

0.0000013489      7

0.0000013789      4

0.0000014569      7

 The text file consists of about 16 million of such rows, I need to extract the information between certain rows, for example, between rows 3-5 and 8-10, etc. I've been using a function: Read from Spreadsheet File vi. but it only allows me to read the required number of rows at a certain character offset. The number of characters per row is not fixed and therefore it's impossible to  relate the number of characters to rows.

Is there a vi that would let me access the desired rows throughout the entire file withough the need to count the charactes?

Your help would be much appreciated!

Many thanks,

Agata

[crossrulz]

You can configure the Read Text File to read lines.  It is a Righ-Click option.  So I would just read the lines that you want to ignore just to set the file pointer to the beginning of your data.  You can then read the data you care about using whatever method you want.

[DFGray]

Unfortunately, LabVIEW does not buffer file data in line read mode very efficiently, so reading lines in one at a time for a file with millions of lines is probably going to run into major performance issues.  If it works for you, go for it.  If you have performance problems, you can do the following:

1. Read a chunk of the file in.  Experience shows a 65,000 character chunk will give best performance.
2. Search through this chunk for end-of-line (EOL) characters.  Depending on the source of your file, this could be \r, \n, \r\n, or \n\r.  You can view a chunk of the file using the codes display option (right click menu) to find out which your file uses.  Or you can open the file in a text or hex editor which can show hidden characters (e.g. Notepad++).  Count these as you find them in the chunk.
3. The EOL characters serve as the boundaries for your lines.  You will need to account for lines going across chunk boundaries (and two character EOL sequences going across chunk boundaries).
4. As you find the line boundaries to the lines you want, convert the sections to whatever format you want for further use.

This may sound like a lot of work, but you will get a 2X to 10X or more performance improvement, depending upon your hardware and LabVIEW version.

There is an example of this embedded in the code on this post, but it is wrapped in a fairly complex, object based file system.  Approach with caution and read the comments on the original post before proceeding.

[camerond]

Actually, you do not have to use any programs other than LV to find the EOL character(s). Just write a quick VI which will read in from a typical file enough data so you know you'll have at least one EOL in it, and put an indicator on the FP for the data string. Right-click on the indicator and set it to show "\ Codes".

Cameron