Read file in chunks

[gigi85]

hello,

as many of you know i'm trying to solve a problem memory with labview 7.1

many of you told me to read file in chunks and i'm trying to do it...

but i have some questions...

in attachment you find see my VI (it's incomplete): i would read my files 1 Mb for each step (now my file is 80 Mb)...i would ask you what i have to link to count input of the readfile vi in the true case (the last for-cycle-step, where i have to save the last part of my file).

Should it be correct done in this way?

Have i done the save-process in the correct way? (initialize the array knowing the array size and the using the arrayreplacesubset vi in the for loop)

thanks all

[gigi85]

in my VI i have forgotten to link the refnum input in close vi 😄

[GerdW]

Hi gigi,

- you don't need to set the read position in each iteration. LabVIEW stores that file position internally as files are generally handled as byte streams...

- reading the file in chunks and then put those chunks into a big string array is no good. You need to process the chunks as they are! (Btw. your string array is a big mess, why create an array of strings?)

- creating an array of strings to use ReplaceArraySubset on doesn't help you on your memory overflow problems as you still have variable sized elements! 

- your logic for calculation of the loop iteration count is also highly flawed (use highlighting and probes...)!

[gigi85]

- you don't need to set the read position in each iteration. LabVIEW stores that file position internally as files are generally handled as byte streams...

So i don't need a for loop right? i use 1 read file vi, is it right?

- reading the file in chunks and then put those chunks into a big string array is no good. You need to process the chunks as they are! (Btw. your string array is a big mess, why create an array of strings?)

mmm ok, so i have to avoid to save my data and handle them directly? or i have to save them in other ways?

- creating an array of strings to use ReplaceArraySubset on doesn't help you on your memory overflow problems as you still have variable sized elements! 

OK

- your logic for calculation of the loop iteration count is also highly flawed (use highlighting and probes...)!

yes, i inverted the 2 outputs 😛

[gigi85]

well, after some tests i have done this little and easy VI 🙂

it reads line by line my file...my question is: there is a way to read more lines in each step?

I have done the same VI reading a tot. of memory each time...but obviously this method cut lines and it becomes more difficult analyze datas... 

[GerdW]

Hi gigi,

because of your old LabVIEW version your VIs are converted when loaded with more recent LabVIEW versions.

To judge more accurate you should also attach a picture of the block diagram of your VI, so we have a chance to look at the "unconverted"/original VI version!

[gigi85]

Ah ok sorry 🙂

here my vi...new version 😄

in the red zone the new part where i have done a concatenate string (1000 string) to have in my output 1000 lines each time - but i don't know if it's the best approach to do that...

I divided the picture in 2 zone to show both case cases 🙂

[GerdW]

Hi gigi,

right now you

- get the file size in bytes using the EOF function (any sufficient high constant will do the same)

- then you read the file line by line

- you concat the read strings into a bigger one

- after 1000 lines you delete the previously read data and start again with concatenating lines

- your loop will run way to many times as it stops when loop counter is equal to file size (in bytes!). You should rather stop on EOF marker...

When will you process your chunks of data? You should do that every time you have read 1000 lines...

[Yamaeda]

The Count input on the read file is the number of files to read, just wire 1000 to read 1k lines (it'll be less if the file ends). If you're gonna concatenate the strings there's no need to read in chunks ... the point is to do calculations and retire the data.

/Y

[gigi85]

EOF function give me the size of the file and not the offset of the EOF? 😞

so in which way could i stop the while cycle?

yes,,,i'm reading the file line by line and the i'll concatenate strings (1000 in the example) to obtain a chunk of data

if i put in the count input of the read file function (with the line mode True) i don't obtain all lines in the output...i don't know why

you can see the image in attachment

in fact was to solve my problem that i tryed to concatanate strings...if i could receive in the output 1000 lines in a automatic way should be better...

and yes...in my case i'll make my operation in the case structure, when i obtaine 1000 lines