Deleting from large array too slow

[_thomas]

Hi,

I have problem with one of my projects. I have large binary files containing measurement data. Values within binary files are written every second. Number of samples per second is constant and it equals 5000. Values are recorded using Write to Binary file.vi. Data type is array of Single precision float.

I'm reading data in 5-minute blocks. That's 5000 x 60 x 5 samples per block. I must read  data in single array. The main problem is that there is additional 4 bytes per second. That number represents number of elements in array, 5000 in my case.

I trying to remove excess elements using single For loop. I have put one Delete from array.vi. I need to remove elements with index (5000 * i) where i ranges from 0 to 299.  That's 300 loop iterations with large arrays so now I have problem with execution speed. I know that I can read my data as array of clusters but than I need to extract data back to single array.

Any suggestion for fast removal of excess number ?

[TCPlomp]

You could better store the data different.

The Write to binary file function has an input 'Prepend Array or String Size'. This default to 'True' changing this to 'False' will eliminate the additional 4 bytes.

Ton

[Pnt]

My PC needs about 1 sec to do that.

Can you post an example vi ?

[Vsh]

Are you trying to delete the elements one after another?

Instead you can delete all of them at one shot 

[Pnt]

---

Vsh wrote:

Instead you can delete all of them at one shot 

---

I think they are not in sequence.

For exmple: First one is index 5000, second 10000

[_thomas]

Pnt,

you are rigth, they are not in sequence. I'm reading up to 4 files in paralell, so execution time is 4 x (time for reading + time for removal). My computer stops for 4-5 seconds and that is to much. Most of that time is spend for removal.

I know that I can write data without array size included but I already have large number of measurement files. If removal of elements can't be done much faster my only choise is to modify existing files and change the way files are written. 

[Pnt]

I think NI should consider a growable "Delete from Array"

[Vsh]

Ok...

Then how you are deleting it?

This example takes less than a second time to execute. My data block is 5000*60*5. 

[TCPlomp]

Here is a way to read the data with skipping the bytes.

 

Oops , I made acoding error. The set file posision should be placed before the read function. 

Ton

Message Edited by TCPlomp on 05-11-2009 12:33 PM

Message Edited by TCPlomp on 05-11-2009 12:35 PM

[Pnt]

Ton, i think this is much slower than simply deleting the useless bytes.