How to continuously save data(total file size bigger than 4G) in serveral mintues

[Frogreen]

We need acquire and save data continuously from PMT in serveral mintues. Two channels, AI rate of >500k samples/s, AC signals. Usually, we save the data to a binary file. But if the data size was bigger than 4 G, saving would be failed. Could you please tell me what other method I should use? Thanks.

[Tom__Sedlack]

Some things I do, that may or may not apply to your situation, when I have to deal with large amounts of data are:

1)  Evaluate what you are saving to see if something can be eliminated.  Are you using hardware timing on your aquisition?  If so, you know the time interval between samples is constant so don't store the time (if you are).

2)  How are the values being stored?  Integers are smaller then singles (See here).  I don't know of any A/D converters that have better then 16 bit resolution so just store the raw 16-bit integer rather then a converted voltage.  Do the conversion off-line.

3) If you are still over-running, add some intelligence to the data file creation.  You should be able to estimate how much data you will need to store.  Have a few empty files already created and when one fills up, close it and open the next.

[UFOslava]

I have similar situation...

My project supposed to be capable of communicating with up to 25 devices at the same time, 4 times per second, via RS232 and ethernet, and recieve large amounts of data in return. All this may continue unlimited amount of time, so i need a data storage solution, that is not based on arrays. after some tests i conducted, arrays may run out of cells, and the computer may run out of ram holding smaller arrays.

Is there any way to quickly write and manage small amounts of data, so it will not clog the program flow?

Or maybe file appending system can write without preloading the file itself?