How to solve memory full issue while reading tdms file

[ACAT02]

Hi,

Basically, I am trying to develop an application to write & read multiple waveforms continuously,

But due to continuous writing, tdms file size is increasing, thus leading to memory full issue & also increases the application memory usage.

Is there a better way to implement this application?

Thank you.

[simo22]

Hi,

The better way is to use a QMH model, Producer Consumer

I've made an example of its use, 

see the attached pic.

[mcduff]

Look at the example in the example finder "Concurrent access to TDMS File".

In your read make sure your are reading chunks of the data instead of the whole file, if you keep trying to read the whole file every time, then yes, eventually you will run out of memory.

mcduff

[ACAT02]

Hi mcduff,

Thank you for the suggestion of use of reading data in chunks,

While reading the file in chunks, I am able to read the file only once, thus unable to update waveforms in Read VI, Please may I know a better solution for continuous read operations.  

[mcduff]

Cannot help much without seeing the VI.

1. What exactly are you doing?
2. Read the first 10k points, then the next 10k points, etc.
3. What are you doing after you read the chunks? Are you just appending them to an array? You will run out of memory. Are you displaying them on a plot? If you are displaying the points you need to decimate your array.
4. Make an array beforehand of the the amount of data you want to store, and replace the dummy data with your data from the file, that way the same memory is used.

mcduff

[ACAT02]

Hello mcduff,

As you suggested, I have tried to read only required number of points, thus avoided the memory crash issue,

But I also want to continuously update the charts in the read VI, for example if any pulse width changes are made write VI, it should be reflected in read VI.

As of now charts in read VI is being updated only once, find the same in the attachment.

Please help me to tackle this issue.

Thank you.

[simo22]

Hi ACAT02,

You want to write data and at the same moment read from the same tdms file or just when you have pulse width variation,

as i suggested you can use a QMH model where you can add another consumer loop to display your graph or even use a DVR (Data Value Reference) in your Read VI, and have a sufficient ressources to do so,

also in your writing VI do you really need to display the Charts?

[mcduff · Accepted solution]

If you continuously to write your file, and try to read it ALL, you will eventually run out of memory.

In your read vi, VI_2, you are reading the same points over and over again. You need to account for the offset, see below. You should decimate your data, your plots are about 200 pixels wide and you want to plot 10000 points on those 200 pixels, that is 50 points per pixel!

mcduff

[ACAT02]

Thank you mcduff for the suggestions, it has solved major part of my application.

Thank you simo22 for your suggestions, I will try to work on it.

Is there a way to put a limit on tdms file size, and on reaching max file size, Is it possible to delete old data & store new data?  

[Yamaeda]

---

@ACAT02 wrote:

Thank you mcduff for the suggestions, it has solved major part of my application.

Thank you simo22 for your suggestions, I will try to work on it.

Is there a way to put a limit on tdms file size, and on reaching max file size, Is it possible to delete old data & store new data?  

---

What you usually do is creating a new file every day or hour or similar. But you can check for data amount instead, and if reached, Close the TDMS file and open a new one.

/Y