tdms viewer+large files issue

[krivan]

I built a VI that stores large [up to 50x50] 2D arrays continuously, every new 2D array is stored as a new group in a TDMS file.

The VI streams out the data, it's OK, however when it finishes and the TDMS Viewer pops up, the LabVIEW keeps hanging and [seemingly] doing nothing. The TDMS files that the viewer tries to display can have easily over 10^6 pieces of 2D arrays inside if the measurements were running for days for example.

Is there a clever way to make the TDMS Viewer display my file or at least say the quarter of it?

Thanks,

Krivan

[YongqingYe]

Hi krivan,

It's possibly because the TDMS file is very big. Can you share the numbers for the size of the .tdms file and the .tdms_index file? If they are really very big, one of the ways is to use TDMS Defragment to defragment the file and then open it in TDMS File Viewer.

[krivan]

Hi YongqingYe,

the last file was ~90 MB and contained 131224 groups. That means at the moment 131224 pieces of 50x50 2D arrays.

I tried to run another code that reads this TDMS file, averages every 1000 2D arrays into one 50x50 array and then stores it into another TDMS file.

The result is the same as with the TDMS viewer, the LabVIEW keeps hanging, no reaction, only the occupied memory space is increasing according to the Task Manager.

Thanks for the idea of the defragmenter, I'll try it!

[Lucither]

Can you post some code, possibly with some data to read? In LabVIEW 2009 is possible.

Rgs,

Lucither

[krivan]

Hi, this is the code that produced that big amount of data. The code works fine actually until the loop finishes and the TDMS Viewer pops up.

[krivan]

Also, if I use the code attached and I try to open that large file the LV keeps hanging [all open LV windows], the program does not enter the while loop because the open tdms VIs do nothing.

However, if I run this program with a very small sized tdms file [~300 kbyte] containing only a couple of thousands of matrixes, it runs without any problem.

I haven't tried the defragmenter so far because here it was reported to be not advisable.

I also tried to read only a small portion of the files wiring 1000 to the read VI but [in case of the large files] the problem remains the same, the operation does not get through the TDMS Open VIs.

Is there a clever way to handle large TDMS files?

Thanks,

Krivan

[Lucither]

Hi Krivan

I havent run the code but i think i can see where your problem will be. Your top loop has no timing function. It will run as fast as your pc will go, hogging all the cpu. Your bottom loop will never be able to keep pace because it has actual things to do, like write to a TDMS file. Basically your queue will be filling far quicker then your bottom loop can empty it. You need to place some sort of timing function in your top loop, otherwise your bottom loop will never be able to keep up.

Rgs,

Lucither.

[krivan]

Hi Lucither,

I don't think that it would be correct because even if I put a 50 ms delay the data is streamed out properly and when I stop the producer loop the TDMS viewer starts having problems [by the way the streaming works without timing as well but you are right, the delay is necessary in the producer loop].

Also, it still does not answers the previous post, why [in the averaging VI] the program cannot get through the TDMS Open VIs, why it cannot even enter the while loop in case of the large files.

Krivan.

[Lucither]

Krivan,

From what i can see i am seeing 2 problems. When you dont have a timing function in the top loop then you are filling the queue so quick that it hangs trying to empty. It will do this until your queue is empty and then stop (I didnt wait that long as my computer completely froze). If you do put a pause function in the top loop then your bottom loop exits because for some reason you exit when your queue is empty? Unless you are putting data in at a quicker rate then your retrieving it (The problem you are getting above) this will normally happen. This exits displaying your TDMS viewer but your top loop still carries on as the bottom loop does not stop the top loop when exiting. It seems to me that your top loop needs a reasonable pause function, enough time for your bottom loop to deal with the data in its buffer. Your bottom loop should not abort when the buffer is empty though, but either on an error or when told to do so by your producer loop.

Rgs,

Lucither.

[Lucither]

Regarding your averaging vi, have you tried inserting a timing function into this also. This too has no timing function so will try and hog all the cpu. Give it a try and see what happens.

Rgs,

Lucither.