Handling large data sets

[phalanx02895]

Hello again all.

 

I’m having a problem where a rather large program I wrote is locking up. In searching for the cause of the lockups, I’m trying to attack any possible inefficient memory use. I don’t know that the attached VIs are causing the problem, but I’m looking at everything.

 

My main program is a producer-consumer QSM. Each of the 4 main producer loops call an external VI running in parallel with the main program. This external VI processes my data and hands the results back to the main program consumer/controller loop.

 

The attached .zip file is one of these external processors (Output processor.vi) along with it's sub-VIs. The main program is taking 8k samples on 3 channels (Amps data) and passing this data to the processor. The main program then measures another 8K samples on 3 different channels (Volts data) and passes this data also.

 

The output processor sits in a loop waiting for data from the main program. When it receives the data, it summarizes the readings and puts this data into arrays of clusters. The processor also performs averaging and then passes this data back to the main program.

 This all happens 3-4 times / second, and the program will run for hours, or even days at a time. 

I think I’m following (mostly) good practices in handling this data, but I’m sure the experts here have a more memory efficient way to do this. I’d appreciate any tips or ideas in that regard.

 

One thing I noticed (and will likely implement), is going from an array of clusters to 2 arrays, 1 with timestamps and 1 with the results as singles. Is there much gain here?

 

Also, are there any known issues with having multiple queues? I currently have 7 queues for different loops.

 

Thanks in advance,

Jim

[phalanx02895]

Why no responses? Did I ask a dumb question? Did my post just get buried?

Thanks,

Jim

[nathand]

Can you be more specific than "locks up"?  Does LabVIEW crash or stop responding, or just your program?  When it "locks up" can you hit the pause button, or execution highlighting, to get an idea of what's happening?  Have you checked all your error clusters?  Many times I've thought my code locked up, and eventually discovered that instead a loop terminated early due to an error.

I've never run into an issue with multiple queues, and I use them constantly in my code.  I don't think you have much to gain from removing your array of clusters, but I didn't look through your code to see where you would be doing that.  8K samples for each of six channels is not a lot of data relative to the amount of memory in any modern machine (let's say 8 bytes/sample * 8000 samples/channel * 6 channels = 384K memory, not even half a megabyte), so even an inefficient representation that takes three times that much space is still only about 1mb, not enough to worry about.

[phalanx02895]

By locks up, the program completely stops responding. The screen won't redraw, and task manager reports it as "Not responding". I'm pretty certain it's that Labview stops responding.

Since yesterday, I've disabled different parts of my code to try and isolate the cause. So far, it appears that the issue is arising in my UUT comms loop, which is reading and writing through the serial port repeatedly. I'm still trying to confirm this, as the program lockups have occurred typically after about 4 hours of running.

Regarding my previous questions, I am still interested in seeing if anyone can recommend a more efficient method.

Thank you for your reponse regarding the queues. At least I can rule that much out.

Thanks again,

Jim

[seVEMes]

Have you run it in highlight execution mode? Everything that I found wrong from an initial highlight execution run through, is that the Init state has a hidden array constant that is empty. And by hidden I mean buried. This blank constant goes into the Q MGR VI. In the Q MGR VI it freezes at the preview queue element VI because the queue is empty and there is nothing populating the queue.

[phalanx02895]

Yeah, I've run into the empty array into Q manager scenario (several times). I believe I've got that part all worked out.

Currently, I've had to move the system away from my computer running LV. I suppose the best bet would be to move my comp to the system and see if it locks up running that way, rather than an executable.

Debugging something that can run fine for hours before locking up is a royal pain.

Thanks.

[seVEMes]

Have you checked the memory usage of your VI? If the memory after a few hours is really high then you need to see if you are handling memory properly. If you have an array that grows continuously you could save it to a file and continually apend the data to the end of the file.

[bassinbc]

I came accross this program and was wondering if you completed this? If so would you share?