Data processing delay after 3 hours of operation

[Chanfara]

Hello,

I developed a project that acquires data from thermocouple modules and 4-20 mA current sensors.
I used the producer-consumer pattern with queues. In the producer part, I sample with a maximum frequency of 10Hz, and in the consumer part, I display the data on a graph and I save them in an excel file.
The problem appears after 3 hours of processing, I observe a delay in the graph, it displays late data and this delay increases with time, and also the program slows down.

I want to know if there is a solution knowing that the program must be launched for several days.

Best Regards,

[GerdW]

Hi Chanfara,

---

@Chanfara wrote:

The problem appears after 3 hours of processing, I observe a delay in the graph, it displays late data and this delay increases with time, and also the program slows down.

I want to know if there is a solution knowing that the program must be launched for several days.

---

The problem appears in your VI, but you refuse to attach that VI.

What kind of help do you expect?

From your description it seems you collect all of your data and try to plot it in a graph. After ~3h you will have collected ~3*60*60*10 = 108k samples and the graph will get into problems displaying such large plots.

Is that guess right?

Suggestion:

Don't plot so large arrays! Decimate your data…

(It doesn't make sense to plot >100k samples in a graph of ~500 pixels width.)

[Chanfara]

Hello,

Sorry I did not attach the VI because the project contains almost 100 VIs and it is well structured.

Attached are screenshots of the main VI diagram.

[GerdW]

Hi Chanfara,

---

@Chanfara wrote:

Sorry I did not attach the VI because the project contains almost 100 VIs and it is well structured.

Attached are screenshots of the main VI diagram.

---

How are those screenshots related to your problem?

How should we help with those images?

You know we cannot edit/run/debug images in LabVIEW…

[Chanfara]

Hi GerdW,

I can not share the project in public.

Could you please give me your email to send you the project?

Thx.

[RTSLVU]

---

@Chanfara wrote:

Hello,

Sorry I did not attach the VI because the project contains almost 100 VIs and it is well structured.

Attached are screenshots of the main VI diagram.

---

Well structured is a matter of opinion... I see a lot of redundant code in just those two pictures. Also you are using a Timed Loop structure. As I recall those are intended for FPGA or LVRT not LabVIEW running in Windows. Here's some reasons why they are no better than a While Loop in Windows   

But Honestly GerdW is probably on the right track, you might want to think about changing your graph type to one that has a finite history or limit the number of points you plot at time goes on...

But without out seeing your code we can only guess.

[Frozen]

How does the code even work if you are using the STOP function?

We would need to see a lot more of you code to help.

Perhaps your DAQ initialization or  DAQ captures are coded incorrectly. Hard to tell without the code, or even pictures of the relevant sections.

If you are running the code on an FPGA, (looks like you might be) maybe you are running out of memory due to an ever increasing array? Again, hard to tell without the code.

Good luck.

[crossrulz]

---

@Frozen wrote:

If you are running the code on an FPGA, (looks like you might be) maybe you are running out of memory due to an ever increasing array?

---

FPGA does not have memory and your arrays must be fixed in size since you are defining hardware.  And then the use of the Event Structure all but eliminates the possibility of this being in RT.  So I am left to conclude that this code is running on Windows.

And I am horrified at the code that was shared.  Shared Variables are a huge red flag to me.  An Event Structure inside of a Timed Loop is contradictory.  The use of the Stop primitive points to issues in being able to properly shut down.

[daveTW]

So perhaps you show us a picture of the code part, where the data is stored which is passed to the graph.

And show us the part, where the data is passed from the producer to the consumer loop.

P.S. It is hard to understand, what all these parallel VIs do (in your first attached PNG). And why do you use shared variables in the 2nd picture?

[Bob_Schor]

You say that you write the data to an Excel file.  Are you using "Write Delimited Spreadsheet"?  This is not an "Excel" file, meaning a file in Excel's proprietary .xlsx format, but rather a plain text file with "rows" designated by EOL's at the end of lines, and column entries separated either by <Tab> characters or by a another delimiter, often a comma.  Files structure in this manner, especially those using commas, are often called "Comma-separated Value" (or .csv) files -- for reasons that defy logic, Microsoft (in Windows) has assigned an Icon to .csv files that closely resembles the icon for .xlsx files, causing confusion.

One thing about Write Delimited Spreadsheet is that each Write with the Append to File input True does the following:

• Opens the file.
• Goes to the end of the file (wherever it is on disk).
• Writes the data.
• Closes the file.

So each time you write a file, you have 4 file operations (slow) to do -- open, find EOF, write and close.  Even if in a Producer/Consumer loop, making the Consumer take more and more time with each call can definitely slow you down.

A more sensible way to get speedy writes (and possibly avoid using the Consumer/Producer at all is the following:

• Before entering the Producer loop, open the file.
• In the Producer loop, acquire the data and write it to the file in a binary (fast) format.
• When all of the data have been acquired, exit the Producer Loop.
• Close the file.

Note that you can keep the Producer/Consumer architecture if you want to.  Leave the Producer loop as before, and change the Consumer loop:

• Open the File.
• Enter Consumer Loop (waiting for data from Producer).
• Write the Data.  Do not use Write Delimited Spreadsheet, but use Write Binary (or Text) File.
• When you exit Consumer, close the file.

Bob Schor