Best way to limit an array size?

[mthheitor]

Hello,

I'm trying to optimize a software that collect, organize and display data obtained from a PLC.

In the beginning the software runs very well, but after 10 min the CPU Usage goes to 20-30% and the User Interface becomes very slow (almost not responsive)

PC config = i7-9700 3.00GHz with 8GB DDR4

I'm focusing in two consuming things: array and xy graph. In this post I'm trying to find the best way to limit the array size from an 2D Array

Details of my application

1. The data comes from a PLC in a 1D Array with variable array size on each iteration

2. LabVIEW organize the data into a 2D Array with 5 Columns (Time, Sensor1, Sensor2, etc.)

3. The number of Rows is the target to limit. More Rows = More time displayed in the XY Graph

In my software the current max number of rows is 100k (that can display around 15 min of data)

Current I'm using a routine like this to limit the 2D Array Size:

Is this the best way to limit an array size?

[Bob_Schor]

Several questions:

• Why is the basic "storage element" an array of 2 values, a number (you used a random number generator in PLC Data) and an elapsed Time (in clock ticks, a Real related to the TimeStamp function).  Do you need time accuracy better than 1 ms?  If not, I recommend the Millisecond Timer instead of High Resolution Relative Seconds (4 bytes compared to 8 bytes).
• Since the PLC data is "closely bound" to the calculated "Elapsed Time", I recommend you use a Cluster of "PLC Value" (Dbl) and "t" (I32, if using Tick Count as the clock).  Now your For Loop spits out a 1D Array of "Data Points", no need to do a Transpose, no need for 2D data for a logically-1D data structure.
• A structure that will save "the last N values" that you feed it is a Lossy Queue.  The virtue of using such a structure are several-fold:
  • You decide, in advance, how much data you want to save.
  • You enqueue your data right in the For loop that generates/acquires it.  If you take more than the size of your Queue, it keeps the last N, and if less, the Queue is simply less full.
  • When the For Loop exits, you can pass the Queue to the "Do something with these data points" routine.  Almost perfect for a Producer/Consumer (note the "Almost" -- you can't "Consume" until the Producer finishes "Producing".
  • Now processing the Data becomes much simpler, as you have only a 1D Array, containing Y, t pairs (Y being the measured value that you acquired at time t, so if you were graphing the data, Y would go on the Y axis, t would go on the X axis).

I don't have LabVIEW 2012, and when I try to "Save for Previous Version, it fails (because I use the 2019 "Time Delay" function, which can be "done by hand" but requires a Frame Sequence to tie it to the Error Line, though, now that I think about it, isn't really necessary for this Demo.  I just wanted to "slow this down" so that I'd have reasonable millisecond values for the "t" parameter).

When this finishes, it will generate "Buffer Size" (default 1000) points with t increasing by 0 to 20 ms per sample.  The default Overflow is 100, so it runs for 1100 loops, taking about 11 seconds, and shows you 1000 points starting at a t value near 1000, ending near 11000.

Bob Schor

[Henrik_Volkers]

Some hint's, your PC is shifting data all the time, you need to avoid the add to array.

Init your array once at the start with max size, use it as a ring buffer with pointers ..  EDIT : the lossy queue Bob mentioned is the way to go

If it's only to show the data: the chart can do that for you

BUT   your screen has 2k-4k pixel for 15 min data (if you want to show all) no need to hold 100k values.   Ok it's 2023 , memory is cheap ,  but do need to store them as DBL?  

Diagram update rate, reduce it as much as acceptable (if you handle 500k values)  

[Hooovahh]

I have a set of Array tools I posted which has a VIM which is made to force a 1D array size to be a size.  This can be a minimum, or a maximum size and will then trim or add from the beginning or end.

https://www.vipm.io/package/hooovahh_array_vims/

I now realize having a 2D array version of this might be useful too.  Under the hood it just does array size checking and isn't very fancy.

[mthheitor]

Sorry @Bob_Schor and @Henrik_Volkers, I think my example VI caused you to focus in the subVI.

The subVI was only an example that I made to have some data to visualize.

The input data of my application is very different from that subVI, there's no Elapsed Time calculation, the Data Type are defined by the PLC (PLC data comes in float array)

I apologize for my mess

I've made another example to replace the first one.

Trying to find the better way to keep the last 100k values of the output array

I've removed the subVI, the While Loop and other things that could cause misunderstanding.

Think in this VI as a subVI of a bigger application.

Input data = PLC Data, comes in a 1D Array with data type defined by the PLC and have fixed size (defined by the PLC too). Let's say that this input array have 1000 elements on each iteration.

I only need a few of these elements, that's why I add the "For Loop", to get the elements that I want. Each element represents one sensor or variable of my process.

In my example the "Time" is the element 21,"Sensor1" is the element 30 and etc.

With "Time" + "Sensor1" data I can plot one xy_graph

With "Time" + "Sensor2" data I can plot another xy_graph

With "Time" + "Sensor1" + "sensor3" + "Sensor4" I can plot another xy_graph

With "Sensor4" + "Sensor5" I can plot another xy_graph... and so on

The output of the "For Loop" is an 1D array with only the elements that I want to store and plot

My doubt starts after the "For Loop":

What I am currently doing is to make a 2D-Array to store the data. The column 0 of the array stores "Time", column 1 stores "Sensor1" data, column 2 stores "Sensor2" data and so on

If I leave the 2D array without size limit, the LabVIEW closes by memory RAM or high CPU usage

What I'm doing for limit the array size is the "In Place Element" structure.

• In the VI of the image "Array (limit size v2).vi", the IPE structure is discarding the oldest value of the 2D-Array

• Is there a better way to discard the oldest value before adding the new one. With this I can keep the array with the right limit_size

Note: This is only one part of my doubt, I am breaking my doubts in more parts, to stay simple

In the future I'm gonna make an example that is more similar to my application.

[Bob_Schor]

Sorry, @mthheitor, I don't think you paid attention to Henrik's and my attempts to demonstrate some methods to limit an array size.  Here's one -- create an array of the pre-determined Maximum Array Size and embue it with the following characteristics:

• If the Array isn't full, when you add to it, it goes to the end.
• If the Array is full, when you add to it, it overwrites the "oldest" element, and the "next-to-oldest" becomes the new "oldest".

There are at least two ways to do this.  One is to implement what is called "a Circular Buffer", which is an Array with two "pointers", one to the "first element", the other to the "last element".  I'll let the name of this "circular buffer" suggest how it is implemented and used.  The other (which probably is implemented as a Circular Buffer) is a Lossy Queue.  We've shown you both techniques, but you failed to understand how to use them in your application.  Either that, or you have so confused us that we cannot figure out why our answers aren't the solution you require.  Oh, well, we tried ...

Bob Schor

[diman27]

As mentioned by Henrik_Volkers, if you only need to display data, you can use Chart. The chart not only displays data but can also act as a fixed size FIFO. The size of Chart history can be predefined in it properties.

If the number of channels is fixed, you can convert the PLC DBL array to a cluster of DBLs and connect it to your Chart.

Don`t forget to define number of plots in array to cluster conversion utility.

Moreover, if you also need the history of the data, you can get it from the chart's "History" property node.

(PLC data comes in float array) - if your PLC return floats (32 bit) - use SGL type instead of DBL 

[mthheitor]

@Bob_Schor I think that I'm starting to understanding the concept of Lossy Queue. Don't noticed that there was a function called "Lossy Enqueue Element"

That's the vi to store 100k of the PLC Data. I think that's the basic solution, right?

Array limit size using Lossy Enqueue Element

[mthheitor]

@diman27, don't know if I can use Waveform Chart or Graph, because the Time (dt) isn't constant.

Searching for the difference of the three (chart, graph, xy graph):

A chart is typically used for when you acquire a single point for 1 or more channels at a regular interval. A graph is used when you acquire multiple points for 1 or more channels at a regular interval. An XY graph is used when you sample at irregular intervals

Displays data typically acquired at a constant rate.
• Graph: displays one or more plots of evenly sampled measurements (does not store historical data)

• Chart: maintains a history of data, or buffer, from previous updates (Chart History Length)

• XU Graph: Displays data acquired at a non-constant rate and data for multivalued functions 

---

The instruction OPC-UA Read sometimes gives 100 ms, 200 ms or 500 ms. It depends of what PLC is doing.

The time variable is obtained by one variable of the PLC (in my example it's 21 of the PLC Data Array)

I'm using this time variable to plot a XY Graph

And there's some graphs that the X-Axis isn't time, but the sensor measurement.

Like Sensor1 (X-Axis) vs Sensor2 (Y-Axis)

[Hooovahh]

On the topic of buffers, there are several tools people have made online for handling this.  I experimented with XNodes over on LAVA here. I then turned this into VIMs and posted it here.  Dr. Powell also posted a VIM version here.