big data for calculate mean, max,min

[Tarzan-Chan]

The current problem is that when there are more and more equivalent measurement values and the standard deviation, average, maximum and minimum values are calculated, there will be more and more slow. The picture is the is my use way .Is there has other  better way? How can i speed up the calculation?

[bseguin]

Initialize your array before the loop. Then use replace array subset to insert.

You might want to do your math on a array subset according to the index of the loop.

Try and let me know.

Benoit

[GerdW]

Hi hnint,

the standard deviation, average, maximum and minimum values are calculated,

For mean, min and max you only need to store their previous (scalar!) value to calculate to new value with the next sample. No need to reevaluate the whole array again!

I guess you could also apply that to stdev and variance too, but never did that on my own before…

[aptivo]

Please keep in mind that the function for calculating mean sums up all array elements. If your array is big enough you will exceed the limit of your datatype double and you will end up with a wrong mean.

[Tarzan-Chan]

My idea is to average every 10,000 pens and then use calculated value with the new 10,000 data. I don't know if this is feasible or not. It is very similar to GerdW, but it still needs to be verified.

[Yamaeda]

Change that Insert into array to Build array and compare. The average can be checked with a running average, which only requires 1 add and 1 divide for each element added. Max and Min also only need to be check against the newest element.

/Y

[thols · Accepted solution]

There are also some point by point methods that can be used (search for example for "mean pt"). 

[Yamaeda]

@thols MeanPtByPt has an internal array of 100 elements (per default), so it might be inaccurate if the array is large.

/Y

[thols]

---

@Yamaeda wrote:

@thols MeanPtByPt has an internal array of 100 elements (per default), so it might be inaccurate if the array is large.

/Y

---

So set sample length to Inf.

[Yamaeda]

Yes, that works unless the data copy creates issues. The VI's, both MeanPtByPt and StdDevPtByPt are open code, so you can e.g. modify it to also send out the array and have it as your only data copy. 🙂

/Y