Improving code efficiency wihen dealing with large array sizes, plotting large 1D waveform graphs

[blessedk]

Now having been bombarded with this wealth of information I will go for decimation 🙂 my plan is to plot every 50th data point ( Actually I intend to make this variable to be adjusted by the user depending on the step size (equivalent of delta t) My "beginner approach towards achieving this is to multiply my iteration count by 50 and wire that to the start index of "array subset" while keeping the length fixed at 50 ( or whatever the user decides). Let me know if My thoughts are OK.

[blessedk]

Sorry I should say to an index array. Nothing about length

[blessedk]

or better still maintain the array subset idea and get a mean of the elements with a length of 50 a lump that up as one point.

[blessedk]

Wouldn't the median be more representative than the mean?

[mikeporter]

It all depends on your data. Mean would represent the overall shape of the data. In other situations it might be median or even maximum.

Mike...

[Bob_Schor]

If you read my previous post where I explained "Decimate" and "Average", the main difference between "Representations" of data (either by a single element, as in "decimate", or a statistic, as in "Mean") has to do with the statistical distribution of the measure.

Let's say your data was noisy (whose data isn't?).  Let's assume you can represent it as a Gaussian Distribution with mean m and standard deviation s.  If you pick a single element to represent, say, N data points (N = 50, say), this element will have mean m and standard deviation s.  However, if you choose the mean of those 50 points, the mean will have mean m, but its standard deviation will be s/sqrt(N).

I once made a display where I used the Mean, and my students (who were using the software) were unhappy because they were using the "noise level" to tell if the electrodes were working properly (noisy = bad), and I'd "cleaned up" the signal too much.  Simple, just use Decimate (I picked the first element of the Array).

The Mean is a good measure of what statisticians call "Central Tendency".  If you have a very skewed distribution (i.e. most values are between 1 and 5, but 5% might be >100, making the Mean definitely not between 1 and 5), the Median might be better.  You need to think about the distribution of your data values, and what is important to show.

Bob Schor

[mikeporter]

A good book that, in a sense, discusses this topic at length is Tukey's Exploratory Data Analysis.

Mike...

[blessedk]

Thanks Bob/Altenbach/ Mike. One reason to go the simpler way : it takes too long to do group them into 50 elements, find the mean /median etc. Indexing out single points at regular spaced intervals runs faster. Unless someone can tell me a more efficient way to do that considering the size of arrays I am dealing with 🙂

[blessedk]

And of course, thanks Yamaeda 🙂

[mikeporter]

How big is your dataset? My blog post was playing with data of over 19000 values.

Mike...