Creating a discrete dataset from messy data

[MancPablo]

Good day Gurus!

I have some data that should be logged at 1Hz, but looks like the paste below (just showing the seconds, not all the timestamp). It has some repeats, some gaps. Never goes backwards! Is there a VI or neat way to resample that at 1Hz or process to remove repeats and fill aps? I have tried the TSA resampling VI, but thats kicking out an error. I have many weeks of 1Hz (or technically not 1Hz....) data I need to process. 

1	69.46882
2	84.88799
3	67.24147
3	35.00587
5	55.49919
6	81.31938
7	37.72144
9	39.8353
10	78.54475
11	82.92258
12	17.00052
12	28.95139
13	27.15973
14	25.52826
16	98.69732

Thanks in advance,

Paul

[wiebe@CARYA]

Do you simply want to fix the timestamps?

Is this always processing afterwards? Or should this ideally be done during measurement?

Any idea of the cause of this? Looks like the ms part of the timestamp got lost, rounding the seconds up or down. If you can fix that, the entire problem would go away. Or is this just to fix old data?

If it's just about fixing the timestamps after logging, I'd try to linear fit the data, and use the a and b of the fix to create new timestamps.

[MancPablo]

Hi There,

Its fixing old data from a completed project. I am looking into the cause, if its bad timing and the seconds are being rounded up and down, then if over the course of day it averages out and I am only losing a few seconds, I will just create a "true" time series and overwrite the timestamp and delete or add a couple of points. But it is looking like I am several minutes over or under on any given day.

Paul

[Henrik_Volkers]

My crystal ball tells me:

Data collected via serial COM port .. 'timestamp' done by host (not in sinc) by simply full second since start at serial read.

re-timestamping could be simple by min(fist) and  max (last)  values of and number of elements (and decide to coerce or to not coerce the time delta)  (ramp pattern)

or .. if values are missing .. maybe by do it in sections and use the slope as an indicator .. or do an alarm if delta is bigger than 1 or 2 (or ?) times the mean delta ...

Q&D

EDIT:  If the old data is collected as assumed or has another asynchronious source, the new timestamps should't be created with coerced values.

   if you have more 'simultanious' data from other (asynchronious) sources, you migth find that the number of samples isn't the same, due to minor different actual samplerates.

   Depending on the type of data you can resample them if needed.

[MancPablo]

Thanks to Germany from the UK.

I do have lots of other data collected simultaneously, but often on different machines and I would have to check them first as well. 

It is simple data acquisitions with serial ports and PC time rather than some external timer, so your crystal ball was correct.

I'll have a look at the code you sent and the coerce options.

Paul

[Henrik_Volkers]

If you assume a constant data source rate (about 1s) and have longer datasets, I think a linear fit would be the best you can do.

Taking the PC as the reference with a timestamp of +- 1s the difference to the linear fit shouldn't be bigger than 1s.

7 minutes difference in a day is just a bad clock (hopefully not drifting too much with temperature as ceap µController RC timers do) but the difference plot will show it.

attached the vi that show the differences ...   have fun

EDIT: made current values default and resaved

[wiebe@CARYA]

---

@MancPablo wrote:

Its fixing old data from a completed project. I am looking into the cause, if its bad timing and the seconds are being rounded up and down, then if over the course of day it averages out and I am only losing a few seconds, I will just create a "true" time series and overwrite the timestamp and delete or add a couple of points. 

---

---

@MancPablo wrote:

But it is looking like I am several minutes over or under on any given day.

---

I think those are two (related) problems.

You have a rounding problem, resulting in the same timestamp for different samples. This could be as simple as logging one or two extra digits. It would depend on the details.

The other one is the different clock of devices. That one is very hard to solve, although simple (poor) solutions could work just fine for some setups.

At some point I calculated the actual dT of several input streams with a PID loop... You can also upsample the data with a few points (with a FFT), but that changes the original data too. In some situations (slow processes) you might even get away with tossing or duplicating a sample every now and then.

The biggest problem is explaining this problem to other people (managers are people too 😊) who don't have the problem, and don't have to fix the problem. I've been told over and over again to simply ignore it, that I am 'inventing' problems. But then they start complaining that the live signals are shifted 2 minutes behind after a week. Yes, that is the problem...