a median filter that is not working as expected

[hiphip]

Hello all!

I'm using a median filter to process a very noisy signal but I'm not getting the signal I want.

The signal comes from an antenna in a noisy environment(hot, windy, dusty), after envelope detection it is transformed into voltage signal. I use ni usb-6002 to acquire the signal. Here is the snippet of the block diagram I use to process the save the data.I use the daq assistant to get the signal from usb-6002. Then I save the original data, the data processed by meditan filter.vi, and the data processed by median filter point by point separately as tdms file.

Here are two pictures of the raw data plotted using matlab.

1st plot2nd plot

as you can see the data in the first plot contains more noise.

Here are the data processed by median filter. 

the first picture shows the data from the 1st plot processed by median filter with left rank=5, right rank=5

median filter with left rank=5, right rank=5

this picture shows the data from the 2nd plot processed by median filter with left rank=10, right rank=10

median filter with left rank=10, right rank=10

I can't upload all the data cause they exceed the size limit. I have attached the 2nd data set in both tdms and m format.

By looking at these two pictures, I think the median filter handles noise that's bigger than the signal pretty well. But it does not take care of the noise that's smaller than the signal. So in the picture,  I can almost see no noise in the upper part but the lower part is still noisy. Why is that? What did I do wrong? 

[altenbach]

Sorry, I don't know how to read your files, but how long are these "glitches" if you zoom in?

(Also, since you have an array of data, your use of the ptbypt filter makes little sense. How is the DAQ configured? Sorry, since I don't have DAQ installed, I cannot look at the express panel)

[JÞB]

First, your scales are different. Explain how we are supposed to interpret the pictorial representations of the data that way.  You do realize that there is an algorithm behind the scene fitting the data to the monitor resolution right?  Of course scaling affects the number of pixels available for display and hence the algorithm changes.   I will not even guess how Matlab does that!  You did post to a LabVIEW forum at least post pictures of how LabVIEW displays the data!

Then, since we don't get to have the data let's have some statistics about the raw and filtered data and a discription of what would be adequate for post processing.  And why use the hardware you are using?  That 6002 is nice but lacks hardware timing and advanced filters. 

Sorry but I can't guess what you are expecting or how your expectations are unmet.

[Bob_Schor]

Sadly, the NI Forum people "broke" the ability to post Snippets to the Forum and have us pull them off to get working VIs (I, also, am "guilty" of ignoring the message saying "Don't post Snippets as images, attach them as files or attach the underlying VI", until Billko pointed out the problem to me).

Please post your code.  Also, post the sampling rate you want to use, and give us some idea what you would ideally like as a filter frequency (I assume you realize that a Median Filter acts like a low-pass filter, right?).  This will dictate a reasonable width of the Median "window".  

There are (at least) three possibilities why the LabVIEW result is not what you expect it to be:

1. You don't quite understand what a Median Filter is/does.
2. You haven't implemented the Median Filter so that it "does what you want", as possibly distinct from "does what you tell it to do".  Note that another way to express this possibility is "Your code is wrong".
3. There's a bug in one or more NI function that you are using (as opposed to the possibility that you are using the wrong function, or using the function in a wrong manner).

I'm betting that #3 is not the answer.  Once we see the code, and know better what you are hoping to see, we can comment further.

Bob Schor

[JÞB]

Side Note to Bob

If you follow the thread here The work-around is to upload the png with the image viewer then Drag-n-Drop to a BD

[hiphip]

hi! Thank you all for your responses!

I'm not sure where is the image viewer when writing this reply so I uploaded both the snippet and the actual vi here and I hope you guys can see it. Just incase, for the daq assistant, I set the sample rate=50 ,samples to read=50, I use RSE mode and continuous sampling. 

For the data I uploaded in the original post, the zip file contains the tdms file. The four files(tdms and m file) are of the same period but two of them are the raw data, two of them are processed by median filter with left rank=10, right rank=10(I didn't change that in the vi I uploaded). The tdms file in the zip file can be viewed by using excel importer. I haven't tried opening tdms with labview cause I wasn't quite comfortable doing that..but if it's necessary I will see how I can do that.

I hope this makes up for the confusions I caused in the original post. Please let me know if there is still anything unclear 

[hiphip]

I managed to upload another set of data in txt format. I just copy them and paste into txt cause the origianl files are too big to get uploaded. 

I'm hoping to at least get rid of the obviously "off" data points. I'm expecting the actual data to be around 0.8 to 0.9. 

[hiphip]

An update:

I get another set of data. This time for the daq assistant, I set the sample rate=100, samples to read=100, RSE mode, continuous mode. For the median filter, I set left rank=50, right rank=50, for the median filter ptbypt, I set the rank=50.

The original data looks like this

The data processed by the median filter looks like this

So I think it's obvious that since median filter is a low-pass filter, it only filters out that particular part of noise. It's a little bit difficult for me to know the spectrum of the signal because it's hard for me to install equipment and the system is set up in a "not very good" environment so I have to kind of guess the configuration of the medium filter. So waht can I do? Is the medium filter not configured correctly? Do I somehow have to know the spectrum before apply a filter?

[Blokk]

You again show us two plots with DIFFERENT Y-scales to compare. It does not make too much sense, if you want to compare signals "by eye", at least use the same y-scale! Do you understand the difference?

About your hardware config and DAQmx settings:

• I would stop using Express VIs, and switch to proper DAQmx functions along with a Producer/consumer design pattern
• As I see you use RSE terminal config, I would try Differential it might improve your signal-to-noise ratio
• Did you check your cabling is properly done? Grounding? Read the grounding consideration whitepapers: https://www.ni.com/en/shop/data-acquisition/measurement-fundamentals/analog-fundamentals/grounding-c... 
• You use a 50 Hz sampling rate. I would just increase the sampling to the possible maximum (50 kHz), and do an averaging in the Consumer loop. You might even do not need any filtering in this way...

[GerdW]

Hi hiphip,

to help you with this recommendation:

I would stop using Express VIs, and switch to proper DAQmx functions

you should read this!

(And always look at the example VIs coming with LabVIEW!)