filtering/removing noise

Question

The question is simple. How do I go about removing noise from data? I have made up some x and y values along with some noise that is a gross simplification of the data I am dealing with (apart from the random noise I cannot make that the same as the noise I have to deal with). I don't really know if I need to filter or smooth. My file contains two sets of data that need to be plotted and there is experimental noise in this data, what is the best way to remove it? smoothing or filtering?

I recently posted this code in another post all I have done is added noise to it.

import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import curve_fit, minimize_scalar 

x1 = [1,2,2.5,3.2,2.8,3.5,4,5]
y = [1,4,9.2,16,16.1,9.6,4,1]
noise = np.random.normal(0,1,8)
x = x1 + noise #bring on the noise

def f(x, p1, p2, p3):
return p3*(p1/((x-p2)**2 + (p1/2)**2))   

p0 = (8, 16, 0.1) # guess perameters 
plt.plot(x,y,"ro")
popt, pcov = curve_fit(f, x, y, p0)

fm = lambda x: -f(x, *popt) #this part and below is not my code but the 
#solution to my previous question     
r = minimize_scalar(fm, bounds=(1, 5))
print "maximum:", r["x"], f(r["x"], *popt)

x_curve = np.linspace(1, 5, 100)
plt.plot(x_curve, f(x_curve, *popt))
plt.plot(r['x'], f(r['x'], *popt), 'ko')
plt.show()

say I remove the noise and replace x1 with x... I get a nice fir to my data points. How can I get close to this when noise is involved?

Answer 1

The easiest way to remove noises by using the Kalman filter. Let's say your data(measurement) has some noises. You want correct with a filter. Use the Kalman filter and change transition_covariance variable based on your data transition.

import matplotlib.pyplot as plt 
from pykalman import KalmanFilter 
import numpy as np

measurements = np.asarray([1, 2, 3, 5, 3, 2, 1, 2, 4, 5,7, 9, 10, 8, 5, 1]) 
kf = KalmanFilter(transition_matrices=[1],
                  observation_matrices=[1],
                  initial_state_mean=measurements[0],
                  initial_state_covariance=1,
                  observation_covariance=5,
                  transition_covariance=1) #0.01) 
state_means, state_covariances = kf.filter(measurements) 
state_std = np.sqrt(state_covariances[:,0]) 
print (state_std) 
print (state_means) 
print (state_covariances)

plt.plot(measurements, '-r', label='measurment') 
plt.plot(state_means, '-g', label='kalman-filter output') 
plt.legend(loc='upper left') 
plt.show()