Lets assume we have a dataset which might be given approximately by
import numpy as np
x = np.linspace(0,2*np.pi,100)
y = np.sin(x) + np.random.random(100) * 0.2
Therefore we have a variation of 20% of the dataset. My first idea was to use the UnivariateSpline function of scipy, but the problem is that this does not consider the small noise in a good way. If you consider the frequencies, the background is much smaller than the signal, so a spline only of the cutoff might be an idea, but that would involve a back and forth fourier transformation, which might result in bad behaviour. Another way would be a moving average, but this would also need the right choice of the delay.
Any hints/ books or links how to tackle this problem?
Check this out! There is a clear definition of smoothing of a 1D signal.
http://scipy-cookbook.readthedocs.io/items/SignalSmooth.html
Shortcut:
If you are interested in a "smooth" version of a signal that is periodic (like your example), then a FFT is the right way to go. Take the fourier transform and subtract out the low-contributing frequencies:
Even if your signal is not completely periodic, this will do a great job of subtracting out white noise. There a many types of filters to use (high-pass, low-pass, etc...), the appropriate one is dependent on what you are looking for.
Fitting a moving average to your data would smooth out the noise, see this this answer for how to do that.
If you'd like to use LOWESS to fit your data (it's similar to a moving average but more sophisticated), you can do that using the statsmodels library:
Finally, if you know the functional form of your signal, you could fit a curve to your data, which would probably be the best thing to do.
A quick and dirty way to smooth data I use, based on a moving average box (by convolution):
Another option is to use KernelReg in statsmodel:
If you are plotting time series graph and if you have used mtplotlib for drawing graphs then use median method to smooth-en the graph
where
timeseries
is your set of data passed you can alterwindowsize
for more smoothining.