Average trend curve for data points in Python

Question

I'd love to reproduce a plot similar to this:

_{(source: brleader.com)}

I mean I have a set of data points and I'd love to have a curve which shows the average trend.

I tried adding random noise to the function y=2x

  from scipy import interpolate

  x=arange(0,1,1e-3)
  noise=np.random.random(len(x))
  y=2*x+noise

And then I used some of the Scipt function to interpolate data

  xnew=arange(0,1,1e-1)
  f=interpolate.UnivariateSpline(x,y)
  g=interpolate.interp1d(x,y)
  plot(x,y,'ro',xnew,f(xnew),'-',xnew,g(xnew),'--')
  show()

But the curve I get hardly resemble y=2*x. I'd love to have a smooth curve which average the data. Which method/function can I use?

Answer 1

You can try fit.py, a curve fitting package for Python.

Answer 2

Univariate looks exactly like 2x+0.5 (which is the average of your noise).

It's to be expected that interp1d varies that much with that amount of noise.

Depending on your purposes you might want to write your own moving averages instead of using stock interpolation methods; which is essentially using the average of last n data points instead of a data point.

That said, stock interpolation method to use depends on your purpose as well. Try a few and choose what serves your purpose.

Answer 3

One of the reasons why the curve doesn't look like y=2*x (I think it does, but that's subject to opinion) is that your noise is large compared to the average change in y. If you try something like:

noise=0.1*np.random.random(len(x))

(i.e make the noise smaller) or

y=5*x**2+noise

(i.e make the change in y larger), you'll see that the interpolation tracks the data better.

You might also want to check out:

http://www.scipy.org/Cookbook/SignalSmooth