Rolling window REVISITED - Adding window rolling q

问题:

I have been searching the web for methods that could create rolling windows so that I can perform a cross-validation technique known as Walk Forward Analysis for time series in a generalized manner.

However, I have not get around to any solution that incorporates flexibility in terms of 1) the window size (almost all methods have this; for example, pandas rolling or a bit different np.roll) and 2) window rolling quantity, understood as how many indexes do we want to roll the window (i.e. haven't found any that incorporates this).

I have been trying to optimize and make concise code, with the help of @coldspeed in this answer (I'm unable to comment there because I don't reach the needed reputation; hope to get there soon!), but I haven't been able to incorporate the window rolling quantity.

My thinkings:

1. I have tried with np.roll together with my below example, with no success.

2. I have also tried to modify the code below multiplying the ith value, but I don't get to fit it within the list comprehension, which I would like to maintain.

3. The example below does great for any window size, BUT, it only "rolls" the window one step ahead and I would like that it could be generalized to any step.

So, ¿is there any way to have this two parameters available within the list comprehension approach? or, ¿is there any other resource which I did not find that makes this easier? All the help is very much appreciated. My example code is the following:

In [1]: import numpy as np
In [2]: arr = np.random.random((10,3))

In [3]: arr

Out[3]: array([[0.38020065, 0.22656515, 0.25926935],
   [0.13446667, 0.04386083, 0.47210474],
   [0.4374763 , 0.20024762, 0.50494097],
   [0.49770835, 0.16381492, 0.6410294 ],
   [0.9711233 , 0.2004874 , 0.71186102],
   [0.61729025, 0.72601898, 0.18970222],
   [0.99308981, 0.80017134, 0.64955358],
   [0.46632326, 0.37341677, 0.49950571],
   [0.45753235, 0.55642914, 0.31972887],
   [0.4371343 , 0.08905587, 0.74511753]])

In [4]: inSamplePercentage = 0.4
In [5]: outSamplePercentage = 0.3 * inSamplePercentage

In [6]: windowSizeTrain = round(inSamplePercentage * arr.shape[0])
In [7]: windowSizeTest = round(outSamplePercentage * arr.shape[0])
In [8]: windowTrPlusTs = windowSizeTrain + windowSizeTest

In [9]: sliceListX = [arr[i: i + windowTrPlusTs] for i in range(len(arr) - (windowTrPlusTs-1))]

Given a window length of 5 and a window roll qty of 2, I could spec something like this:

Out [15]: 

[array([[0.38020065, 0.22656515, 0.25926935],
    [0.13446667, 0.04386083, 0.47210474],
    [0.4374763 , 0.20024762, 0.50494097],
    [0.49770835, 0.16381492, 0.6410294 ],
    [0.9711233 , 0.2004874 , 0.71186102]]),
 array([[0.4374763 , 0.20024762, 0.50494097],
    [0.49770835, 0.16381492, 0.6410294 ],
    [0.9711233 , 0.2004874 , 0.71186102],
    [0.61729025, 0.72601898, 0.18970222],
    [0.99308981, 0.80017134, 0.64955358]]),
 array([[0.9711233 , 0.2004874 , 0.71186102],
    [0.61729025, 0.72601898, 0.18970222],
    [0.99308981, 0.80017134, 0.64955358],
    [0.46632326, 0.37341677, 0.49950571],
    [0.45753235, 0.55642914, 0.31972887]]),
 array([[0.99308981, 0.80017134, 0.64955358],
   [0.46632326, 0.37341677, 0.49950571],
   [0.45753235, 0.55642914, 0.31972887],
   [0.4371343 , 0.08905587, 0.74511753]])]

(This incorporates the last array, although its lenght is less than 5).

OR:

Out [16]: 

[array([[0.38020065, 0.22656515, 0.25926935],
    [0.13446667, 0.04386083, 0.47210474],
    [0.4374763 , 0.20024762, 0.50494097],
    [0.49770835, 0.16381492, 0.6410294 ],
    [0.9711233 , 0.2004874 , 0.71186102]]),
 array([[0.4374763 , 0.20024762, 0.50494097],
    [0.49770835, 0.16381492, 0.6410294 ],
    [0.9711233 , 0.2004874 , 0.71186102],
    [0.61729025, 0.72601898, 0.18970222],
    [0.99308981, 0.80017134, 0.64955358]]),
 array([[0.9711233 , 0.2004874 , 0.71186102],
    [0.61729025, 0.72601898, 0.18970222],
    [0.99308981, 0.80017134, 0.64955358],
    [0.46632326, 0.37341677, 0.49950571],
    [0.45753235, 0.55642914, 0.31972887]])]

(Only the arrays with lenght == 5 -> However, this could be derived from the one above with a simple mask).

EDIT: Forgot to mention this also -- Something could be done if pandas rolling objects support iter metho.

回答1:

IIUC what you want, you can use np.lib.stride_tricks.as_strided to create the view of the windows size and the rolling quantity such as:

#redefine arr to see better what is happening than with random numbers
arr = np.arange(30).reshape((10,3))
#get arr properties
arr_0, arr_1 = arr.shape
arr_is = arr.itemsize #the size of element in arr
#parameter window and rolling
win_size = 5
roll_qty = 2
# use as_stribed by defining the right parameters:
from numpy.lib.stride_tricks import as_strided
print (as_strided( arr, 
                   shape=(int((arr_0 - win_size)/roll_qty+1), win_size,arr_1),
                   strides=(roll_qty*arr_1*arr_is, arr_1*arr_is, arr_is)))

array([[[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11],
        [12, 13, 14]],

       [[ 6,  7,  8],
        [ 9, 10, 11],
        [12, 13, 14],
        [15, 16, 17],
        [18, 19, 20]],

       [[12, 13, 14],
        [15, 16, 17],
        [18, 19, 20],
        [21, 22, 23],
        [24, 25, 26]]])

and for another window size and rolling quantity:

win_size = 4
roll_qty = 3
print( as_strided( arr, 
                   shape=(int((arr_0 - win_size)/roll_qty+1), win_size,arr_1),
                   strides=(roll_qty*arr_1*arr_is, arr_1*arr_is, arr_is)))

array([[[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11]],

       [[ 9, 10, 11],
        [12, 13, 14],
        [15, 16, 17],
        [18, 19, 20]],

       [[18, 19, 20],
        [21, 22, 23],
        [24, 25, 26],
        [27, 28, 29]]])

回答2:

So, giving my two cents (with all the help of @Ben.T), here goes the code to create a Walk Forward Analysis basic tool to get a view on how will your model/models perform in a more generalized manner.

Non-anchored WFA

def walkForwardAnal(myArr, windowSize, rollQty):

    from numpy.lib.stride_tricks import as_strided

    ArrRows, ArrCols = myArr.shape

    ArrItems = myArr.itemsize

    sliceQtyAndShape = (int((ArrRows - windowSize) / rollQty + 1), windowSize, ArrCols)
    print('The final view shape is {}'.format(sliceQtyAndShape))

    ArrStrides = (rollQty * ArrCols * ArrItems, ArrCols * ArrItems, ArrItems)
    print('The final strides are {}'.format(ArrStrides))

    sliceList = list(as_strided(myArr, shape=sliceQtyAndShape, strides=ArrStrides, writeable=False))

    return sliceList

wSizeTr = 400
wSizeTe = 100
wSizeTot = wSizeTr + wSizeTe
rQty = 200

sliceListX = wf.walkForwardAnal(X, wSizeTot, rQty)
sliceListY = wf.walkForwardAnal(y, wSizeTot, rQty)

for sliceArrX, sliceArrY in zip(sliceListX, sliceListY):

    ## Consider having to make a .copy() of each array, so that we don't modify the original one. 

    # XArr = sliceArrX.copy() and hence, changing Xtrain, Xtest = XArr[...]
    # YArr = sliceArrY.copy() and hence, changing Ytrain, Ytest = XArr[...]

    Xtrain = sliceArrX[:-wSizeTe,:]
    Xtest = sliceArrX[-wSizeTe:,:]

    Ytrain = sliceArrY[:-wSizeTe,:]
    Ytest = sliceArrY[-wSizeTe:,:]

Anchored WFA

timeSeriesCrossVal = TimeSeriesSplit(n_splits=5)

    for trainIndex, testIndex in timeSeriesCrossVal.split(X):
        ## Check if the training and testing quantities make sense. If not, increase or decrease the n_splits parameter. 

        Xtrain = X[trainIndex]
        Xtest = X[testIndex]

        Ytrain = y[trainIndex]
        Ytest = y[testIndex]

Then, you could just create the following (in any of the two approaches) and keep modelling:

        # Fit on training set only - The targets (y) are already encoded in dummy variables, so no need to standarize them.
    scaler = StandardScaler()
    scaler.fit(Xtrain)

    # Apply transform to both the training set and the test set.
    trainX = scaler.transform(Xtrain)
    testX = scaler.transform(Xtest)

    ## PCA - Principal Component Analysis #### APPLY PCA TO THE STANDARIZED TRAINING SET! :::: Fit on training set only.
    pca = PCA(.95)
    pca.fit(trainX)

    # Apply transform to both the training set and the test set.
    trainX = pca.transform(trainX)
    testX = pca.transform(testX)

    ## Predict and append predictions...

The one liner for a non-anchored case with generalized window rolling quantity:

sliceListX = [arr[i: i + wSizeTot] for i in range(0, arr.shape[0] - wSizeTot+1, rQty)]