I'm trying to find the best parameters for NN regression model using GridSearchCV with following code:
param_grid = dict(optimizer=optimizer, epochs=epochs, batch_size=batches, init=init
grid = GridSearchCV(estimator=model, param_grid=param_grid, scoring='neg_mean_squared_error')
grid_result = grid.fit(input_train, target_train)
pred = grid.predict(input_test)
As I understand, grid.predict(input_test) uses best parameters to predict the given input set. Is there any way to evaluate GridSearchCV for each set of parameters using test set?
Actually, my test set includes some special records and I want to test the generality of the model along with the accuracy. Thank you.
You can replace standard 3-folds cv parameter of GridSearchCV with custom iterator, which yields train and test indices of concatenated train and test dataframes. In result, while 1-fold cross validation you'l train your model on input_train objects and test your fitted model on input_test objects:
def modified_cv(input_train_len, input_test_len):
yield (np.array(range(input_train_len)),
np.array(range(input_train_len, input_train_len + input_test_len)))
input_train_len = len(input_train)
input_test_len = len(input_test)
data = np.concatenate((input_train, input_test), axis=0)
target = np.concatenate((target_train, target_test), axis=0)
grid = GridSearchCV(estimator=model,
param_grid=param_grid,
cv=modified_cv(input_train_len, input_test_len),
scoring='neg_mean_squared_error')
grid_result = grid.fit(data, target)
By accessing grid_result.cv_results_ dictionary, you'l see your metrics value on test set for all grid of specified model parameters.
