How to perform GridSearchCV with cross validation

问题:

I am performing hyperparameter tuning of RandomForest as follows using GridSearchCV.

X = np.array(df[features]) #all features
y = np.array(df['gold_standard']) #labels

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

param_grid = { 
    'n_estimators': [200, 500],
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_depth' : [4,5,6,7,8],
    'criterion' :['gini', 'entropy']
}
CV_rfc = GridSearchCV(estimator=rfc, param_grid=param_grid, cv= 5)
CV_rfc.fit(x_train, y_train)
print(CV_rfc.best_params_)

The result I got is as follows.

{'criterion': 'gini', 'max_depth': 6, 'max_features': 'auto', 'n_estimators': 200}

Afterwards, I reapply the tuned parameters to x_test as follows.

rfc=RandomForestClassifier(random_state=42, criterion ='gini', max_depth= 6, max_features = 'auto', n_estimators = 200, class_weight = 'balanced')
rfc.fit(x_train, y_train)
pred=rfc.predict(x_test)
print(precision_recall_fscore_support(y_test,pred))
print(roc_auc_score(y_test,pred))

However, I am still not clear how to use GridSearchCV with 10-fold cross validation (i.e. not just apply the tuned parameters to x_test). i.e. something like below.

kf = StratifiedKFold(n_splits=10)
for fold, (train_index, test_index) in enumerate(kf.split(X, y), 1):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]

OR

sinceGridSearchCV uses crossvalidation can we use all X and y and get the best result as the final result?

I am happy to provide more details if needed.

回答1:

You should not perform a grid search in this scenario.

Internally, GridSearchCV splits the dataset given to it into various training and validation subsets, and, using the hyperparameter grid provided to it, finds the single set of hyperparameters that give the best score on the validation subsets.

The point of a train-test split is then, after this process is done, to perform one final scoring on the test data, which has so far been unknown to the model, to see if your hyperparameters have been overfit to the validation subsets. If it does well, then the next step is putting the model into production/deployment.

If you perform a grid search within cross-validation, then you will have multiple sets of hyperparameters, each of which did the best on their grid-search validation sub-subset of the cross-validation split. You cannot combine these sets into a single coherent hyperparameter specification, and therefore you cannot deploy your model.

回答2:

sinceGridSearchCV uses crossvalidation can we use all X and y and get the best result as the final result?

No, you should not tune your hyper parameter (either by GridSearchCV or single gridSearch()) because model will choose the hyper parameter which can work best on the test data as well. The real purpose of test data is lost by this approach. This model performance is not generalize-able one since it has seen this data during hyper parameter tuning.

Look at this documentation for better understanding of hyper parameter tuning and cross validation.

Some pictures from documentation: