specify scoring metric in GridSearch function with

Question

I'm using Gridsearch function from hypopt package to do my hyperparameter searching using specified validation set. The default metric for classification seems to be accuracy (not very sure). Here I want to use F1 score as the metric. I do not know where I should specify the metric. I looked at the documentation but kind of confused.

Does anyone who are familiar with hypopt package know how I can do this? Thanks a lot in advance.

from hypopt import GridSearch

log_reg_params = {"penalty": ['l1'], 'C': [0.001, 0.01]}
opt = GridSearch(model=LogisticRegression())
opt.fit(X_train, y_train, log_reg_params, X_val, y_val)

Answer 1

The default metric of the hypopt package is the the score() function for whatever model you use, so in your case it is LogisticRegression().score() which defaults to accuracy.

If you upgrade the hypopt package to version 1.0.8 via pip install hypopt --upgrade, you can specify any metric of your choosing in the scoring parameter of GridSearch.fit(), for example, fit(scoring='f1'). Here is a simple working example based on your code that uses the F1 metric:

from hypopt import GridSearch

param_grid = {"penalty": ['l1'], 'C': [0.001, 0.01]}
opt = GridSearch(model=LogisticRegression(), param_grid = param_grid)
# This will use f1 score as the scoring metric that you optimize.
opt.fit(X_train, y_train, X_val, y_val, scoring='f1')

hypopt supports most any scoring function that sklearn supports.

• For classification, hypopt supports these metrics (as strings): 'accuracy', 'brier_score_loss', 'average_precision', 'f1', 'f1_micro', 'f1_macro', 'f1_weighted', 'neg_log_loss', 'precision', 'recall', or 'roc_auc'.
• For regression, hypopt supports: "explained_variance", "neg_mean_absolute_error", "neg_mean_squared_error", "neg_mean_squared_log_error", "neg_median_absolute_error", "r2".

You can also create your own metric your_custom_score_func(y_true, y_pred) by wrapping it into an object like this:

from sklearn.metrics import make_scorer
scorer = make_scorer(your_custom_score_func)
opt.fit(X_train, y_train, X_val, y_val, scoring=scorer)

You can learn more in the hypopt.GridSearch.fit() docstring here:

• https://github.com/cgnorthcutt/hypopt/blob/master/hypopt/model_selection.py#L240

You can learn more about creating your own custom scoring metrics here:

• Example: https://github.com/cgnorthcutt/hypopt/blob/master/tests/test_core.py#L371
• Source code: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html