I'm using Gridsearch function from hypopt package to do my hyperparameter searching using specified validation set. The default metric for classification seems to be accuracy (not very sure). Here I want to use F1 score as the metric. I do not know where I should specify the metric. I looked at the documentation but kind of confused.
Does anyone who are familiar with hypopt package know how I can do this? Thanks a lot in advance.
from hypopt import GridSearch
log_reg_params = {"penalty": ['l1'], 'C': [0.001, 0.01]}
opt = GridSearch(model=LogisticRegression())
opt.fit(X_train, y_train, log_reg_params, X_val, y_val)
The default metric of the hypopt
package is the the score()
function for whatever model you use, so in your case it is LogisticRegression().score()
which defaults to accuracy.
If you upgrade the hypopt package to version 1.0.8 via pip install hypopt --upgrade
, you can specify any metric of your choosing in the scoring
parameter of GridSearch.fit()
, for example, fit(scoring='f1')
. Here is a simple working example based on your code that uses the F1 metric:
from hypopt import GridSearch
param_grid = {"penalty": ['l1'], 'C': [0.001, 0.01]}
opt = GridSearch(model=LogisticRegression(), param_grid = param_grid)
# This will use f1 score as the scoring metric that you optimize.
opt.fit(X_train, y_train, X_val, y_val, scoring='f1')
hypopt
supports most any scoring function that sklearn
supports.
- For classification,
hypopt
supports these metrics (as strings): 'accuracy', 'brier_score_loss', 'average_precision', 'f1', 'f1_micro', 'f1_macro', 'f1_weighted', 'neg_log_loss', 'precision', 'recall', or 'roc_auc'.
- For regression,
hypopt
supports: "explained_variance", "neg_mean_absolute_error", "neg_mean_squared_error", "neg_mean_squared_log_error", "neg_median_absolute_error", "r2".
You can also create your own metric your_custom_score_func(y_true, y_pred)
by wrapping it into an object like this:
from sklearn.metrics import make_scorer
scorer = make_scorer(your_custom_score_func)
opt.fit(X_train, y_train, X_val, y_val, scoring=scorer)
You can learn more in the hypopt.GridSearch.fit()
docstring here:
- https://github.com/cgnorthcutt/hypopt/blob/master/hypopt/model_selection.py#L240
You can learn more about creating your own custom scoring metrics here:
- Example: https://github.com/cgnorthcutt/hypopt/blob/master/tests/test_core.py#L371
- Source code: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html