I am using Spark 1.6.1:
Currently I am using a CrossValidator to train my ML Pipeline with various parameters. After the training process I can use the bestModel property of the CrossValidatorModel to get the Model that performed best during the Cross Validation. Are the other models of the cross validation automatically discarded or can I select a model that performed worse than the bestModel?
I am asking because I am using the F1 Score metric for the cross validation but I am also interested in the weighedRecall of all of the models and not just of the model that has performed best during the crossvalidation
val folds = 6
val cv = new CrossValidator()
.setEstimator(pipeline)
.setEvaluator(new MulticlassClassificationEvaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(folds)
val avgF1Scores = cvModel.avgMetrics
val predictedDf = cvModel.bestModel.transform(testDf)
// Here I would like to predict as well with the other models of the cross validation
Spark >= 2.4.0 ( >= 2.3.0 in Scala)
SPARK-21088 CrossValidator, TrainValidationSplit should collect all models when fitting - adds support for collecting submodels.
Spark < 2.4
If you want to access all intermediate models you'll have to create custom cross validator from scratch.
o.a.s.ml.tuning.CrossValidator
discards other models, and only the best one and metrics are copied to theCrossValidatorModel
.See also Pyspark - Get all parameters of models created with ParamGridBuilder
If you're just looking to do this for experimentation as opposed to a production implementation of something, I recommend monkey-patching. Here is what I did to print out the intermediate training results. Just use
CrossValidatorVerbose
as a drop-in replacement forCrossValidator
.NOTE: this solution also corrects what I see as a bug in v2.0.0 where the CrossValidationModel.avgMetrics are set to the sum of the metrics instead of the average.
Here is an example of the output for a simple 5-fold validation of
ALS
: