Weka: Results of each fold in 10-fold CV

Question

For Weka Explorer (GUI), when we do a 10-fold CV for any given ARFF file, then what Weka Explorer provides (as far as I can see) is the average result for all the 10 folds.

Q. Is there any way to get the results of each fold? For instance, I need the error rates (incorrectly identified instances) for each fold.

Help appreciated.

Answer 1

Weka Explorer does not have an option to give the results for individual folds when using the crossvalidation option, there are some workarounds. If you explicitly don't want to change any code, you need to do some manual fiddling, but I think this gives more or less what you want

1. Instead of Cross-validation, select Percentage split and set it to 90%
2. Start classifier
3. Click More options... and change the Random seed for XVal / % Split value to something you haven't used before.
4. Repeat ten times.

This is not exactly equivalent to 10-fold crossvalidation though, since the pseudo-folds you make this way might overlap.

An alternative that is equivalent to crossvalidation, but more cumbersome, would be to make 10 folds manually by using the unsupervised instance filter RemoveFolds or RemoveRange. Generate and save 10 training sets and 10 test sets. Then for every fold, load the training set, select Supplied test set in the classify tab, and select the appropriate test fold.

Answer 2

I think this is possible using Weka's GUI. You need to use the Experimenter though instead of the Explorer. Here are the steps:

1. Open the Experimenter from the GUI Chooser
   • Create a new experiment (New button @ top-right)
   • [optional] Enter a filename and location in the Results Destination to save the results to
   • Set the Number of (cross-validation) folds to your liking (start experimenting with 2 folds for easy results)
   • Add your dataset (if your dataset needs preprocessing then you should do this in the Explorer first and then save the preprocessed dataset)
   • Set the Number of repetitions (I recommend 1 to start of with)
   • Add the algorithm(s) you want to test (again start easy, start with one algorithm)
2. Go to the Run tab and Start the experiment and wait till it finishes
3. Go to the Analyse tab and import the experiment results by clicking Experiment (top-right)
   • For Row select: Fold
   • For Column select: Percent_incorrect or Number_incorrect (or any other measure you want to see)
   • You now see the specified results for each fold