I'm trying to know which loss function uses XGBoost for multi-class classification. I found in this question the loss function for logistic classification in the binary case.
I had though that for the multi-class case it might be the same as in GBM (for K classes) which can be seen here, where y_k=1 if x's label is k and 0 in any other case, and p_k(x) is the softmax function. However, I have made the first and second order gradient using this loss function and the hessian doesn't match the one defined in the code here (in function GetGradient in SoftmaxMultiClassObj) by a constant 2.
Could you please tell me which is the loss function used?
Thank you in advance.
The loss function used for multiclass is, as you suspect, the softmax objective function. As of now the only options for multiclass are shown in the quote below, the multi:softprob returning all probabilities instead of just those of the most likely class.
“multi:softmax” –set XGBoost to do multiclass classification using the softmax objective, you also need to set num_class(number of classes)
“multi:softprob” –same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The result contains predicted probability of each data point belonging to each class.
See https://xgboost.readthedocs.io/en/latest//parameter.html#learning-task-parameters.
A good example is given on:
http://machinelearningmastery.com/avoid-overfitting-by-early-stopping-with-xgboost-in-python/
Basically, you can define the loss function with the "eval_metric" parameter. (default being rmse for regression, and error for classification)
A description of the "error" function is given on the official github repo:
""error": Binary classification error rate. It is calculated as #(wrong cases)/#(all cases). For the predictions, the evaluation will regard the instances with prediction value larger than 0.5 as positive instances, and the others as negative instances."
A full list of eval metrics available can also be found at the "Learning Task Parameters" section of https://github.com/dmlc/xgboost/blob/master/doc/parameter.md
Hope that answers your question, best of luck,
