stop xgboost based on eval_metric

Question

I am trying to run xgboost for a problem with very noisy features and interested in stopping the number of rounds based on a custom eval_metric that I have defined.

Based on domain knowledge I know that when the eval_metric (evaluated on the training data) goes above a certain value xgboost is overfitting. And I would like to just take the fitted model at that specific number of rounds and not proceed further.

What would be the best way to achieve this ?

It would be somewhat in line with the early stopping criteria but not exactly.

Alternately, if there is a possibility to get the model from an intermediate round ?

Here is an example to better explain by question. (Using the toy example that comes with xgboost help docs and using the default eval_metric)

library(xgboost)
data(agaricus.train, package='xgboost')
train <- agaricus.train
bstSparse <- xgboost(data = train$data, label = train$label, max.depth =   2, eta = 1, nthread = 2, nround = 5, objective = "binary:logistic")

Here is the output

[0] train-error:0.046522
[1] train-error:0.022263
[2] train-error:0.007063
[3] train-error:0.015200
[4] train-error:0.007063

Now lets say from domain knowledge I know that once the train error goes below 0.015 (third round in this case), any further rounds only lead to over fitting. How would I stop the training process after the third round and get hold of the trained model to use it for prediction over a different dataset ?

I need to run the training process over many different datasets and I have no sense of how many rounds it might take to train to get the error below a fixed number, hence I can't set the nrounds argument to a predetermined value. Only intuition I have is that once the training error goes below a number I need to stop further training rounds.

Answer 1

# In the absence of any code you have tried or any data you are using then try something like this:

require(xgboost)
library(Metrics) # for rmse to calculate errors

# Assume you have a training set db.train and have some feature indices of interest and a test set db.test
    predz <- c(2,4,6,8,10,12)
    predictors <- names(db.train[,predz])
# you have some response you are interested in
    outcomeName <- "myLabel"

# you may like to include for testing some other parameters like: eta, gamma, colsample_bytree, min_child_weight

    # here we look at depths from 1 to 4 and rounds 1 to 100 but set your own values

        smallestError <- 100 # set to some sensible value depending on your eval metric

        for (depth in seq(1,4,1)) {
        for (rounds in seq(1,100,1)) {

                    # train
                    bst <- xgboost(data = as.matrix(db.train[,predictors]),
                                   label = db.train[,outcomeName],
                                   max.depth = depth, nround = rounds,
                                   eval_metric = "logloss",
                                   objective = "binary:logistic", verbose=TRUE)
                    gc()

                    # predict
                    predictions <- as.numeric(predict(bst, as.matrix(db.test[,predictors]), outputmargin=TRUE))
                    err <- rmse(as.numeric(db.test[,outcomeName]), as.numeric(predictions))

                    if (err < smallestError) {
                            smallestError = err
                            print(paste(depth,rounds,err))
                     }     
              }
       }  

# You could adapt this code for your particular evaluation metric and print this out to suit your situation. Similarly you could introduce a break in the code when some specified number of rounds is reached that satisfies some condition you seek to achieve.