ROCR package… what am I not getting?

问题:

I am testing a simple case using ROCR package in R. Basically, here is my code. I have a set of true values, and for each value, I have a set of predictions, and my labels are 1 if the prediction is within |2| of the true value, and 0 otherwise, like this:

  ID<- c(1,2,3,4,5)
  preds<-c(6,3,2,1,4)
  truevals<- c(8,4,2,1,7)
  df<-data.frame(ID, preds,truevals)
  df<- mutate(df, labels = ifelse(abs(df$preds - df$truevals) < 2, 1, 0))
  predtest<-prediction(df$preds, df$labels)
  auc <- performance(predtest, 'auc')

But my calculated AUC is 0, i.e.

> [e-mail][[1]]
[1] 0

My question is, what am I doing wrong? Clearly, some of the classifications are "correct", so why should the AUC be zero? What am I not understanding? My values are arranged by ID, i.e. I am assuming they are measurements of people with IDs 1 to 5. Is there some ordering issue I'm not considering? Thanks.

回答1:

To understand what's going on, draw a box plot of your data:

boxplot(preds ~ labels, data = df)

Notice how the predictions on the 0 class are higher than those of the 1 class.

Now look at the definition of the AUC from Wikipedia:

[The AUC] is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative'). (1)

Now by convention, 1s will usually be considered positives, and 0s negatives. As we just saw, your 1s, now the positives, rank lower than negatives (0s), so the probability that they are higher is 0.

You have 3 options:

A. If your 1s are the negatives, ROCR has a label.ordering argument:

predtest <- prediction(df$preds, df$labels, label.ordering = c(1, 0))
auc <- performance(predtest, 'auc')
[e-mail][[1]]
[1] 1

B. If your 1s are indeed the positives, you can reverse your predictions so that positives are higher (notice the - sign in front of df$labels):

predtest <- prediction(df$preds, -df$labels)
auc <- performance(predtest, 'auc')
[e-mail][[1]]
[1] 1

C. You could also reverse the definition of the AUC so that becomes the probability that a classifier will rank a randomly chosen positive instance lower than a randomly chosen negative one. ROCR doesn't support this but other packages do and might even choose this automatically for you.

In the end, what matters is not whether your AUC is above or below 0.5, but how far it is from the diagonal. If it is lower than 0.5, or "worse than random", you only need to reverse your interpretation to perform better than random.

回答2:

Your data is clearly separable. pred 1 to 3 go with label 1 and 4 and 6 go with label 0.

That should give you an AUC equal to 1, which is the same as an AUC of 0. It's just a matter of reference.

Here's an example:

library(ROCR)
ID = c(1,2,3,4,5)
preds = c(6,3,2,1,4)
truevals = c(8,4,2,1,7)
df = data.frame(ID, preds,truevals)
df = mutate(df, labels = ifelse(abs(df$preds - df$truevals) < 2, 1, 0))
#Changing the labels is just a matter of reference
#the algorithm is oblivious to their meaning
df$labels = 1 - df$labels
predtest = prediction(df$preds, df$labels)
auc = performance(predtest, "auc")

Output:

> auc@yvalues[[1]]
[1] 1

Switching labels raise an issue regarding leakage though, but I suppose that's not within the scope of the question.

EDIT: The AUC is a measure of separability, it's the probability you will rank a random positive instance higher than a random negative one. The ROC curve is simply x:1-Specificity and y:Sensitivity, given different threshold for classification on your predictor.

So, regarding:

[...] if I have a an arbitrary set of values and a set of of predictions of those values, how do I get an ROC curve? I'm really confused. I assume the closer the prediction, the better? I'm just not sure how to do this. I don't know how to assign classes to the true values. Don't there need to be rankings of some sort???

You have a set of binary categorical data and also a continuous predictor. Now set a threshold in the predictor, classifying observations higher than that threshold one class or otherwise the other class. Measure Specificity and Sensitivity, and mark that point in your curve. Try other thresholds (there's a finite number of possibilities that change Sens and Spec) and plot those points in the curve. That's the ROC curve.

The AUC will be higher the more separated your classes are regarding the predictor. The more superimposed they are the lower the AUC.