For each row extract the value in the column name

Question

I have a question which can be easily solved with a for-loop. However, since I have hundred-thousands rows in a dataframe, this would take very long computational time, and thus I am looking for a quick and smart solution.

For each row in my dataframe, I would like to paste the value of the cell whose column name matches the one from the first column (INDEX)

The dataframe looks like this

> mydata
  INDEX    1   2    3   4    5   6
1     2 18.9 9.5 22.6 4.7 16.2 7.4
2     2 18.9 9.5 22.6 4.7 16.2 7.4
3     2 18.9 9.5 22.6 4.7 16.2 7.4
4     4 18.9 9.5 22.6 4.7 16.2 7.4
5     4 18.9 9.5 22.6 4.7 16.2 7.4
6     5 18.9 9.5 22.6 4.7 16.2 7.4

Here's the code for reproducing it:

mydata <- data.frame(INDEX=c(2,2,2,4,4,5), ONE=(rep(18.9,6)), TWO=(rep(9.5,6)), 
                     THREE=(rep(22.6,6)), FOUR=(rep(4.7,6)), FIVE=(rep(16.2,6)), SIX=(rep(7.4,6)))
colnames(mydata) <- c("INDEX",1,2,3,4,5,6)

And this is the new dataframe with the newly calculated variable:

> new_mydf
  INDEX    1   2    3   4    5   6 VARIABLE
3     2 18.9 9.5 22.6 4.7 16.2 7.4      9.5
2     2 18.9 9.5 22.6 4.7 16.2 7.4      9.5
1     2 18.9 9.5 22.6 4.7 16.2 7.4      9.5
5     4 18.9 9.5 22.6 4.7 16.2 7.4      4.7
4     4 18.9 9.5 22.6 4.7 16.2 7.4      4.7
6     5 18.9 9.5 22.6 4.7 16.2 7.4     16.2

I solved it using the for-loop here below, but, as I wrote above, I am looking for a more straightforward solution (maybe using packages like dplyr, or other functions?), as the loop is to slow for my extended dataset

id = mydata$INDEX
new_mydf <- data.frame()
for (i in 1:length(id)) {
  mydata_row <- mydata[i,]
  value <- mydata_row$INDEX
  mydata_row["VARIABLE"] <- mydata_row[,names(mydata_row) == value]
  new_mydf <- rbind(mydata_row,new_mydf)
}
new_mydf <- new_mydf[ order(new_mydf[,1]), ] 

Answer 1

What you want can be accomplished by:

new_mydf <- data.frame(mydata, 
                       VARIABLE=mydata[cbind(seq_len(nrow(mydata)),
                                             match(as.character(mydata$INDEX),colnames(mydata)))])

This uses subsetting with indices, which will be faster than apply. For example, if your data set is:

    INDEX Alpha Beta Charlie Delta Epsilon Foxtrot
1    Beta  18.9  9.5    22.6   4.7    16.2     7.4
2    Beta  18.9  9.5    22.6   4.7    16.2     7.4
3    Beta  18.9  9.5    22.6   4.7    16.2     7.4
4   Delta  18.9  9.5    22.6   4.7    16.2     7.4
5   Delta  18.9  9.5    22.6   4.7    16.2     7.4
6 Epsilon  18.9  9.5    22.6   4.7    16.2     7.4

This will give:

    INDEX Alpha Beta Charlie Delta Epsilon Foxtrot VARIABLE
1    Beta  18.9  9.5    22.6   4.7    16.2     7.4      9.5
2    Beta  18.9  9.5    22.6   4.7    16.2     7.4      9.5
3    Beta  18.9  9.5    22.6   4.7    16.2     7.4      9.5
4   Delta  18.9  9.5    22.6   4.7    16.2     7.4      4.7
5   Delta  18.9  9.5    22.6   4.7    16.2     7.4      4.7
6 Epsilon  18.9  9.5    22.6   4.7    16.2     7.4     16.2

To benchmark, simulate a larger data set:

## simulate some data with 1000 columns and 1000 rows
INDEX <- ceiling(runif(1000,0,1000))
data <- rep(runif(1000,0,1), each=1000)
mydata <- data.frame(INDEX=INDEX,matrix(data,nrow=1000))
colnames(mydata) <- c("INDEX", seq_len(1000))

## using indexing
system.time(new_mydf <- data.frame(mydata, VARIABLE=mydata[cbind(seq_len(nrow(mydata)),match(as.character(mydata$INDEX),colnames(mydata)))]))
##   user  system elapsed 
##  0.030   0.001   0.031 

## using apply
system.time(mydata$VARIABLE<-apply(mydata, 1, function(x) { x[names(x)==x[names(x)=="INDEX"]] }))
##   user  system elapsed 
##  0.268   0.010   0.291 

## check that we computed the same thing
all.equal(mydata,new_mydf,check.names=FALSE)
##[1] TRUE

Answer 2

Based on your loop, this use of apply with an anonymous function may be faster (with your mydata initial definition) :

mydata$VARIABLE<-apply(mydata, 1, function(x) { x[names(x)==x[names(x)=="INDEX"]] })

Edit : And it works even with INDEX in characters :

mydata <- data.frame(INDEX=c("B","B","B","D","D","E"), "A"=(rep(18.9,6)), "B"=(rep(9.5,6)), 
                 "C"=(rep(22.6,6)), "D"=(rep(4.7,6)), "E"=(rep(16.2,6)), "F"=(rep(7.4,6)))

mydata$VARIABLE<-apply(mydata, 1, function(x) { x[names(x)==x[names(x)=="INDEX"]] })

> mydata INDEX A B C D E F VARIABLE 1 B 18.9 9.5 22.6 4.7 16.2 7.4 9.5 2 B 18.9 9.5 22.6 4.7 16.2 7.4 9.5 3 B 18.9 9.5 22.6 4.7 16.2 7.4 9.5 4 D 18.9 9.5 22.6 4.7 16.2 7.4 4.7 5 D 18.9 9.5 22.6 4.7 16.2 7.4 4.7 6 E 18.9 9.5 22.6 4.7 16.2 7.4 16.2