How to get summary statistics for multiple variabl

Question

I know that there are many answers provided in this forum on how to get summary statistics (e.g. mean, se, N) for multiple groups using options like aggregate , ddply or data.table. I'm not sure, however, how to apply these functions over multiple columns at once.

More specifically, I would like to know how to extend the following ddply command over multiple columns (dv1, dv2, dv3) without re-typing the code with different variable name each time.

library(reshape2)
library(plyr)

group1 <- c(rep(LETTERS[1:4], c(4,6,6,8)))
group2 <- c(rep(LETTERS[5:8], c(6,4,8,6)))
group3 <- c(rep(LETTERS[9:10], c(12,12)))
my.dat <- data.frame(group1, group2, group3, dv1=rnorm(24),dv2=rnorm(24),dv3=rnorm(24))
my.dat

data1 <- ddply(my.dat, c("group1", "group2","group3"), summarise,
               N    = length(dv1),
               mean = mean(dv1,na.rm=T),
               sd   = sd(dv1,na.rm=T),
               se   = sd / sqrt(N)
)
data1

How can I apply this ddply function over multiple columns such that the outcome will be data1, data2, data3... for each outcome variable? I thought this could be the solution:

dfm <- melt(my.dat, id.vars = c("group1", "group2","group3"))
lapply(list(.(group1, variable), .(group2, variable),.(group3, variable)), 
   ddply, .data = dfm, .fun = summarize, 
   mean = mean(value), 
   sd = sd(value),
   N=length(value),
   se=sd/sqrt(N))

Looks like it's in the right direction but not exactly what I need. This solution provides the statistics by each group separately. What I need an outcome as in data1 (e.g. first aggregated group is people who are at A, E and I; the second is those who are at group B, E and I etc...)

Answer 1

If you don't want to melt into long format, you can also do:

library(data.table)
setDT(my.dat)[, as.list(unlist(lapply(.SD, function(x) list(mean = mean(x),
                                                            sd = sd(x),
                                                            n = .N,
                                                            se = sd(x)/sqrt(.N))))),
              by = .(group1, group2, group3), .SDcols=c("dv1","dv2","dv3")]

which gives:

   group1 group2 group3    dv1.mean    dv1.sd dv1.n     dv1.se    dv2.mean    dv2.sd dv2.n     dv2.se     dv3.mean    dv3.sd dv3.n    dv3.se
1:      A      E      I  0.09959774 0.4704498     4 0.23522491  0.05020096 0.8098882     4 0.40494412 -0.134137210 0.7832841     4 0.3916420
2:      B      E      I  0.72726477 0.3651544     2 0.25820315  0.73743314 1.4260172     2 1.00834641 -0.120188202 0.5532434     2 0.3912022
3:      B      F      I -0.68661572 0.7212631     4 0.36063157  0.06670216 0.7678781     4 0.38393905  0.096275469 0.8993015     4 0.4496508
4:      C      G      I -0.54577363 0.0798962     2 0.05649515  0.18293371 0.1022325     2 0.07228926 -0.947603264 2.3118016     2 1.6346906
5:      C      G      J  0.17434075 0.8503874     4 0.42519369 -0.11485558 1.4184031     4 0.70920154 -0.005140781 0.6871591     4 0.3435796
6:      D      G      J  0.17943465 0.4943486     2 0.34955725 -0.22223273 0.3679613     2 0.26018796 -0.373289114 1.0737512     2 0.7592568
7:      D      H      J  0.38090937 0.7904832     6 0.32271340  0.02107597 1.0094695     6 0.41211422  0.118277330 0.9024006     6 0.3684035

Answer 2

Here is a solution using dplyr. This gives the result in a "wide" format (i.e. the stats for dv1, dv2, dv3 are on the same line).

se <- function(x) { sd(x)/sqrt(length(x)) }
my.dat                                                        %>%
    group_by(group1, group2, group3)                          %>%
    summarise_each(funs(mean, sd, length, se), dv1, dv2, dv3) %>%
    ungroup

If having the stats for dv1, dv2, and dv3 on separate lines is desired, this can be modified using melt or gather (from tidyr).

Answer 3

Here's an illustration of reshaping your data first. I've written a custom function to improve readability:

mysummary <- function(x,na.rm=F){
  res <- list(mean=mean(x, na.rm=na.rm),
              sd=sd(x,na.rm=na.rm),
              N=length(x))
  res$se <- res$sd/sqrt(res$N)
  res
}

library(data.table)

res <- melt(setDT(my.dat),id.vars=c("group1","group2","group3"))[,mysummary(value),
    by=.(group1,group2,group3,variable)]

> head(res)
   group1 group2 group3 variable  mean        sd N       se
1:      A      E      I      dv1  9.75  6.994045 4 3.497023
2:      B      E      I      dv1  9.50  7.778175 2 5.500000
3:      B      F      I      dv1 16.00  4.082483 4 2.041241
4:      C      G      I      dv1 14.50 10.606602 2 7.500000
5:      C      G      J      dv1 10.75 10.372239 4 5.186119
6:      D      G      J      dv1 13.00  4.242641 2 3.000000

Or without the custom function, thanks to @Jaap

melt(setDT(my.dat),
     id=c("group1","group2","group3"))[, .(mean = mean(value),
                                           sd = sd(value),
                                           n = .N,
                                           se = sd(value)/sqrt(.N)),
                                       .(group1, group2, group3, variable)]