I am trying to use foreach and am having problems making the .combine function scalable. For example, here is a simple combine function
MyComb <- function(part1,part2){
xs <- c(part1$x,part2$x)
ys <- c(part1$y,part2$y)
return(list(xs,ys))
}
When I use this function to combine a foreach statement with an iterator other than 2 it returns it incorrectly. For example this works:
x = foreach(i=1:2,.combine=MyComb) %dopar% list("x"=i*2,"y"=i*3)
But not this:
x = foreach(i=1:3,.combine=MyComb) %dopar% list("x"=i*2,"y"=i*3)
Is there a way to generalize the combine function to make it scalable to n iterations?
Your .combine function must take either two pieces and return something that "looks" like a piece (could be passed back in as a part) or take many arguments and put all of them together at once (with the same restrictions). Thus at least your MyComb must return a list with components x and y (which is what each piece of your %dopar% do.
A couple of ways to do this:
MyComb1 <- function(part1, part2) {
list(x=c(part1$x, part2$x), y=c(part1$y, part2$y))
}
x = foreach(i=1:3,.combine=MyComb1) %dopar% list("x"=i*2,"y"=i*3)
This version takes only two pieces at a time.
MyComb2 <- function(...) {
dots = list(...)
ret <- lapply(names(dots[[1]]), function(e) {
unlist(sapply(dots, '[[', e))
})
names(ret) <- names(dots[[1]])
ret
}
s = foreach(i=1:3,.combine=MyComb2) %dopar% list("x"=i*2,"y"=i*3)
x = foreach(i=1:3,.combine=MyComb2, .multicombine=TRUE) %dopar% list("x"=i*2,"y"=i*3)
This one can take multiple pieces at a time and combine them. It is more general (but more complex).
