How to kill a doMC worker when it's done?

Question

The documentation for doMC seems very sparse, listing only doMC-package and registerDoMC(). The problem I'm encountering is I'll spawn several workers via doMC/foreach, but then when the job is done they just sit there taking up memory. I can go and hunt their process IDs, but I often kill the master process by accident.

library(doMC)
library(foreach)

registerDoMC(32)

foreach(i=1:32) %dopar% foo()

##kill command here?

I've tried following with registerDoSEQ() but it doesn't seem to kill off the processes.

Answer 1

By using registerDoSEQ() you simply register the sequential worker, so all parallel workers should stop. This is not a complete solution, but it should work in some cases.

Answer 2

I never did find a suitable solution for doMC, so for a while I've been doing the following:

library(doParallel)
cl <- makePSOCKcluster(4) # number of cores to use
registerDoParallel(cl)

## computation

stopCluster(cl)

Works every time.

Answer 3

The doMC package is basically a wrapper around the mclapply function, and mclapply forks workers that should exit before it returns. It doesn't use persistent workers like the snow package or the snow-derived functions in the parallel package, so it doesn't need a function like stopCluster to shutdown the workers.

Do you see the same problem when using mclapply directly? Does it work any better when you call registerDoMC with a smaller value for cores?

Are you using doMC from a IDE such as RStudio or R.app on a Mac? If so, you might want try using R from a terminal to see if that makes a difference. There could be a problem calling fork in an IDE.