R function scope and parallelism

问题:

Consider the following function definitions

library(doParallel)
f_print <- function(x)
{
  print(x)
}
f_foreach <- function(l)
{
  foreach (i=l) %do%
  {
    f_print(i)
  }
}

f_foreach_parallel <- function(l)
{
  doParallel::registerDoParallel(1)
  foreach (i=l) %dopar%
  {
    f_print(i)
  }
}

Function use :

> f_foreach(c(1,2))
[1] 1
[1] 2
[[1]]
[1] 1

[[2]]
[1] 2

> f_foreach_parallel(c(1,2))
 Show Traceback

 Rerun with Debug
 Error in { : 
  task 1 failed - "impossible de trouver la fonction "f_print"" 
  [Error: could not find function f_print]
> 

Can you help explain why the f_print() is not visible when parallelism is involved in foreach ? How can we use f_print() in this paralleled foreach ?Any documentations related to this point ?

回答1:

In addition to what has already been said in the comments of the top post, especially the one on specifying .export, when using the doFuture package your code will indeed work regardless of parallel backend, operating system, and .export. Here's an adapted version of your setup:

f_print <- function(x) {
  print(x)
}

f_foreach <- function(l) {
  foreach(i=l) %do% {
    f_print(i)
  }
}

f_foreach_dopar <- function(l) {
  foreach(i=l) %dopar% {
    f_print(i)
  }
}

Instead of doing:

library("doParallel")

## Setup PSOCK workers (just as on Windows)
workers <- parallel::makeCluster(1L, outfile = "")
registerDoParallel(workers)

f_foreach_dopar(c(1,2))
## Error in { : task 1 failed - "could not find function "f_print""

you can do:

library("doFuture")
registerDoFuture()

## As above
workers <- parallel::makeCluster(1L, outfile = "")
plan(cluster, workers = workers)

f_foreach_dopar(c(1,2))
## [1] 1
## [1] 2
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 2

The reason why this works is that doFuture does a more thorough search to identify global variables (here f_print()).

PS. The reason for outfile = "" is so that stdout/stderr output (e.g. as from print()) is actually displayed. Redirecting stdout/stderr in parallel processing, which I don't recommend, is a whole different discussion, but I'll assume you used print() just for your example.