雪群结合多核(Combining Multicore with Snow Cluster)

相当新的并行R.快速的问题。我有一个算法是计算密集型的。幸运的是它可以很容易地被分解成片，以利用multicore或snow 。如果它被认为是罚款的做法是使用我想知道的是multicore会同snow ？

我想这样做是分裂我的负荷在一个群集，每台机器多台机器上运行。我想利用机器上的所有核心。对于这种类型的处理，是合理的雪与混合multicore ？

我已经采用的方法通过lockedoff上面所建议的，也就是使用并行包在具有多个内核的多个机器分发的尴尬的并行工作量。首先工作量分布在所有的机器，然后每台机器的工作量分配它的所有核心上。这种方法的缺点是，没有负载机器之间的平衡（至少我不知道如何）。

所有装载R代码里面应该是相同的，并在所有机器上（SVN）相同的位置。因为初始化群集需要相当长的时间，下面的代码可以通过重用所创建的簇被改善。

foo <- function(workload, otherArgumentsForFoo) {
    source("/home/user/workspace/mycode.R")
    ...
}

distributedFooOnCores <- function(workload) {
    # Somehow assign a batch number to every record
    workload$ParBatchNumber = NA
    # Split the assigned workload into batches according to DistrParNumber
    batches = by(workload, workload$ParBatchNumber, function(x) x)

    # Create a cluster with workers on all machines 
    library("parallel")
    cluster = makeCluster(detectCores(), outfile="distributedFooOnCores.log")
    batches = parLapply(cluster, batches, foo, otherArgumentsForFoo)
    stopCluster(cluster)

    # Merge the resulting batches
    results = someEmptyDataframe
    p = 1;
    for(i in 1:length(batches)){
        results[p:(p + nrow(batches[[i]]) - 1), ] = batches[[i]]
        p = p + nrow(batches[[i]])      
    }

    # Clean up
    workload$ParBatchNumber = NULL
    return(invisible(results))
}

distributedFooOnMachines <- function(workload) {
    # Somehow assign a batch number to every record
    workload$DistrBatchNumber = NA
    # Split the assigned activity into batches according to DistrBatchNumber
    batches = by(workload, workload$DistrBatchNumber, function(x) x)

    # Create a cluster with workers on all machines 
    library("parallel")
    # If makeCluster hangs, please make sure passwordless ssh is configured on all machines
    cluster = makeCluster(c("machine1", "etc"), master="ub2", user="", outfile="distributedFooOnMachines.log")
    batches = parLapply(cluster, batches, foo, otherArgumentsForFoo)
    stopCluster(cluster)

    # Merge the resulting batches
    results = someEmptyDataframe
    p = 1;
    for(i in 1:length(batches)){
        results[p:(p + nrow(batches[[i]]) - 1), ] = batches[[i]]
        p = p + nrow(batches[[i]])      
    }

    # Clean up
    workload$DistrBatchNumber = NULL
    return(invisible(results))
}

我感兴趣的是如何上面的方法可以改善。