How to setup workers for parallel processing in R using snowfall and multiple Windows nodes?
<p>I’ve successfully used snowfall to setup a cluster on a single server with 16 processors.</p>

<pre><code>require(snowfall)
if (sfIsRunning() == TRUE) sfStop()

number.of.cpus &lt;- 15
sfInit(parallel = TRUE, cpus = number.of.cpus)
stopifnot( sfCpus() == number.of.cpus )
stopifnot( sfParallel() == TRUE )

# Print the hostname for each cluster member
sayhello &lt;- function()
{
    info &lt;- Sys.info()[c("nodename", "machine")]
    paste("Hello from", info[1], "with CPU type", info[2])
}
names &lt;- sfClusterCall(sayhello)
print(unlist(names))
</code></pre>

<p>Now, I am looking for complete instructions on <strong>how to move to a distributed model</strong>. I have 4 different Windows machines with a total of 16 cores that I would like to use for a 16 node cluster. So far, I understand that I could manually setup a SOCK connection or leverage MPI. While it appears possible, I haven’t found clear and complete directions as to how.</p>

<p><strong>The SOCK route</strong> appears to depend on code in a snowlib script. I can generate a stub from the master side with the following code:</p>

<pre><code>winOptions &lt;-
    list(host="172.01.01.03",
         rscript="C:/Program Files/R/R-2.7.1/bin/Rscript.exe",
         snowlib="C:/Rlibs")

cl &lt;- makeCluster(c(rep(list(winOptions), 2)), type = "SOCK", manual = T)
</code></pre>

<p>It yields the following:</p>

<pre><code>Manually start worker on 172.01.01.03 with
     "C:/Program Files/R/R-2.7.1/bin/Rscript.exe"
      C:/Rlibs/snow/RSOCKnode.R
      MASTER=Worker02 PORT=11204 OUT=/dev/null SNOWLIB=C:/Rlibs
</code></pre>

<p>It feels like a reasonable start. I found code for <code>RSOCKnode.R</code> on GitHub under the snow package:</p>

<pre><code>local({
    master &lt;- "localhost"
    port &lt;- ""
    snowlib &lt;- Sys.getenv("R_SNOW_LIB")
    outfile &lt;- Sys.getenv("R_SNOW_OUTFILE") ##**** defaults to ""; document

    args &lt;- commandArgs()
    pos &lt;- match("--args", args)
    args &lt;- args[-(1 : pos)]
    for (a in args) {
        pos &lt;- regexpr("=", a)
        name &lt;- substr(a, 1, pos - 1)
        value &lt;- substr(a,pos + 1, nchar(a))
        switch(name,
               MASTER = master &lt;- value,
               PORT = port &lt;- value,
               SNOWLIB = snowlib &lt;- value,
               OUT = outfile &lt;- value)
    }

    if (! (snowlib %in% .libPaths()))
        .libPaths(c(snowlib, .libPaths()))
    library(methods) ## because Rscript as of R 2.7.0 doesn't load methods
    library(snow)

    if (port == "") port &lt;- getClusterOption("port")

    sinkWorkerOutput(outfile)
    cat("starting worker for", paste(master, port, sep = ":"), "\n")
    slaveLoop(makeSOCKmaster(master, port))
})
</code></pre>

<p>It’s not clear how to actually start a SOCK listener on the workers, unless it is buried in <code>snow::recvData</code>.</p>

<p>Looking into <strong>the MPI route</strong>, as far as I can tell, Microsoft MPI version 7 is a starting point. However, I could not find a Windows alternative for sfCluster. I was able to start the MPI service, but it does not appear to listen on port 22 and no amount of bashing against it with <code>snowfall::makeCluster</code> has yielded a result. I’ve disabled the firewall and tried testing with makeCluster and directly connecting to the worker from the master with PuTTY.</p>

<hr>

<p>Is there a comprehensive, step-by-step guide to setting up a snowfall cluster on Windows workers that I’ve missed? I am fond of <code>snowfall::sfClusterApplyLB</code> and would like to continue using that, but if there is an easier solution, I’d be willing to change course. Looking into Rmpi and parallel, I found alternative solutions for the master side of the work, but still little to no specific detail on how to setup workers running Windows.</p>

<p>Due to the nature of the work environment, neither moving to AWS, nor Linux is an option.</p>

<p>Related questions without definitive answers for Windows worker nodes:</p>

<ul>
<li><a href="https://stackoverflow.com/q/22605238">How to set up cluster slave nodes (on Windows)</a></li>
<li><a href="https://stackoverflow.com/q/17547994">Parallel R on a Windows cluster</a></li>
<li><a href="https://stackoverflow.com/q/15039039">Create a cluster of co-workers' Windows 7 PCs for parallel processing in R?</a></li>
</ul>