Can't replace doMPI with doFuture

[damonbayer]

Hello,

I am trying to migrate some code from using doMPI to using doFuture.
From doFuture's README, this seems like it might be straightforward: doFuture replaces existing doNnn packages.

Following from the example there, I translate my doMPI code:

library(Rmpi)
library(doMPI)
library(foreach)

cl <- doMPI::startMPIcluster(mpi.comm.size(0) - 1)
registerDoMPI(cl)

# Code here

to doFuture as:

library(parallelly)
library(Rmpi)
library(doFuture)
registerDoFuture()
cl <- parallel::makeCluster(mpi.comm.size(0) - 1, type = "MPI")
plan(cluster, workers = cl)

# Code here

The doMPI code works perfectly, but the doFuture code will not run and produces these errors:

All nodes which are allocated for this job are already filled.

Then it repeats

Error in Rmpi::mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count,  : 
  MPI_ERR_SPAWN: could not spawn processes
Calls: <Anonymous> -> <Anonymous> -> <Anonymous>
Execution halted

48 times. Note that this is running on 64 CPU's across 4 nodes, so I'm not sure why there are 48 repetitions of this error.

Then this error, which seems to know about the 64:

There are not enough slots available in the system to satisfy the 63
slots that were requested by the application:

  /usr/local/apps/R/4.1/4.1.0/lib64/R/bin/Rscript

Either request fewer slots for your application, or make more slots
available for use.

I'm not sure if this is a bug, or if I am doing something wrong. Any help would be appreciated.

Thanks,

Damon

[HenrikBengtsson]

Thanks for reporting. I've found MPI to be very picky and sensitive to the local setup. It's not unlikely that doMPI got more love than the MPI backend from parallel w/ Rmpi over the years, and therefore works better in more cases.

Having said that, let's leave out the future framework for now, and see if even the following works for you:

library(parallel)
cl <- parallel::makeCluster(2L, type="MPI")

info <- parLapply(cl, X=seq_along(cl), fun=function(i) {
  data.frame(i=i, pid=Sys.getpid())
})
info <- do.call(rbind, info)
print(info)

Also, what's your sessionInfo() after loading all packages of interest.

[damonbayer]

This appears to produce the same error. I have this note from the documentation from my HPC admins:

The higher level snow MPI cluster interface is currently not supported. However, the doMPI parallel backend for foreach is supported.

Is the "higher level snow MPI cluster interface" related to parallel::makeCluster(type="MPI")?

[HenrikBengtsson]

This appears to produce the same error. ...

Okay, so this is beyond the future framework, or more specifically, nothing that plan(cluster, workers = cl) can solve because internally that rely on the same infrastructure as parallel::parLapply().

Is the "higher level snow MPI cluster interface" related to parallel::makeCluster(type="MPI")?

Yes, I strongly believe so. The parallel package was extracted more or less as-is from the now semi-deprecated snow package. The term "snow" is also loosely used for all types of clusters that the parallel::makeCluster() can create.

Now, the above creates a cluster of type 'MPIcluster':

> cl <- parallel::makeCluster(2L, type="MPI")
> class(cl)
[1] "spawnedMPIcluster" "MPIcluster"        "cluster" 

One could imagine implementing a better alternative borrowing from the internals of doMPI, since that works better. Such a solution would be independent of the future framework. It could for instance be implemented in the parallel package or the parallelly package.