Rmpi
From FarmShare
(Difference between revisions)
m (→running) |
m (mktmp and awk were on the same line. Added a newline) |
||
Line 17: | Line 17: | ||
#$ -pe orte 32 | #$ -pe orte 32 | ||
- | tmphosts=`mktemp` awk '{ for (i=0; i < $2; ++i) { print $1} }' $PE_HOSTFILE > $tmphosts | + | tmphosts=`mktemp` |
+ | awk '{ for (i=0; i < $2; ++i) { print $1} }' $PE_HOSTFILE > $tmphosts | ||
echo "Got $NSLOTS slots" | echo "Got $NSLOTS slots" | ||
Line 102: | Line 103: | ||
== running == | == running == | ||
- | If you look at the first few lines of rmpi.submit you will see -pe orte 32. This indicates to grid engine to allocate 32 slots, across any hosts. | + | If you look at the first few lines of rmpi.submit you will see -pe orte 32. This indicates to grid engine to allocate 32 slots, across any hosts. |
Lets submit the job: | Lets submit the job: |
Revision as of 18:23, 15 February 2012
Rmpi
Rmpi is R with mpi support. On the barley this means OpenMPI. Rmpi can be installed from CRAN packages if you wish to have the latest version. As a convenience Rmpi_0.5-9 is installed on the barley in /mnt/glusterfs/apps/R. The setup section following this one will describe how to setup Rmpi so that you may submit jobs to the barley cluster.
setup
grid engine submit script (rmpi.submit)
# #$ -cwd #$ -j y #$ -S /bin/bash #$ -N rmpi32 #$ -pe orte 32 tmphosts=`mktemp` awk '{ for (i=0; i < $2; ++i) { print $1} }' $PE_HOSTFILE > $tmphosts echo "Got $NSLOTS slots" echo "jobid $JOB_ID" mpirun -np $NSLOTS -machinefile $tmphosts R --no-save -q < Rmpitest.R
Save following as .Rprofile. This should be in the current directory from where you submit your Rmpi jobs
# This R profile can be used when a cluster does not allow spawning or a job # scheduler is required to launch any parallel jobs. Saving this file as # .Rprofile in the working directory or root directory. For unix platform, run # mpirexec -n [cpu numbers] R --no-save -q # For windows platform with mpich2, use mpiexec wrapper and specify a working # directory where .Rprofile is inside. # Cannot be used as Rprofile.site because it will not work # Following system libraries are not loaded automatically. So manual loads are # needed. .libPaths(c("/mnt/glusterfs/apps/R", "/usr/lib/R/library")) library(utils) library(stats) library(datasets) library(grDevices) library(graphics) library(methods) if (!invisible(library(Rmpi,logical.return = TRUE))){ warning("Rmpi cannot be loaded") q(save = "no") } options(error=quote(assign(".mpi.err", FALSE, env = .GlobalEnv))) if (mpi.comm.size(0) > 1) invisible(mpi.comm.dup(0,1)) if (mpi.comm.rank(0) >0){ #sys.load.image(".RData",TRUE) options(echo=FALSE) .comm <- 1 mpi.barrier(0) repeat try(eval(mpi.bcast.cmd(rank=0,comm=.comm)),TRUE) #try(eval(mpi.bcast.cmd(rank=0,comm=.comm),env=sys.parent()),TRUE) #mpi.barrier(.comm) if (is.loaded("mpi_comm_disconnect")) mpi.comm.disconnect(.comm) else mpi.comm.free(.comm) mpi.quit() } if (mpi.comm.rank(0)==0) { #options(echo=TRUE) mpi.barrier(0) if(mpi.comm.size(0) > 1) slave.hostinfo(1) } .Last <- function(){ if (is.loaded("mpi_initialize")){ if (mpi.comm.size(1) > 1){ print("Please use mpi.close.Rslaves() to close slaves") mpi.close.Rslaves(comm=1) } } print("Please use mpi.quit() to quit R") mpi.quit() }
Here is a very short R program. Save this as Rmpitest.R
# Tell all slaves to return a message identifying themselves mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size())) # Tell all slaves to close down, and exit the program mpi.close.Rslaves() mpi.quit()
running
If you look at the first few lines of rmpi.submit you will see -pe orte 32. This indicates to grid engine to allocate 32 slots, across any hosts.
Lets submit the job:
bishopj@corn14:/mnt/glusterfs/bishopj$ qsub rmpi.submit Your job 81714 ("rmpi32") has been submitted bishopj@corn14:/mnt/glusterfs/bishopj$ cat rmpi32.o81714 Got 32 slots jobid 81714 master (rank 0 , comm 1) of size 32 is running on: barley01 slave1 (rank 1 , comm 1) of size 32 is running on: barley01 slave2 (rank 2 , comm 1) of size 32 is running on: barley04 slave3 (rank 3 , comm 1) of size 32 is running on: barley04 ... ... ... slave30 (rank 30, comm 1) of size 32 is running on: barley11 slave31 (rank 31, comm 1) of size 32 is running on: barley20 > > # Tell all slaves to return a message identifying themselves > mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size())) $slave1 [1] "I am 1 of 32" $slave2 [1] "I am 2 of 32" $slave3 [1] "I am 3 of 32" $slave4 [1] "I am 4 of 32" $slave5 [1] "I am 5 of 32" $slave6 [1] "I am 6 of 32" $slave7 [1] "I am 7 of 32" $slave8 [1] "I am 8 of 32" $slave9 [1] "I am 9 of 32" $slave10 [1] "I am 10 of 32" $slave11 [1] "I am 11 of 32" $slave12 [1] "I am 12 of 32" $slave13 [1] "I am 13 of 32" $slave14 [1] "I am 14 of 32" $slave15 [1] "I am 15 of 32" $slave16 [1] "I am 16 of 32" $slave17 [1] "I am 17 of 32" $slave18 [1] "I am 18 of 32" $slave19 [1] "I am 19 of 32" $slave20 [1] "I am 20 of 32" $slave21 [1] "I am 21 of 32" $slave22 [1] "I am 22 of 32" $slave23 [1] "I am 23 of 32" $slave24 [1] "I am 24 of 32" $slave25 [1] "I am 25 of 32" $slave26 [1] "I am 26 of 32" $slave27 [1] "I am 27 of 32" $slave28 [1] "I am 28 of 32" $slave29 [1] "I am 29 of 32" $slave30 [1] "I am 30 of 32" $slave31 [1] "I am 31 of 32" > > # Tell all slaves to close down, and exit the program > mpi.close.Rslaves() [1] 1 > mpi.quit()