From FarmShare

Revision as of 10:32, 12 October 2013

Abinit

Parallel Abinit is available on FarmShare. This install uses MPI and ACML. To use it you need to submit a parallel job to the barley cluster.

MPI example on barley cluster

abinittest/abinit-test/tutoparal/README_dfpt.txt

Second test : BaTiO3 slab (29 atoms), computation of the phonon frequencies at qpt 0.0 0.375 0.0

This test, with 29 atom, is quite slow, but scales very well.

There is one preparatory step, before running the DFPT calculation. The preparatory step can be run on 16 processors at most with the current input file. It might use more processors as well, with the kgb parallelism (but the input file has to be modified). On 8 processors, the preparatory step is about three hours. It generates well-converged wavefunctions. For a quick trial, simply set nstep 1 instead of nstep 50 , this will run in about 6 minutes.

The test case itself is an underconverged calculation of the response with respect to one perturbation (atomic displacement). It is underconverged because nstep has been set to 10, while more than 30 are needed. Moreover, obtaining the interatomic force constants would need computing many more perturbations than the present one. In any case, the present test case run in about 45 minutes on a 8 core machine. Since the number of k points to be kept for the present perturbation is is 8x8x1 with 4 symmetries, that is 16, and the number of bands is 120, the perfectly scalable part of the test case should have a maximum speed up of 1920.

From tests for the 8 core case, on a total of 20200 secs, there were 305 secs for vtorho3:synchro (sequential) and 260.460 for inwffil (sequential). The latter will not increase with a bigger value of nstep, and for more perturbations, while the former will increase proportionally.

Hence, in the present status, for 8 cores, the sequential part is about 3%, leading to a maximum speed-up with respect to sequential, of about 240. For a larger test case (bigger nstep, more perturbations), the maximum speed up might be twice bigger.

Preparatory step 1 (mpirun ...) abinit < tdfpt_03.files > tdfpt_03.log cp tdfpt_03.o_WFK tdfpt_04.i_WFK cp tdfpt_03.o_WFK tdfpt_04.i_WFQ

Test case, step 2 (DFPT calculation) (mpirun ...) abinit < tdfpt_04.files > tdfpt_04.log

This translates to this job submission script:

#!/bin/bash

#$ -cwd
#$ -S /bin/bash
#$ -N abinit
#$ -M bishopj@stanford.edu
#$ -m beas
#$ -R y
#$ -l mem_free=1G
##$ -l testq=1
#$ -pe orte 4

echo "Got $NSLOTS slots"
echo "jobid $JOB_ID"

tmphosts=`mktemp`
awk '{ for (i=0; i < $2; ++i) { print $1} }' $PE_HOSTFILE > $tmphosts


echo "pwd"
pwd

echo ""
echo "nslots: $NSLOTS"
echo ""

module load abinit acml
#openmpi

date
mpirun -np $NSLOTS -machinefile $tmphosts -x LD_LIBRARY_PATH /farmshare/software/free/abinit/7.4.2/bin/abinit < tdfpt_03.files > tdfpt_03.log
date

cp tdfpt_03.o_WFK tdfpt_04.i_WFK
cp tdfpt_03.o_WFK tdfpt_04.i_WFQ

date
mpirun -np $NSLOTS -machinefile $tmphosts -x LD_LIBRARY_PATH /farmshare/software/free/abinit/7.4.2/bin/abinit < tdfpt_04.files > tdfpt_04.log
date

Here is an example run:

$ module load abinit
$ mkdir abinittest
$ cd abinittest
$ cp -rp $ABINITHOME/share/abinit-test .
$ cd abinit-test/tutoparal/Input/

$ qsub abinit.submit 
Your job 1143544 ("abinit") has been submitted
bishopj@scorn:~/abinittest/abinit-test/tutoparal/Input$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
1143544 0.39219 abinit     bishopj      r     10/12/2013 10:25:56 raring.q@barley07.Stanford.EDU     4