[GE users] OMP_NUM_THREADS in openmpi jobs

icaci hristo at mc.phys.uni-sofia.bg
Fri Mar 19 10:33:38 GMT 2010


Hi,

On 19.03.2010, at 12:08, buudo wrote:

> Hi everybody,
> i've got trouble with an openmpi code which is using MKL_NUM_THREADS (similar to OMP_NUM_THREADS) to mix openmp and openmpi to a prallel job. We have 12-core machines (12 slots).  I want to use e.g. 48 slots in all, the 12 slots of a machine for the openmp threads on 4 machines. The pe environment has allocation rule 12 as I read in http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=225043
> here is the script file:
> ---------------------------------------------
> #!/bin/sh
> #$ -S /bin/bash
> #$ -N Cu
> #$ -cwd
> #$ -pe openmp_12 48
> #
> #$ -l vf=.3G
> export OMP_NUM_THREADS=12
> export MKL_NUM_THREADS=12
> #$ -l h_rt=80:30:00
> 
> # Initialise environment module
> . /usr/local/scripts/profile_motd.no
> . /etc/profile.d/modules.sh
> module purge
> module load ompi-ifort-11.1
> #$ -m bea
> #$ -cwd
> #$ -o /t2work/username/sge_log/$JOB_ID.out
> #$ -e /t2work/username/sge_log/$JOB_ID.err
> #$ -R y
> #$ -m n
> mpirun --mca btl openib,self -np $NSLOTS /t2work/username/bin/skkr.run >./out
> --------------------------------------------
> Unfortunately the job starts with 48 slots on 4 nodes as expected, but overloads the nodes with 12x12 threads. Tis is seen in a output log file (12x48=576 threads) an in the qstat by enormous load, after a while the job crashes. If I use adifferent allocation rule e.g.2 and OMP_NUM_THREADS=2, we can survive, but the problem still exists.
> Does anyone can give me an advice ?
> Regards
> Udo
> 

If I get you right, you would like to run an MPI job with 4 MPI tasks, one task per node, 12 threads per task. As I can deduce from the mpirun invocation line above you use OpenMPI so you can pass the --byhost option and divide the number of tasks by the value of OMP_NUM_THREADS:

mpirun --byhost -np $(($NSLOTS/$OMP_NUM_THREADS)) ....

The --byhost option instructs OpenMPI to distribute tasks on a round-robin node basis. With 48 slots allocated and OMP_NUM_THREADS set to 12 you get 4 MPI tasks, one per each cluster node. With OMP_NUM_THREADS set to 6 you get 8 MPI tasks, two per node, distributed like that:

node 1: task 1, task 5
node 2: task 2, task 6
node 3: task 3, task 7
node 4: task 4, task 8

Hope that helps,

Hristo

> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=249640
> 
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

--
Dr Hristo Iliev
Monte Carlo research group
Faculty of Physics, University of Sofia
5 James Bourchier blvd, 1164 Sofia, Bulgaria
http://cluster.phys.uni-sofia.bg/hristo/

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=249646

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list