[GE users] SGE and OpenMPI 1.3.2

jess jac67 at georgetown.edu
Wed Jan 20 20:03:59 GMT 2010


I have a 28-node cluster (Sun quad-core Opterons) with 8 cores per node 
running RHEL 5.4 (64-bit), SGE 6.2u4 (64-bit), and OpenMPI 1.3.2 
compiled with SEG support (--with-sge). The cluster has DDR Infiniband 
running OFED 1.4.

I'm seeing errors with simple OpenMPI jobs (e.g., cpi and mpi_hello) 
when I launch them using SGE and I specify more than 96 slots. If I  
specify less than 96 slots, the job runs via SGE without error. Here is 
the error that I see:

*******

[r3:22101] [[14835,0],0] ORTE_ERROR_LOG: The system limit on number of 
pipes a process can open was reached in file base/iof_base_setup.c at 
line 108
[r3:22101] [[14835,0],0] ORTE_ERROR_LOG: The system limit on number of 
pipes a process can open was reached in file odls_default_module.c at 
line 203
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
     orterun:sys-limit-pipe
from the file:
     help-orterun.txt
But I couldn't find any file matching that name.  Sorry!
--------------------------------------------------------------------------
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).

*******

Here is the job submission file:

*******

#!/bin/sh
#
#$ -N mpi_cpi-gcc4
#
#$ -q standard.q
#
# pe request
#$ -pe mpi 128
#
#$ -j y
#
#$ -V
#
#$ -P hpc_admins-p
# ---------------------------
#
# needs in
#   $NSLOTS
#       the number of tasks to be used
#   $TMPDIR/machines
#       a valid machiche file to be passed to mpirun

echo "Got $NSLOTS slots."

mpiexec --mca btl openib,self -np $NSLOTS -machinefile $TMPDIR/machines 
/home/jac67/tests/sge_tests/cpi-gcc4

*******

I can launch the same job outside of SGE from all of the nodes using up 
to 4096 processes without error. Here the command that I run outside of 
SGE, which always works:

mpiexec -machinefile ~/tests/mpi/nodes -np 2048 --mca btl openib,self 
/home/jac67/tests/mpi/cpi-gcc4

I increased some of the system limits to see if that would help, but it 
doesn't. Here are the limits:

*******

[jac67 at rush ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 266240
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

*******

Here are the settings for the parallel environment:

*******

[root at rush spool]# qconf -sp mpi
pe_name            mpi
slots              224
user_lists         NONE
xuser_lists        NONE
start_proc_args    /home/sge/mpi/startmpi.sh -catch_rsh $pe_hostfile
stop_proc_args     /home/sge/mpi/stopmpi.sh
allocation_rule    $round_robin
control_slaves     TRUE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary TRUE

*******

I've done a lot of searching on the the OpenMPI, SGE, and Internet for 
this issue, and am not finding a solution. It seems to be an issue with 
the way SGE launches the MPI job since the job always works outside of 
SGE. Any pointers would be greatly appreciated.

Thanks.
Jess

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=239984

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list