[GE users] SGE and OpenMPI 1.3.2

doraz dobre.razvan at gmail.com
Thu Jan 21 09:00:46 GMT 2010


    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hello

As far as i know openmpi automatically detect that it is run onder the Sun Grid Engine. I think you should try to configure the parallel environment without the {start|stop}_proc_args scripts (those are used when running IntelMPI to run the mpiboot command automatically but..manually):

start_proc_args    TRUE
stop_proc_args     TRUE


On Wed, Jan 20, 2010 at 10:03 PM, jess <jac67 at georgetown.edu<mailto:jac67 at georgetown.edu>> wrote:
I have a 28-node cluster (Sun quad-core Opterons) with 8 cores per node
running RHEL 5.4 (64-bit), SGE 6.2u4 (64-bit), and OpenMPI 1.3.2
compiled with SEG support (--with-sge). The cluster has DDR Infiniband
running OFED 1.4.

I'm seeing errors with simple OpenMPI jobs (e.g., cpi and mpi_hello)
when I launch them using SGE and I specify more than 96 slots. If I
specify less than 96 slots, the job runs via SGE without error. Here is
the error that I see:

*******

[r3:22101] [[14835,0],0] ORTE_ERROR_LOG: The system limit on number of
pipes a process can open was reached in file base/iof_base_setup.c at
line 108
[r3:22101] [[14835,0],0] ORTE_ERROR_LOG: The system limit on number of
pipes a process can open was reached in file odls_default_module.c at
line 203
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orterun:sys-limit-pipe
from the file:
    help-orterun.txt
But I couldn't find any file matching that name.  Sorry!
--------------------------------------------------------------------------
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).
[r3:22101] mca_oob_tcp_accept: accept() failed: Too many open files (24).

*******

Here is the job submission file:

*******

#!/bin/sh
#
#$ -N mpi_cpi-gcc4
#
#$ -q standard.q
#
# pe request
#$ -pe mpi 128
#
#$ -j y
#
#$ -V
#
#$ -P hpc_admins-p
# ---------------------------
#
# needs in
#   $NSLOTS
#       the number of tasks to be used
#   $TMPDIR/machines
#       a valid machiche file to be passed to mpirun

echo "Got $NSLOTS slots."

mpiexec --mca btl openib,self -np $NSLOTS -machinefile $TMPDIR/machines
/home/jac67/tests/sge_tests/cpi-gcc4

*******

I can launch the same job outside of SGE from all of the nodes using up
to 4096 processes without error. Here the command that I run outside of
SGE, which always works:

mpiexec -machinefile ~/tests/mpi/nodes -np 2048 --mca btl openib,self
/home/jac67/tests/mpi/cpi-gcc4

I increased some of the system limits to see if that would help, but it
doesn't. Here are the limits:

*******

[jac67 at rush ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 266240
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

*******

Here are the settings for the parallel environment:

*******

[root at rush spool]# qconf -sp mpi
pe_name            mpi
slots              224
user_lists         NONE
xuser_lists        NONE
start_proc_args    /home/sge/mpi/startmpi.sh -catch_rsh $pe_hostfile
stop_proc_args     /home/sge/mpi/stopmpi.sh
allocation_rule    $round_robin
control_slaves     TRUE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary TRUE

*******

I've done a lot of searching on the the OpenMPI, SGE, and Internet for
this issue, and am not finding a solution. It seems to be an issue with
the way SGE launches the MPI job since the job always works outside of
SGE. Any pointers would be greatly appreciated.

Thanks.
Jess

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=239984

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].



--
Dobre Razvan



More information about the gridengine-users mailing list