[GE users] MPICH-MX tight integration

Chris Rudge chris.rudge at astro.le.ac.uk
Mon Nov 5 10:11:00 GMT 2007

On Fri, 2007-11-02 at 15:00 -0400, Constantinos Evangelinos wrote:
> Anyway, two basic questions:
> a) Does anyone have a working PE configuration for a tight MPICH-MX 

Yes, I have this working using mpich-mx / mx-2g. The PE is defined as:
# qconf -sp mpich-mx
pe_name           mpich-mx
slots             256
user_lists        NONE
xuser_lists       NONE
start_proc_args   /usr/local/sge6.0/mpi/startmpi.sh  -catch_rsh
stop_proc_args    /usr/local/sge6.0/mpi/stopmpi.sh
allocation_rule   $fill_up
control_slaves    TRUE
job_is_first_task FALSE
urgency_slots     min

The start/stopmpi.sh scripts are the examples provided. I initially
tried using the versions in the Myrinet subdirectory which do
essentially the same thing but in a slightly more complicated way and it
didn't work reliably - processes weren't all killed properly. When I
changed to using the standard examples rather than the myrinet ones I
also stopped using a common TMP directory for the MPI jobs (the myrinet
README claims you need this but not the standard MPI readme).

The only component of the myrinet example I use is the sge_mpirun
wrapper script as it avoids the need for the user to specify either
'-np' or '-machinefile' arguments as part of their mpirun command - i.e.
	mpirun -np 16 -machinefile $machinefile <executable>
can be replaced with simply
	sge_mpirun <executable>

Note that you'll need to have sge_mpirun in the user's default PATH. The
myrinet README suggests a symbolic link from SGE's bin/$arch directory.
You'll also need to edit the script to explicitly set the path to the
mpirun command as the following command in the script won't work:

mpirun=$($qconf -sp $PE | grep "^start_proc_args" | awk '{ print $NF; }')

Note also that some of the supplied scripts use ksh which we didn't have
installed on all of our nodes. I opted to modify the scripts rather than
install ksh everywhere.

The important thing here is that it "catches" the rsh commands spawned
by the controlling mpirun process and replaces them with 'qrsh -inherit'
so that every MPI process is then under SGE's control. Our mpich-mx
installation is configured to use rsh and qrsh also uses rsh. If you've
set up qrsh to use ssh then you'll need to have ssh configured so that
no password is required going being compute nodes - the default
requirement is that you only need passwordless logins between the head
node and each compute node.

I'm also not certain what happens if mpich-mx is configured to use ssh.
I guess it may be that SGE won't "catch" the ssh at all so the tight
integration won't work.


Dr Chris Rudge
chris.rudge at astro.le.ac.uk

UKAFF Facility Manager & Dept. Research Computing Manager
Dept of Physics & Astronomy
University of Leicester

web.  www.ukaff.ac.uk
Tel.  +44 (0)116 2523331
Fax.  +44 (0)116 2231283
Mob.  +44 (0)794 1379420

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list