[GE users] Anybody able to get SGE MPICH2 Tight Integration via SSH working?

Jonathan Schreiter jonathanschreiter at yahoo.com
Thu Jan 5 01:09:08 GMT 2006


    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hi Reuti - thanks for the helpful reply!

I've left all the files from your howto alone and they
appear to be working correctly.  But now it looks like
it's a specific qrsh problem (timeout) trying to
remotely start smpd on the exec hosts.

I setup a user with a passphraseless ssh keypair
#ssh-keygen -d stored in ~/.ssh/id_dsa (no password)
#cat id_dsa.pub >> authorized_keys2 (for each exehost
- but it's a shared ~ for each user anyways)

#ssh <exehostname> hostname 
#ssh <exehostname>
both work fine (without the need for a passphrase)

My Cluster configuration is (from the ssh howto):
qlogin daemon:  /usr/bin/sshd -i
qlogin command:
/usr/local/bin/n1ge6/mpi/qlogin-wrapper (this file
matches the howto doc)
rsh daemon:     /usr/sbin/sshd -i
rsh command:    /usr/bin/ssh
rlogin daemon:  /usr/sbin/sshd -i
rlogin command: /usr/bin/ssh

If I try to run (as the same user):
#qrsh -verbose <exehostname> hostname
waiting for interactive job to be scheduled ...timeout
(5 s) expired while waiting on socket fd 5

I think this is the root cause of the problems.  Below
are the errors if I try to schedule a simple cpi job
via QSUB.

This is where the qsub mpi job seems to be failing
cpi_mpi_2.sh is from the template:
# ---------------------------
# our name
# $ -N MPI_Job
#
# pe request
# $ -pe mpi 2
#
# MPIR_HOME from submitting environment
# $ -v MPIR_HOME
# ---------------------------

#
# needs in
#   $NSLOTS
#       the number of tasks to be used
#   $TMPDIR/machines
#       a valid machine file to be passed to mpirun

echo "Got $NSLOTS slots."

$MPIR_HOME/bin/mpiexec -np $NSLOTS -machinefile
$MPIR_HOME/machines -port 6502 $MPIR_HOME/examples/cpi
---

Running
#qsub -pe <mympienvironment> 2 -v
MPIR_HOME=/usr/local/bin/n1ge6/mpich2_smpd/
cpi_mpi_2.sh

cpi_mpi_2.sh.po<JOBID>:
-catch_rsh
/var/tmp/sgespool/alpha/active_jobs/122.1/pe_hostfile
/usr/local/bin/n1ge6/mpich2_smpd
alpha
alpha
startmpich2.sh: check for smpd daemons (1 of 10)
startmpich2.sh: missing smpd on alpha
/usr/local/bin/n1ge6/bin/lx24-x86/qrsh -inherit alpha
/usr/local/bin/n1ge6/mpich2_smpd/bin/smpd -port 20122
-d 0
startmpich2.sh: check for smpd daemons (2 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (3 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (4 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (5 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (6 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (7 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (8 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (9 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: check for smpd daemons (10 of 10)
startmpich2.sh: missing smpd on alpha
startmpich2.sh: got only 0 of 1 nodes, aborting
-catch_rsh /usr/local/bin/n1ge6/mpich2_smpd

And cpi_mpi_2.sh.pe<JOBID>
error: executing task of job 122 failed:

EMAIL on <exechost> (alpha):
Job 122 caused action: Queue
"all.q at alpha.mydomain.com" set to ERROR
 User        = griduser1
 Queue       = all.q at alpha.mydomain.com
 Host        = alpha.mydomain.com
 Start Time  = <unknown>
 End Time    = <unknown>
failed in pestart:01/04/2006 17:52:02 [504:3551]:
exit_status of pe_start = 1
Shepherd trace:
01/04/2006 17:51:11 [504:3551]: shepherd called with
uid = 0, euid = 504
01/04/2006 17:51:11 [504:3551]: starting up 6.0u7
01/04/2006 17:51:11 [504:3551]: setpgid(3551, 3551)
returned 0
01/04/2006 17:51:11 [504:3551]: no prolog script to
start
01/04/2006 17:51:11 [504:3551]:
/usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh
-catch_rsh $pe_hostfile
/usr/local/bin/n1ge6/mpich2_smpd
01/04/2006 17:51:11 [504:3551]:
/usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh
-catch_rsh
/var/tmp/sgespool/alpha/active_jobs/122.1/pe_hostfile
/usr/local/bin/n1ge6/mpich2_smpd
01/04/2006 17:51:11 [504:3552]: pid=3552 pgrp=3552
sid=3552 old pgrp=3551 getlogin()=<no login set>
01/04/2006 17:51:11 [504:3552]: reading passwd
information for user 'griduser1'
01/04/2006 17:51:11 [504:3551]: forked "pe_start" with
pid 3552
01/04/2006 17:51:11 [504:3551]: using signal delivery
delay of 120 seconds
01/04/2006 17:51:11 [504:3551]: child: pe_start - pid:
3552
01/04/2006 17:51:11 [504:3552]: setting limits
01/04/2006 17:51:11 [504:3552]: setting environment
01/04/2006 17:51:11 [504:3552]: Initializing error
file
01/04/2006 17:51:11 [504:3552]: switching to
intermediate/target user
01/04/2006 17:51:11 [503:3552]: closing all
filedescriptors
01/04/2006 17:51:11 [503:3552]: further messages are
in "error" and "trace"
01/04/2006 17:51:11 [503:3552]: using "/bin/bash" as
shell of user "griduser1"
01/04/2006 17:51:11 [503:3552]: now running with
uid=503, euid=503
01/04/2006 17:51:11 [503:3552]:
execvp(/usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh,
"/usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh"
"-catch_rsh"
"/var/tmp/sgespool/alpha/active_jobs/122.1/pe_hostfile"
"/usr/local/bin/n1ge6/mpich2_smpd")
01/04/2006 17:52:02 [504:3551]: wait3 returned 3552
(status: 256; WIFSIGNALED: 0,  WIFEXITED: 1,
WEXITSTATUS: 1)
01/04/2006 17:52:02 [504:3551]: pe_start exited with
exit status 1
01/04/2006 17:52:02 [504:3551]: reaped "pe_start" with
pid 3552
01/04/2006 17:52:02 [504:3551]: pe_start exited not
due to signal
01/04/2006 17:52:02 [504:3551]: pe_start exited with
status 1
01/04/2006 17:52:02 [504:3551]: exit_status of
pe_start = 1
01/04/2006 17:52:02 [504:3551]:
/usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh
-catch_rsh /usr/local/bin/n1ge6/mpich2_smpd
01/04/2006 17:52:02 [504:3551]:
/usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh
-catch_rsh /usr/local/bin/n1ge6/mpich2_smpd
01/04/2006 17:52:02 [504:3551]: forked "pe_stop" with
pid 3670
01/04/2006 17:52:02 [504:3551]: using signal delivery
delay of 120 seconds
01/04/2006 17:52:02 [504:3551]: child: pe_stop - pid:
3670
01/04/2006 17:52:02 [504:3670]: pid=3670 pgrp=3670
sid=3670 old pgrp=3551 getlogin()=<no login set>
01/04/2006 17:52:02 [504:3670]: reading passwd
information for user 'griduser1'
01/04/2006 17:52:02 [504:3670]: setting limits
01/04/2006 17:52:02 [504:3670]: setting environment
01/04/2006 17:52:02 [504:3670]: Initializing error
file
01/04/2006 17:52:02 [504:3670]: switching to
intermediate/target user
01/04/2006 17:52:02 [503:3670]: closing all
filedescriptors
01/04/2006 17:52:02 [503:3670]: further messages are
in "error" and "trace"
01/04/2006 17:52:02 [503:3670]: using "/bin/bash" as
shell of user "griduser1"
01/04/2006 17:52:02 [503:3670]: now running with
uid=503, euid=503
01/04/2006 17:52:02 [503:3670]:
execvp(/usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh,
"/usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh"
"-catch_rsh" "/usr/local/bin/n1ge6/mpich2_smpd")
01/04/2006 17:52:02 [504:3551]: wait3 returned 3670
(status: 0; WIFSIGNALED: 0,
 WIFEXITED: 1, WEXITSTATUS: 0)
01/04/2006 17:52:02 [504:3551]: pe_stop exited with
exit status 0
01/04/2006 17:52:02 [504:3551]: reaped "pe_stop" with
pid 3670
01/04/2006 17:52:02 [504:3551]: pe_stop exited not due
to signal
01/04/2006 17:52:02 [504:3551]: pe_stop exited with
status 0
01/04/2006 17:52:02 [504:3551]: no tasker to notify
01/04/2006 17:52:02 [504:3551]: no epilog script to
start

Shepherd error:
01/04/2006 17:52:02 [504:3551]: exit_status of
pe_start = 1

Shepherd pe_hostfile:
alpha.mydomain.com 2 all.q at alpha.mydomain.com <NULL>


Any thoughts on what I'm doing wrong?  

Many thanks!
Jonathan


--- Reuti <reuti at staff.uni-marburg.de> wrote:

> Hi Jonathan,
> 
> first some remarks about the security. The easiest
> way would be to  
> install two network cards in the headnode of the
> cluster, so that the  
> users can connect to it, and use the second one to
> connect to all the  
> nodes from the master. So they are not connected to
> the outside  
> world, and the only break-in attempts could be made
> from the head  
> node, on which the granted users have access to. I
> see the situation,  
> that by using some desktop PCs to work on during the
> night, this  
> might not be possible to set up, unless also the
> desktop PCs would  
> have two network cards.
> 
> But in both cases, you don't need a running rshd or
> sshd on the nodes  
> at all, as SGE is using it's on rshd to achieve the
> Tight  
> Integration. This special rshd will only be started
> to allow access  
> on a chosen port. If you still see the need for
> using ssh, you can  
> also do this as you already found in the appropriate
> Howto. I'm not  
> sure, what you mean with "passphraseless keys".
> Usually you generate  
> the keypair with ssh-keygen for each user and then
> put the public key  
> into their .ssh/authorized_keys (and maybe adjust
> the .ssh/ 
> known_hosts file).
> 
> If you go for ssh, then it shouldn't be necessary to
> change any of  
> the start-scripts for the parallel libs support at
> all. Just edit the  
> entries in the SGE config as mentioned in the ssh
> Howto. The idea  
> behind the Tight Integration in SGE is:
> 
> - the start-script for the PE will create a link
> called "rsh" in the  
> $TMPDIR to the rsh-wrapper
> 
> - this link will be found first by a call to "rsh"
> of the  
> application, as $TMPDIR is the first in the
> generated $PATH
> 
> - if the application has a compiled-in "ssh", you
> could change RSHCMD  
> to be "ssh", so that also by a ssh the rsh-wrapper
> will be found (the  
> link will simply be named ssh, although it still
> points to the rsh- 
> wrapper - just a convenience)
> 
> - the rsh-wrapper will call SGE's qrsh, which will
> in the end start a  
> private rshd (or if configured sshd) on the slave
> nodes for this job  
> on a dedicated port and start the local rsh/ssh with
> a command
> 
> HTH - Reuti
> 
> 
> Am 04.01.2006 um 18:27 schrieb Jonathan Schreiter:
> 
> > Hello all,
> > I'm new to SGE, and trying to enable tight
> integration
> > with mpich2 and ssh (SGE 6.07, mpich2 1.0.3, FC4
> linux
> > 2.6 kernel with latest ssh).  I found the two
> howto's
> > on the project site re ssh integegration and
> > integration with mpich2 via rsh.  The rsh security
> > concerns are the primary reason I'd like to use
> ssh -
> > (specifically ssh based passphraseless keys on a
> per
> > user basis which is a bit better) - also because
> of
> > the requirement to disable the firewall for mpich2
> to
> > work with dynamic port assignments (even if one
> > specifies the primary listen to port).
> >
> > If I use the original scripts included in
> > $SGE_ROOT/mpi and have smpd started on the
> execution
> > hosts, I am able to sucessfully submit and execute
> mpi
> > jobs via a PE mpich2 environment on SGE.  I can
> also
> > start the smpd process on the exe hosts via a
> submit
> > job using ssh.  However, I do not know how one
> could
> > ever implement SGE w/o tight integration this way
> with
> > failed scripts / memory leaks / limbo processes,
> etc.
> >
> >
> > So I've been following the section "Tight
> Integration
> > of the daemon-based smpd startup method" closely.
> > Looking at the start_mpich2.c file, there doesn't
> > appear to be any rsh specific methods that need
> > changing (just a fork()).  In startmpi2.sh the
> area
> > where I think needs modification is:
> >
> > rshcmd=rsh
> > to something like rshcmd="ssh -i <~/.ssh/user's
> > passphraseless key>"
> >
> > I have the $SGE_ROOT/mpich2_smpd and home
> directories
> > shared on each execution host (and master/submit
> > hosts).
> >
> > If I try to execute the line:
> > $SGE_ROOT/mpich2_smpd/bin/lx24-x86/start_mpich2 -n
> > <host> $MPICH2_ROOT/bin/smpd <port> from a bash
> shell
> > I recieve connection refused errors (naturally). 
> I'm
> > not 100% sure how the RSH wrapper script and the
> howto
> > on ssh integration work together to make this
> happen.
> >
> > I guess what I'm asking is if anyone was able to
> get
> > this working, and how, rather then reinvent the
> > wheel...or perhaps I'm just way off.  I've been
> > reading just about all the posts on this mailing
> list
> > and I haven't found anyone who's been sucessful
> (or at
> > least posted the solution).  It may n ot even be
> > possible given the differences between rsh and
> ssh.
> >
> > Any help would be greatly appreciated!
> >
> > Many thanks,
> > Jonathan

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list