[GE users] Anybody able to get SGE MPICH2 Tight Integration via SSH working?

Reuti reuti at staff.uni-marburg.de
Thu Jan 5 15:31:25 GMT 2006


Hi,

Am 05.01.2006 um 02:09 schrieb Jonathan Schreiter:

> Hi Reuti - thanks for the helpful reply!
>
> I've left all the files from your howto alone and they
> appear to be working correctly.  But now it looks like
> it's a specific qrsh problem (timeout) trying to
> remotely start smpd on the exec hosts.
>
> I setup a user with a passphraseless ssh keypair
> #ssh-keygen -d stored in ~/.ssh/id_dsa (no password)
> #cat id_dsa.pub >> authorized_keys2 (for each exehost

I think nowadays you can just put it all in authorized_keys (without  
the 2). And you need there only one line there with the public key,  
but you will need one line for each exechost in ~/.ssh/known_hosts to  
avoid any ssh request to add it or not while you log in the first  
time. This might be a difference depending on the invocation: for a  
qrsh from the command line, you need the entry for the head node, but  
later during the parallel job, the connection is made from the  
headnode of your parallel job - which is one of the exec nodes itself.


> - but it's a shared ~ for each user anyways)
>
> #ssh <exehostname> hostname
> #ssh <exehostname>
> both work fine (without the need for a passphrase)
>
> My Cluster configuration is (from the ssh howto):
> qlogin daemon:  /usr/bin/sshd -i
> qlogin command:
> /usr/local/bin/n1ge6/mpi/qlogin-wrapper (this file
> matches the howto doc)
> rsh daemon:     /usr/sbin/sshd -i
> rsh command:    /usr/bin/ssh
> rlogin daemon:  /usr/sbin/sshd -i
> rlogin command: /usr/bin/ssh
>
> If I try to run (as the same user):
> #qrsh -verbose <exehostname> hostname
> waiting for interactive job to be scheduled ...timeout
> (5 s) expired while waiting on socket fd 5
>

yes, this should work. I'm a little bit puzzled with your command  
syntax:

qrsh hostname

should work (without specifying any hostname), as SGE will select a  
machine for you.

Do you still have the firewall active on the nodes? - Reuti


> I think this is the root cause of the problems.  Below
> are the errors if I try to schedule a simple cpi job
> via QSUB.
>
> This is where the qsub mpi job seems to be failing
> cpi_mpi_2.sh is from the template:
> # ---------------------------
> # our name
> # $ -N MPI_Job
> #
> # pe request
> # $ -pe mpi 2
> #
> # MPIR_HOME from submitting environment
> # $ -v MPIR_HOME
> # ---------------------------
>
> #
> # needs in
> #   $NSLOTS
> #       the number of tasks to be used
> #   $TMPDIR/machines
> #       a valid machine file to be passed to mpirun
>
> echo "Got $NSLOTS slots."
>
> $MPIR_HOME/bin/mpiexec -np $NSLOTS -machinefile
> $MPIR_HOME/machines -port 6502 $MPIR_HOME/examples/cpi
> ---
>
> Running
> #qsub -pe <mympienvironment> 2 -v
> MPIR_HOME=/usr/local/bin/n1ge6/mpich2_smpd/
> cpi_mpi_2.sh
>
> cpi_mpi_2.sh.po<JOBID>:
> -catch_rsh
> /var/tmp/sgespool/alpha/active_jobs/122.1/pe_hostfile
> /usr/local/bin/n1ge6/mpich2_smpd
> alpha
> alpha
> startmpich2.sh: check for smpd daemons (1 of 10)
> startmpich2.sh: missing smpd on alpha
> /usr/local/bin/n1ge6/bin/lx24-x86/qrsh -inherit alpha
> /usr/local/bin/n1ge6/mpich2_smpd/bin/smpd -port 20122
> -d 0
> startmpich2.sh: check for smpd daemons (2 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (3 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (4 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (5 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (6 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (7 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (8 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (9 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: check for smpd daemons (10 of 10)
> startmpich2.sh: missing smpd on alpha
> startmpich2.sh: got only 0 of 1 nodes, aborting
> -catch_rsh /usr/local/bin/n1ge6/mpich2_smpd
>
> And cpi_mpi_2.sh.pe<JOBID>
> error: executing task of job 122 failed:
>
> EMAIL on <exechost> (alpha):
> Job 122 caused action: Queue
> "all.q at alpha.mydomain.com" set to ERROR
>  User        = griduser1
>  Queue       = all.q at alpha.mydomain.com
>  Host        = alpha.mydomain.com
>  Start Time  = <unknown>
>  End Time    = <unknown>
> failed in pestart:01/04/2006 17:52:02 [504:3551]:
> exit_status of pe_start = 1
> Shepherd trace:
> 01/04/2006 17:51:11 [504:3551]: shepherd called with
> uid = 0, euid = 504
> 01/04/2006 17:51:11 [504:3551]: starting up 6.0u7
> 01/04/2006 17:51:11 [504:3551]: setpgid(3551, 3551)
> returned 0
> 01/04/2006 17:51:11 [504:3551]: no prolog script to
> start
> 01/04/2006 17:51:11 [504:3551]:
> /usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh
> -catch_rsh $pe_hostfile
> /usr/local/bin/n1ge6/mpich2_smpd
> 01/04/2006 17:51:11 [504:3551]:
> /usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh
> -catch_rsh
> /var/tmp/sgespool/alpha/active_jobs/122.1/pe_hostfile
> /usr/local/bin/n1ge6/mpich2_smpd
> 01/04/2006 17:51:11 [504:3552]: pid=3552 pgrp=3552
> sid=3552 old pgrp=3551 getlogin()=<no login set>
> 01/04/2006 17:51:11 [504:3552]: reading passwd
> information for user 'griduser1'
> 01/04/2006 17:51:11 [504:3551]: forked "pe_start" with
> pid 3552
> 01/04/2006 17:51:11 [504:3551]: using signal delivery
> delay of 120 seconds
> 01/04/2006 17:51:11 [504:3551]: child: pe_start - pid:
> 3552
> 01/04/2006 17:51:11 [504:3552]: setting limits
> 01/04/2006 17:51:11 [504:3552]: setting environment
> 01/04/2006 17:51:11 [504:3552]: Initializing error
> file
> 01/04/2006 17:51:11 [504:3552]: switching to
> intermediate/target user
> 01/04/2006 17:51:11 [503:3552]: closing all
> filedescriptors
> 01/04/2006 17:51:11 [503:3552]: further messages are
> in "error" and "trace"
> 01/04/2006 17:51:11 [503:3552]: using "/bin/bash" as
> shell of user "griduser1"
> 01/04/2006 17:51:11 [503:3552]: now running with
> uid=503, euid=503
> 01/04/2006 17:51:11 [503:3552]:
> execvp(/usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh,
> "/usr/local/bin/n1ge6/mpich2_smpd/startmpich2.sh"
> "-catch_rsh"
> "/var/tmp/sgespool/alpha/active_jobs/122.1/pe_hostfile"
> "/usr/local/bin/n1ge6/mpich2_smpd")
> 01/04/2006 17:52:02 [504:3551]: wait3 returned 3552
> (status: 256; WIFSIGNALED: 0,  WIFEXITED: 1,
> WEXITSTATUS: 1)
> 01/04/2006 17:52:02 [504:3551]: pe_start exited with
> exit status 1
> 01/04/2006 17:52:02 [504:3551]: reaped "pe_start" with
> pid 3552
> 01/04/2006 17:52:02 [504:3551]: pe_start exited not
> due to signal
> 01/04/2006 17:52:02 [504:3551]: pe_start exited with
> status 1
> 01/04/2006 17:52:02 [504:3551]: exit_status of
> pe_start = 1
> 01/04/2006 17:52:02 [504:3551]:
> /usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh
> -catch_rsh /usr/local/bin/n1ge6/mpich2_smpd
> 01/04/2006 17:52:02 [504:3551]:
> /usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh
> -catch_rsh /usr/local/bin/n1ge6/mpich2_smpd
> 01/04/2006 17:52:02 [504:3551]: forked "pe_stop" with
> pid 3670
> 01/04/2006 17:52:02 [504:3551]: using signal delivery
> delay of 120 seconds
> 01/04/2006 17:52:02 [504:3551]: child: pe_stop - pid:
> 3670
> 01/04/2006 17:52:02 [504:3670]: pid=3670 pgrp=3670
> sid=3670 old pgrp=3551 getlogin()=<no login set>
> 01/04/2006 17:52:02 [504:3670]: reading passwd
> information for user 'griduser1'
> 01/04/2006 17:52:02 [504:3670]: setting limits
> 01/04/2006 17:52:02 [504:3670]: setting environment
> 01/04/2006 17:52:02 [504:3670]: Initializing error
> file
> 01/04/2006 17:52:02 [504:3670]: switching to
> intermediate/target user
> 01/04/2006 17:52:02 [503:3670]: closing all
> filedescriptors
> 01/04/2006 17:52:02 [503:3670]: further messages are
> in "error" and "trace"
> 01/04/2006 17:52:02 [503:3670]: using "/bin/bash" as
> shell of user "griduser1"
> 01/04/2006 17:52:02 [503:3670]: now running with
> uid=503, euid=503
> 01/04/2006 17:52:02 [503:3670]:
> execvp(/usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh,
> "/usr/local/bin/n1ge6/mpich2_smpd/stopmpich2.sh"
> "-catch_rsh" "/usr/local/bin/n1ge6/mpich2_smpd")
> 01/04/2006 17:52:02 [504:3551]: wait3 returned 3670
> (status: 0; WIFSIGNALED: 0,
>  WIFEXITED: 1, WEXITSTATUS: 0)
> 01/04/2006 17:52:02 [504:3551]: pe_stop exited with
> exit status 0
> 01/04/2006 17:52:02 [504:3551]: reaped "pe_stop" with
> pid 3670
> 01/04/2006 17:52:02 [504:3551]: pe_stop exited not due
> to signal
> 01/04/2006 17:52:02 [504:3551]: pe_stop exited with
> status 0
> 01/04/2006 17:52:02 [504:3551]: no tasker to notify
> 01/04/2006 17:52:02 [504:3551]: no epilog script to
> start
>
> Shepherd error:
> 01/04/2006 17:52:02 [504:3551]: exit_status of
> pe_start = 1
>
> Shepherd pe_hostfile:
> alpha.mydomain.com 2 all.q at alpha.mydomain.com <NULL>
>
>
> Any thoughts on what I'm doing wrong?
>
> Many thanks!
> Jonathan
>
>
> --- Reuti <reuti at staff.uni-marburg.de> wrote:
>
>> Hi Jonathan,
>>
>> first some remarks about the security. The easiest
>> way would be to
>> install two network cards in the headnode of the
>> cluster, so that the
>> users can connect to it, and use the second one to
>> connect to all the
>> nodes from the master. So they are not connected to
>> the outside
>> world, and the only break-in attempts could be made
>> from the head
>> node, on which the granted users have access to. I
>> see the situation,
>> that by using some desktop PCs to work on during the
>> night, this
>> might not be possible to set up, unless also the
>> desktop PCs would
>> have two network cards.
>>
>> But in both cases, you don't need a running rshd or
>> sshd on the nodes
>> at all, as SGE is using it's on rshd to achieve the
>> Tight
>> Integration. This special rshd will only be started
>> to allow access
>> on a chosen port. If you still see the need for
>> using ssh, you can
>> also do this as you already found in the appropriate
>> Howto. I'm not
>> sure, what you mean with "passphraseless keys".
>> Usually you generate
>> the keypair with ssh-keygen for each user and then
>> put the public key
>> into their .ssh/authorized_keys (and maybe adjust
>> the .ssh/
>> known_hosts file).
>>
>> If you go for ssh, then it shouldn't be necessary to
>> change any of
>> the start-scripts for the parallel libs support at
>> all. Just edit the
>> entries in the SGE config as mentioned in the ssh
>> Howto. The idea
>> behind the Tight Integration in SGE is:
>>
>> - the start-script for the PE will create a link
>> called "rsh" in the
>> $TMPDIR to the rsh-wrapper
>>
>> - this link will be found first by a call to "rsh"
>> of the
>> application, as $TMPDIR is the first in the
>> generated $PATH
>>
>> - if the application has a compiled-in "ssh", you
>> could change RSHCMD
>> to be "ssh", so that also by a ssh the rsh-wrapper
>> will be found (the
>> link will simply be named ssh, although it still
>> points to the rsh-
>> wrapper - just a convenience)
>>
>> - the rsh-wrapper will call SGE's qrsh, which will
>> in the end start a
>> private rshd (or if configured sshd) on the slave
>> nodes for this job
>> on a dedicated port and start the local rsh/ssh with
>> a command
>>
>> HTH - Reuti
>>
>>
>> Am 04.01.2006 um 18:27 schrieb Jonathan Schreiter:
>>
>>> Hello all,
>>> I'm new to SGE, and trying to enable tight
>> integration
>>> with mpich2 and ssh (SGE 6.07, mpich2 1.0.3, FC4
>> linux
>>> 2.6 kernel with latest ssh).  I found the two
>> howto's
>>> on the project site re ssh integegration and
>>> integration with mpich2 via rsh.  The rsh security
>>> concerns are the primary reason I'd like to use
>> ssh -
>>> (specifically ssh based passphraseless keys on a
>> per
>>> user basis which is a bit better) - also because
>> of
>>> the requirement to disable the firewall for mpich2
>> to
>>> work with dynamic port assignments (even if one
>>> specifies the primary listen to port).
>>>
>>> If I use the original scripts included in
>>> $SGE_ROOT/mpi and have smpd started on the
>> execution
>>> hosts, I am able to sucessfully submit and execute
>> mpi
>>> jobs via a PE mpich2 environment on SGE.  I can
>> also
>>> start the smpd process on the exe hosts via a
>> submit
>>> job using ssh.  However, I do not know how one
>> could
>>> ever implement SGE w/o tight integration this way
>> with
>>> failed scripts / memory leaks / limbo processes,
>> etc.
>>>
>>>
>>> So I've been following the section "Tight
>> Integration
>>> of the daemon-based smpd startup method" closely.
>>> Looking at the start_mpich2.c file, there doesn't
>>> appear to be any rsh specific methods that need
>>> changing (just a fork()).  In startmpi2.sh the
>> area
>>> where I think needs modification is:
>>>
>>> rshcmd=rsh
>>> to something like rshcmd="ssh -i <~/.ssh/user's
>>> passphraseless key>"
>>>
>>> I have the $SGE_ROOT/mpich2_smpd and home
>> directories
>>> shared on each execution host (and master/submit
>>> hosts).
>>>
>>> If I try to execute the line:
>>> $SGE_ROOT/mpich2_smpd/bin/lx24-x86/start_mpich2 -n
>>> <host> $MPICH2_ROOT/bin/smpd <port> from a bash
>> shell
>>> I recieve connection refused errors (naturally).
>> I'm
>>> not 100% sure how the RSH wrapper script and the
>> howto
>>> on ssh integration work together to make this
>> happen.
>>>
>>> I guess what I'm asking is if anyone was able to
>> get
>>> this working, and how, rather then reinvent the
>>> wheel...or perhaps I'm just way off.  I've been
>>> reading just about all the posts on this mailing
>> list
>>> and I haven't found anyone who's been sucessful
>> (or at
>>> least posted the solution).  It may n ot even be
>>> possible given the differences between rsh and
>> ssh.
>>>
>>> Any help would be greatly appreciated!
>>>
>>> Many thanks,
>>> Jonathan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list