[GE users] LAM & SGE

Chris Duncan Christopher.Duncan at Sun.COM
Mon Jul 12 20:09:37 BST 2004


Alexandre Barras wrote:
> Hello,
> 
> I did not find on Internet any way to configure LAM within SGE without 
> changing the LAM boot schema and qrsh-lam.
> 
> So I wrote my own scripts and I suggest you my solution that is in fact 
> very similar to the MPI integration provided with SGE
> Thanks to them, you can submit LAM job to SGE with the command: qsub -pe 
> lam myscript

The qrsh-lam script is meant for what is often called a tight 
integration where the batch system has full control over the processes 
and also has accounting and resource usage info on the job.

When rsh or ssh is used for the job launch SGE is not fully aware of the 
processes since they will not be child processes of the shepherd procs.

For the PE below you change control_slaves to FALSE.

Attached is the latest sge-lam script which should work with SGE5.3p5 or 
later patch level and LAM 7.0.6 or later. I'll try to get a web page 
with it up onto the open source site. I have not had time to test it 
with SGE 6.0 but don't know of any reason it shouldnt work.

Still need to get this working with ssh to get to the remote nodes which 
should fix the qrsh problem Bogdan is referencing. the problem is a 
limitation where this scheme fails for parallel jobs with only 1 process
per host. If you have dual CPU boxes or fatter the problem should not 
occur. I'm having some trouble reproducing the problem on my small 
cluster so if you have a larger cluster of single procs and try this and 
it works let me know.

If you try this with SGE 6.0 and have problems or it works let me know.


> 
> For those interested:
> 
> 
> Add the PE:
>        % qconf -sp lam        pe_name lam
>        slots 999
>        user_lists NONE
>        xuser_lists NONE
>        start_proc_args /gridware/sge/mpi/startlam.sh
>        stop_proc_args /gridware/sge/mpi/stoplam.sh
>        allocation_rule $round_robin
>        control_slaves TRUE
>        job_is_first_task FALSE
>              
> startlam.sh
> ------------------------------------------------------------
> #!/bin/sh
> #
> #
> # preparation of the mpi/lam machine file
> #
> # usage: startlam.sh <pe_hostfile>
> #
> 
> PeHostfile2MachineFile()
> {
>   cat $1 | while read line; do
>          # echo $line
>          host=`echo $line|cut -f1 -d" "|cut -f1 -d"."`
>          nslots=`echo $line|cut -f2 -d" "`
>          echo $host cpu=$nslots
>   done
> }
> 
> me=`basename $0`
> 
> # test number of args
> if [ $# -ne 1 ]; then
>   echo "$me: got wrong number of arguments" >&2
>   exit 1
> fi
> 
> # get arguments
> pe_hostfile=$1
> 
> # ensure pe_hostfile is readable
> if [ ! -r $pe_hostfile ]; then
>   echo "$me: can't read $pe_hostfile" >&2
>   exit 1
> fi
> 
> # create machine-file
> # remove column with number of slots per queue
> # mpi does not support them in this form
> machines="$TMPDIR/machines"
> PeHostfile2MachineFile $pe_hostfile >> $machines
> 
> 
> #
> # Make script wrapper for 'rsh' available in jobs tmp dir
> #
> rsh_wrapper=$SGE_ROOT/mpi/rsh
> if [ ! -x $rsh_wrapper ]; then
>      echo "$me: can't execute $rsh_wrapper" >&2
>      echo "     maybe it resides at a file system not available at this 
> machine" >&2
>      exit 1
> fi
> 
> rshcmd=rsh
> case "$ARC" in
>      hp|hp10|hp11|hp11-64) rshcmd=remsh ;;
>      *) ;;
> esac
> # note: This could also be done using rcp, ftp or s.th.
> #       else. We use a symbolic link since it is the
> #       cheapest in case of a shared filesystem
> #
> ln -s $rsh_wrapper $TMPDIR/$rshcmd
> 
> 
> # signal success to caller
> exit 0
> 
> 
> 
> 
> stoplam.sh
> ------------------------------------------------------------
> #!/bin/sh
> #       rm $TMPDIR/machines
> 
> rshcmd=rsh
> case "$ARC" in
>   hp|hp10|hp11|hp11-64) rshcmd=remsh ;;
>   *) ;;
> esac
> rm $TMPDIR/$rshcmd
> 
> 
> 
> 
> LAM script Template
> ------------------------------------------------------------
> #!/bin/csh #
> # pe request
> #$ -pe lam 15
> #
> 
> cat $TMPDIR/machines echo " "
> 
> # change this line to fit your needs
> setenv PATH /usr/local/lam7.0/bin:$PATH lamboot -v $TMPDIR/machines
> 
> # work to do cd /home/barras/BENCHMARK_LAM/AVBP/RUN/RUN3
> mpirun C $AVBP_HOME/HOST/PENTIUM4/AVBPEXEC/AVBP_V5.2.PENTIUM4
> lamhalt -v
> 
> 
>             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>          Alexandre Barras - Computing Support Group
>        CERFACS, 42 Av. Coriolis, F-31057 TOULOUSE Cedex 1, FRANCE
>        Tel.: (+33) [0]5 61 19 30 75   Fax: (+33) [0]5 61 19 30 00
> 
>             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
> 


-- 
  ___________________________________________________________
| Christopher Duncan    Sun HPC Support Engineer [PTS-AMER] |
| Email: christopher.duncan at sun.com    Tel: +1 781-442-2309 |
|                   -=Carpe Noctem=-                        |
|___________________________________________________________|


    [ Part 2: "Attached Text" ]

#!/usr/bin/perl 

### INSTALL DIRECTIONS:
#
#  1. Install this PERL executable, sge-lam inside the LAM bin dir. 
#     Make sure it is executable.
#  2. Modify the following variables: LAMHOME below to fit your site setup. 
#

$LAMHOME="/home/dunk/LAM7";

#  3. Create an SGE PE that can be used to submit lam jobs. The following 
#     is an example assuming the scripts exist in /usr/local/lam/bin. 
#     You should replace the queue_list and slots with your site specific 
#     values or set it to "all" to use all the queues.  
#
#        % qconf -sp lammpi 
#        pe_name lammpi
#        queue_list all
#        slots 6
#        user_lists NONE
#        xuser_lists NONE
#        start_proc_args /usr/local/lam/bin/sge-lam start
#        stop_proc_args /usr/local/lam/bin/sge-lam stop
#        allocation_rule $fill_up
#        control_slaves TRUE
#        job_is_first_task FALSE
#
#    NOTE: It is probably easiest to use the qmon GUI to create the PE.
#
#   4. Add a new LAM node process schema into the $LAMHOME/etc area
#      named sge-lam-conf.lamd. This should be a single line that
#      adds the "sge-lam qrsh-local" prefix to the lamd startup.
#
#       % cat /usr/local/lam/etc/sge-lam-conf.lamd
#       /usr/local/lam/bin/sge-lam qrsh-local /usr/local/lam/bin/lamd  
#         $inet_topo $debug $session_prefix $session_suffix
#
#### Submitting SGE JOBS
#
#   Once this is setup users can submit jobs as normal and should not need to 
#   lamboot on their own. Users need only call mpirun for their MPI programs. 
#   Here is an example job:
#
#        % cat lamjob.csh
#        #$ -cwd
#        set path=(/usr/local/lam/bin $path)
#        echo "Starting my LAM MPI job"
#        mpirun C conn-60
#        echo "LAM MPI job done"
#
#
#
#### Comments/Issues email: christopher.duncan at sun.com
#
# END INSTALL


$verbose=1;
$debug=0;

# close STDIN to avoid stdio race conditions and tty issues
close(STDIN);

if( $debug eq 1){
	open(SGEDEBUG,"> /tmp/sgedebug.$ENV{JOB_ID}.$$");
	select(SGEDEBUG); $|=1;
	open(STDERR,">> /tmp/sgedebug.$ENV{JOB_ID}.$$");
}

# set output for stderr and stdout to be unbuffered
select(STDERR); $|=1;
select(STDOUT); $|=1;

$lamboot="$LAMHOME/bin/lamboot";
$lamhalt="$LAMHOME/bin/lamhalt";
$sgelamconf="sge-lam-conf.lamd";

$SGE_ROOT="$ENV{SGE_ROOT}";

$arch=`${SGE_ROOT}/util/arch`;
chomp($arch);
$qrsh="${SGE_ROOT}/bin/${arch}/qrsh";

# add LAM and SGE to path
$ENV{'PATH'}.=":${SGE_ROOT}/bin/${arch}";
$ENV{'PATH'}.=":${LAMHOME}/bin";

debug_print("LAMHOME = $LAMHOME");
debug_print("SGE_ROOT = $SGE_ROOT");
debug_print("PATH = $ENV{PATH}");
debug_print("qrsh = $qrsh");
debug_print("ARGV = \"".join("\" \"", at ARGV)."\"");

# read in the args to figure out our task
$func=shift @ARGV;

if("$func" eq "start"){
	debug_print("func=start");
	print "Starting SGE + LAM Integration\n";
	print "\t using tight integration scheme\n";
	start_proc_args();
}elsif("$func" eq "stop"){
	debug_print("func=stop");
	print "Stoping SGE + LAM Integration\n";
	stop_proc_args();
}elsif("$func" eq "qrsh-remote"){
	debug_print("func=qrsh-remote");
        qrsh_remote();
}elsif("$func" eq "qrsh-local"){
	debug_print("func=qrsh-local");
        qrsh_local();
}else{
	print STDERR "\nusage: $0 {start|stop|qrsh-remote|qrsh-local}\n\n";	
	exit(-1);
}


sub start_proc_args()
{

  # we currently place the LAM host file in the TMPDIR that SGE uses.
  # if we place it elsewhere we need to clean it up
  $lamhostsfile="$ENV{TMPDIR}/lamhostfile";

  # flags and options for lamboot (-x, -s and -np may be useful in some envs)
  @lambootargs=("-nn","-ssi","boot","rsh","-ssi","boot_rsh_agent","${LAMHOME}/bin/sge-lam qrsh-remote","-c","$sgelamconf");
  if($verbose){ push(@lambootargs,"-v"); }
  if($debug){ push(@lambootargs,"-d"); }
  push(@lambootargs,"$lamhostsfile");
  debug_print("LAMBOOT ARGS: @lambootargs $lamhostsfile");

  ### Need to convert the SGE hostfile to a LAM hostfile format
  # open and read the PE hostfile
  open(SGEHOSTFILE,"< $ENV{PE_HOSTFILE}");
  # convert to LAM bhost file format
  @lamhostslist=();
  while(<SGEHOSTFILE>){
	($host,$ncpu,$junk)=split(/\s+/);
	push( @lamhostslist,"$host cpu=$ncpu");
  }
  close(SGEHOSTFILE);

  # create the new lam bhost file
  open(LAMHOSTFILE,"> $lamhostsfile");
  print LAMHOSTFILE join("\n", at lamhostslist);
  print LAMHOSTFILE "\n";
  close(LAMHOSTFILE);


  if($debug){ close(SGEDEBUG); }
  exec($lamboot, at lambootargs);
}


sub stop_proc_args(){

  if($verbose){ push(@lamhaltargs,"-v"); }
  if($debug){ push(@lamhaltargs,"-d"); }

  if($debug){ close(SGEDEBUG); }
  exec($lamhalt, at lamhaltargs);
}


sub qrsh_remote()
{

  @myargs=("-inherit","-nostdin","-V", at ARGV);

  debug_print("QRSH REMOTE CONFIG: @myargs");
  if($debug){ close(SGEDEBUG); }
  exec($qrsh, at myargs);
}


sub qrsh_local()
{
  # we are running a local qrsh to add the lamd into the current job
  # on the local node using the LAM boot schema

  # get the hostname to pass to qrsh
  $hostname=`/bin/hostname`;
  chomp($hostname);

  # tell SGE to add this command into the JOB_ID job by using qrsh -inherit
  # the hostname is not passed in this case in ARGV by lamboot
  @myargs=("-inherit","-nostdin","-V","$hostname", at ARGV);

  debug_print("QRSH LOCAL CONFIG: @myargs");
  if($debug){ close(SGEDEBUG); }
  exec($qrsh, at myargs);
}


sub debug_print()
{
  if($debug){
    print SGEDEBUG "SGE-LAM DEBUG: @_\n";
  }
}



    [ Part 3: "Attached Text" ]

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net



More information about the gridengine-users mailing list