[GE users] Gaussian03 + LINDA, looking for tight integration details

Reuti reuti at staff.uni-marburg.de
Tue Aug 17 13:04:23 BST 2004


Hi,

all the things I made (and I hope I didn't forgot anything):


1. Modify linda_rsh

You are completely right, you have to change the call in linda_rsh from /usr/bin/rsh to rsh.


2. Create parallel environments

Depending, whether you have dual machines, you have to setup two parallel environment,
where the allocation rule is fixed to one and two slots. This is neccessary, because you can
only tell LINDA by the contents of GAUSS_LFLAGS to use one ow two CPUs for all nodes, although
you can get a different distribution of nodes, when you setup SGE with e.g. $round_robin (watch out
to include the allocation rule in the name, we need it later):

> qconf -sp linda2
pe_name           linda2
queue_list        para00 para01 para02 para03 para04 para05 para06 para07 para08 para09
slots             20
user_lists        NONE
xuser_lists       NONE
start_proc_args   /usr/sge/linda/startlinda.sh -catch_rsh -unique $pe_hostfile
stop_proc_args    /usr/sge/linda/stoplinda.sh $pe_hostfile
allocation_rule   2
control_slaves    TRUE
job_is_first_task TRUE
> 

> qconf -sp linda1
pe_name           linda1
queue_list        para00 para01 para02 para03 para04 para05 para06 para07 para08 para09
slots             20
user_lists        NONE
xuser_lists       NONE
start_proc_args   /usr/sge/linda/startlinda.sh -catch_rsh -unique $pe_hostfile
stop_proc_args    /usr/sge/linda/stoplinda.sh $pe_hostfile
allocation_rule   1
control_slaves    TRUE
job_is_first_task TRUE
> 


3. Create directories on the nodes

Gaussian is expecting the scratch directory, which you specify e.g. with GAUSS_SCRDIR,
to be present on all nodes. SGE on itself is creating on each node a directory with the
nodename in the name (which is still used by SGE on its own). Therefore you have to
modify the the startlinda.sh and stoplinda.sh, which started out from the provided
startmpi.sh and stopmpi.sh from SGE. I suggest to duplicate the mpi directory in your
SGE installation and name it linda. For the startup script insert the lower part of this
after the trace of the machines file:

<snip>
# trace machines file
cat $machines

# Making job subdirectories on all nodes
myhost=`hostname`
for node in `cat $TMPDIR/machines` ; do
  [ "$myhost" != "$node" ] && $SGE_ROOT/bin/$ARC/qrsh -inherit $node "mkdir $TMPDIR"
done
</snip>


For the stoplinda.sh add this (which will also remove the outputfiles of SGE, if they are empty anyway):

<snip>
rm $TMPDIR/$rshcmd

#
# Remove the directories on the nodes.
#

# test number of args
if [ $# -ne 1 ]; then
   echo "stoplinda.sh: got wrong number of arguments" >&2
   exit 1
fi

# get arguments
pe_hostfile=$1

myhost=`hostname`
for node in `cat $TMPDIR/machines` ; do
  [ "$myhost" != "$node" ] && $rshcmd $node rm -rf $TMPDIR
done

rm $TMPDIR/machines

if [ -r "$SGE_STDOUT_PATH" -a -f "$SGE_STDOUT_PATH" ] ; then
  counter=`wc -l $SGE_STDOUT_PATH`
  [ "${counter%%$SGE_STDOUT_PATH}" -eq $((NHOSTS+1)) ] && rm -f $SGE_STDOUT_PATH
fi
[ -r "$SGE_STDERR_PATH" -a -f "$SGE_STDERR_PATH" ] && [ ! -s "$SGE_STDERR_PATH" ] && rm -f $SGE_STDERR_PATH

exit 0
<end>

(Side note: I set a filename for the output and error files, then they will be joined with the output and error files of the PE.)


4. Adjust rsh-wrapper

To get all environment variables with the PATH to the util.so also spread to the nodes,
you have to modify the rsh-wrapper to include the -V in your copied linda directory of SGE
(I commented out the echo, after I got it working):

if [ x$just_wrap = x ]; then
   if [ $minus_n -eq 1 ]; then
#      echo $SGE_ROOT/bin/$ARC/qrsh -V -inherit -nostdin $rhost $cmd
      exec $SGE_ROOT/bin/$ARC/qrsh -V -inherit -nostdin $rhost $cmd
   else
#      echo $SGE_ROOT/bin/$ARC/qrsh -V -inherit $rhost $cmd
      exec $SGE_ROOT/bin/$ARC/qrsh -V -inherit $rhost $cmd
   fi
else


5. Create serial queues

Because not all steps of Gaussian are parallel, I suggest to have a SGE setup,
where you have two parallel slots on a dual machine and in addition one serial slot.
So you can adjust the "priority" of the serial queue to set it to 15* (and leave the
parallel one at 0), otherwise the parallel tasks may have a too different behavior
of the required time for their calculations between the nodes.

*) SGE 5.3 or SGE 6.0 with reprioritization turned off.


6. Remove serial output files if you like

Also for the serial case I created an epilog, to remove the created files of SGE:

#!/bin/sh
#
#

#
# Just remove the standard output- and error-file, if they are empty.
#

[ -r "$SGE_STDOUT_PATH" -a -f "$SGE_STDOUT_PATH" ] && [ ! -s "$SGE_STDOUT_PATH" ] && rm -f $SGE_STDOUT_PATH
[ -r "$SGE_STDERR_PATH" -a -f "$SGE_STDERR_PATH" ] && [ ! -s "$SGE_STDERR_PATH" ] && rm -f $SGE_STDERR_PATH

exit 0


7. Create a submit script for your users

There you have to setup the GAUSS_LFLAGS in a proper way. I also prefer having
the working directory set to the temporary directory on the nodes, so you don't
need any setting of the GAUSS_SCRDIR at all. You have to include the following lines,
to set the value of -mp in the correct way (adjust it, so that it fits to your
scripts):

...
#$ -pe linda* $cpunr
...
MP=`echo $PE | sed "s/[^[:digit:]]//g"`
export GAUSS_LFLAGS="-mp $MP -nodelist '`cat $TMPDIR/machines`'"
export g03root=$binary_location
. $g03root/g03/bsd/g03.profile
cd $TMPDIR
g03l < $parallel_infile > $outfile


I hope, this all will give you a good starting point,

Cheers - Reuti



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list