[GE users] RE: Myrinet setup (was: RE: [Beowulf])

Ron Chen ron_chen_123 at yahoo.com
Sun Mar 27 21:50:50 BST 2005


Reuti,

The original SGE-Myrinet integration was written by
Shannon in 2002, and I believe it needs to write to a
common location to get the host/GM port info.

Newer MPICH-GM versions automatically allocate the
required number of GM ports on each of the nodes, and
thus SGE doesn't need to list the port in the machine
file.

Myrinet (GM-2) with SGE6
========================
1) GM2 has 16 ports (vs. 8 with GM-1) available per
NIC, and is usually more than the number of processors
on each node. If the setup is like this, then just
setup a normal tight-MPICH PE should be fine. (unless
your users DoS the cluster by allocating a large
number of GM-ports)

Make sure you read Reuti's "Tight Integration of MPICH
and SGE" HOWTO.


2) If the nodes have more CPUs than the number of
Myrinet ports, and if you want to use serial jobs to
backfill the cluster, then you need to setup GM-port
as a resource and also a loadsensor to report the
number of free GM-ports available. (use gm_board_info
or gm_yp)

Jobs then request for a GM-port (SGE will multiplie
that with the number of SLOTS) and the SGE scheduler
will pick nodes with the needed number of CPUs and
GM-ports available to run the parallel job.

 -Ron


--- Reuti <reuti at staff.uni-marburg.de> wrote:
> in principle you can have $TMPDIR on a file server.
> The 65534 was the 
> nobody/nogroup as the root_squash to the
> anonuid/anongid is the default for 
> exports. But instead of trimming the file server to
> support it in a proper way, 
> it's safer (and faster) to have it locally on the
> nodes as you will have it 
> now. 
> 
> CU - Reuti
> 
> Quoting William Burke <wburke999 at msn.com>:
> 
> > Hi Reuti,
> > 
> > Quoting Reuti [reuti at staff.uni-marburg.de]:
> > 
> > <snip>
> > > First one additional question (before I route
> you in the wrong
> > direction):
> > 
> > > is it necessary for you to have a shared $TMPDIR
> for SGE? ...More common 
> > > and faster is to use the local /tmp on the nodes
> for this
> > 
> > Actually there is no reason that I should have a
> shared $TMPDIR for SGE,
> > except that I read some where (I am not sure
> where)that it was recommended
> > to share the $TMPDIR. However I just seached the
> SGE and N1GE docs I found
> > no evidence supporting that notion and now
> thinking about it setting
> > $TMPDIR
> > to local /tmp would simplfy things. Thxs
> > 
> > Regards,
> > William
> > 
> > -----Original Message-----
> > From: Reuti [mailto:reuti at staff.uni-marburg.de] 
> > Sent: Sunday, March 27, 2005 7:09 AM
> > To: William Burke
> > Cc: users at gridengine.sunsource.net;
> beowulf at beowulf.org; dag at sonsorol.org;
> > 'John Hearns'
> > Subject: Myrinet setup (was: RE: [Beowulf])
> > 
> > Hi Will,
> > 
> > Quoting William Burke <wburke999 at msn.com>:
> > 
> > <snip>
> > > I have but I do not see my name yet? How long is
> the verification
> > process?
> > 
> > did you register as an observer - AFAIK you can
> post to the SGE list
> > without
> > 
> > being registered.
> > 
> > <snip>
> > > The mpi directory's mpich.template doesn't use
> mpirun.ch_gm so how does
> > it
> > > know what version of mpirun to use? If I use the
> mpi what changes do I
> > have
> > > to make?
> > 
> > You can have more than one MPI implementation
> installed in your cluster,
> > and
> > it 
> > may need some planning to set up the correct $PATH
> for each of the 
> > implementations you want to use (and the this way
> located mpirun must fit
> > to
> > 
> > your used version of MPI during compilation of
> your program). You may use a
> > 
> > "which mpirun" to check it in your job script.
> > 
> > Also the supplied sge_mpirun will not use any
> Myrinet version on it's own -
> > 
> > it's just a wrapper to the mpirun you set in the
> PE, so that you don't have
> > to 
> > specify the usual options 'mpirun -machinefile
> $TMPDIR/machines -np $NSLOTS
> > 
> > mypgm'.
> > 
> > I must admit: seems that the Myrinet stuff was
> more for 5.3 and not
> > updated,
> > as 
> > in 6.0 you can have more than one line for
> "start_proc_args" in your PE 
> > definition - so it just grabs the last \ in the
> first and only line
> > beginning 
> > with "start_proc_args" line as mpirun command -
> will give the error message
> > you 
> > got, that "\" is not existing.
> > 
> > As I said: we can use the default MPICH
> integration also for Myrinet and 
> > proceed this way.
> > 
> > > >> Can you please give more details of your
> queue and PE setup (qconf
> > > -sq/sp
> > 
> > Thx, I will keep the stuff. First one additional
> question (before I route
> > you 
> > in the wrong direction): is it necessary for you
> to have a shared $TMPDIR
> > for 
> > SGE? This is the one you set in your queue
> configuration (tmpdir 
> > /WEMS/grid/tmp) and seems for now to be on a file
> server.
> > 
> > More common and faster is to use the local /tmp on
> the nodes for this (you
> > are 
> > right: SGE want to create there a directory for
> this job and some file for
> > its 
> > own usage - but you are free to use this directory
> $TMPDIR also in your job
> > 
> > script). It will be created for your job, and
> cleanly deleted after the
> > job,
> > so 
> > you won't have any leftover files.
> > 
> > Cheers - Reuti
> > 
> 
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail:
> users-help at gridengine.sunsource.net
> 
> 




		
__________________________________ 
Do you Yahoo!? 
Yahoo! Small Business - Try our new resources site!
http://smallbusiness.yahoo.com/resources/ 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list