[GE users] File staging problem

Gerhard Venter gventer at sun.ac.za
Tue Jul 8 17:03:17 BST 2008


Kevin,

Thank you very much.  I do not have the $TMPDIR/machines files on my
system, but was able to use the $PE_HOSTFILE variable to gain similar
information.  I have a solution running at the moment based on your
script that a run as a prolog.

Thanks,
Gerhard

On Tue, 2008-07-08 at 12:54 +0100, Pursley, Kevin J wrote:
> This is an epilog that we use to clean scratch space on all nodes after
> an MPI job.
> We use loose integration so SGE just aloocates slots for the machines
> and starts
> A single process on the master node. Don't know about tight mpi
> integration, we don't use that.
> Also note that prologs and epilogs run as the user, nothing special
> there either. 
> 
> 
> #!/bin/csh -f
> # clean /scratch
> echo " start woodcrest epilog "
> foreach i (`cat $TMPDIR/machines`)
>    /usr/bin/rsh $i 'rm -Rf /scratch/*'
> End
> echo " end woodcrest epilog " 
> 
> -----Original Message-----
> From: Gerhard Venter [mailto:gventer at sun.ac.za] 
> Sent: Tuesday, July 08, 2008 2:33 AM
> To: users at gridengine.sunsource.net
> Subject: [GE users] File staging problem
> 
> I am using SGE 6.0 and OpenMPI.  I am distributing my job to multiple
> slots using a round robin mpi parallel environment - the result is that
> each slot is on a different compute node, each with its own local disk
> storage.  I would like to stage files from my home directory to the
> TMPDIR on each compute node that MPI will use.  My submit script looks
> something like this:
> 
> #!/bin/bash
> #$ -cwd
> #$ -j y
> #$ -pe openmpi_rr 4
> #
> cp input.dat $TMPDIR
> cd $TMPDIR
> mpirun -np ${NSLOTS} $SGE_O_WORKDIR/mpijob
> 
> The cd $TMPDIR does change the pwd on each node to $TMPDIR (I print the
> value for all slots from my mpi program), however, the files are only
> copied to the first MPI slot (the master slot) and not the other slots
> (the worker slots).  I have also implemented the prolog and epilog
> scripts from
> 
> http://gridengine.sunsource.net/project/gridengine/howto/filestaging/
> 
> but with the same result.  
> 
> Am I missing something?  Is there a way that I can stage input files for
> all the slots that will be used in a MPI job?
> 
> Thanks,
> Gerhard
> 
-- 
+------------------------------------------------------------------+
|| Prof. Gerhard Venter
||
|| Departement Meganiese en        |  Department of Mechanical and
||   Megatroniese Ingenieurswese   |    Mechatronic Engineering
|| Universiteit Stellenbosch       |  Stellenbosch  University
|| Privaat Sak X1 Matieland 7602   |  Private Bag X1 Matieland 7602  
|| Suid-Afrika                     |  South Africa
||
|| Tel: +27 21 808 3560
|| E-Mail: gventer at sun.ac.za          Web: www.eng.sun.ac.za
+------------------------------------------------------------------+
-- 
+------------------------------------------------------------------+
|| Prof. Gerhard Venter
||
|| Departement Meganiese en        |  Department of Mechanical and
||   Megatroniese Ingenieurswese   |    Mechatronic Engineering
|| Universiteit Stellenbosch       |  Stellenbosch  University
|| Privaat Sak X1 Matieland 7602   |  Private Bag X1 Matieland 7602  
|| Suid-Afrika                     |  South Africa
||
|| Tel: +27 21 808 3560
|| E-Mail: gventer at sun.ac.za          Web: www.eng.sun.ac.za
+------------------------------------------------------------------+


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list