[GE users] File copy from local /scratch on job termination

Bart Willems b-willems at northwestern.edu
Mon Dec 15 14:09:18 GMT 2008


Hi All,

I recently urged our cluster users to use local scratch space on the
cluster nodes instead of the NFS mounted RAID during their calculations.
In the example submission file below all required files for the job are
copied over to the node's local hard disk ($TMPDIR is /scratch) and copied
back when the job completes. However, the files only get copied back when
the job exits normally. If SGE terminates the job because it exceeds the
requested CPU time or if a user manually terminates a job with qdel, the
files are not copied back from the node's local hard disk to the RAID. Is
there any way around this?

Thanks,
Bart


#!/bin/bash

#$ -S /bin/bash
#$ -j y
#$ -N helloworld_test
#$ -l h_cpu=00:02:00
#$ -cwd

# Copy job files to local scratch space
JOBFILE=jobfiles.job-id-$JOB_ID.tgz
tar cfz $JOBFILE helloworld
cp $JOBFILE $TMPDIR
rm -rf $JOBFILE
cd $TMPDIR
tar xfz $JOBFILE
rm -rf $JOBFILE

# Computational command to run
./helloworld

# Copy all files back.
OUTFILE=outfiles.job-id-$JOB_ID.tgz
tar cfz $OUTFILE *
cp $OUTFILE $SGE_CWD_PATH
cd $SGE_CWD_PATH
tar xfz $OUTFILE
rm -rf $OUTFILE

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=92671

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list