[GE users] File copy from local /scratch on job termination

Pursley, Kevin J kevin.pursley at bp.com
Mon Dec 15 14:18:21 GMT 2008


Maybe use an epilog script? 

-----Original Message-----
From: Bart Willems [mailto:b-willems at northwestern.edu] 
Sent: Monday, December 15, 2008 8:09 AM
To: users at gridengine.sunsource.net
Subject: [GE users] File copy from local /scratch on job termination

Hi All,

I recently urged our cluster users to use local scratch space on the
cluster nodes instead of the NFS mounted RAID during their calculations.
In the example submission file below all required files for the job are
copied over to the node's local hard disk ($TMPDIR is /scratch) and
copied back when the job completes. However, the files only get copied
back when the job exits normally. If SGE terminates the job because it
exceeds the requested CPU time or if a user manually terminates a job
with qdel, the files are not copied back from the node's local hard disk
to the RAID. Is there any way around this?

Thanks,
Bart


#!/bin/bash

#$ -S /bin/bash
#$ -j y
#$ -N helloworld_test
#$ -l h_cpu=00:02:00
#$ -cwd

# Copy job files to local scratch space
JOBFILE=jobfiles.job-id-$JOB_ID.tgz
tar cfz $JOBFILE helloworld
cp $JOBFILE $TMPDIR
rm -rf $JOBFILE
cd $TMPDIR
tar xfz $JOBFILE
rm -rf $JOBFILE

# Computational command to run
./helloworld

# Copy all files back.
OUTFILE=outfiles.job-id-$JOB_ID.tgz
tar cfz $OUTFILE *
cp $OUTFILE $SGE_CWD_PATH
cd $SGE_CWD_PATH
tar xfz $OUTFILE
rm -rf $OUTFILE

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessage
Id=92671

To unsubscribe from this discussion, e-mail:
[users-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=92674

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list