[GE users] merging stdout

Olesen, Mark Mark.Olesen at emcontechnologies.com
Wed Nov 12 12:27:36 GMT 2008


> when I do qsub -t 1000-1049 main.sh
> (50 array jobs)
> 
> I also need to do a
> 
> $ ./combine (PREVIOUS_JOB_ID)
> 
> And I would like to do this once main.sh is completed and all in one
> shot.
[snip]
> > What about submitting this as a followup job:
> >
> > qsub -hold_jid <JOB_ID_OF_THE_ARRAY_JOB> combine
> >
> > It will wait (i.e. being on hold), until all tasks have finished.

Reuti actually gave you most of the answer; you just have to work out a
few details on your own. For this, you just have to reformulate things a
little bit.

The simplest is to write a submission wrapper for 'main.sh', eg
'qmain.sh' that passes all its arguments (or up to the '--' delimiter)
to qsub as options ... although I write most of my wrappers in Perl.
 
The wrapper submits two jobs.
The first job is the script itself.
The second job has a hold on the first job:

Eg,

# submit job and capture the job id
# need to add any other qsub options in here
job_id=$($SGE_BINARY_PATH/qsub -terse $0 $@ 2>&1)
exitcode=$?

if [ "$exitcode" -eq 0 ]
then
   # first job submit worked, now add combine task
   qsub -hold_jid $job_id -N _$job_id combine $job_id 
else
   echo "failed"
   exit $exitcode
fi

The second job name was named "_$job_id" to make it easier to find
later.

Note, you can even just rewrite main.sh to act as its own wrapper. Just
have different behaviour when called directly or when running as a job: 
if [ -z "$JOB_ID" ]
then
   echo "called directly (not a GridEngine job)"
else
   echo "running as GridEngine job $JOB_ID $SGE_TASK_ID"
fi


You cannot directly enforce that the second job runs immediately after
the first one, but if you add a complex with really high urgency, you
can increase the probability. Eg,

#name   shortcut  type  relop  requestable  consumable default  urgency
#---------------------------------------------------------------------
cleanup cleanup   BOOL  ==     YES         NO         0        1e12 

Or have it run on a queue that no other jobs can use (ie, by using a
forced complex). Maybe you could even make the second job a deadline job
with an immediate start time. The hold should prevent it from running
immediately and the deadline should make the priority skyrocket!

There are *lots* of ways to solve your problem, you just have to
experiment a bit now to see which way works best for you.

/mark
This e-mail message and any attachments may contain 
legally privileged, confidential or proprietary Information, 
or information otherwise protected by law of EMCON 
Technologies, its affiliates, or third parties. This notice 
serves as marking of its "Confidential" status as defined 
in any confidentiality agreements concerning the sender 
and recipient. If you are not the intended recipient(s), 
or the employee or agent responsible for delivery of this 
message to the intended recipient(s), you are hereby 
notified that any dissemination, distribution or copying 
of this e-mail message is strictly prohibited. 
If you have received this message in error, please 
immediately notify the sender and delete this e-mail 
message from your computer.

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=88528

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list