[GE users] dependant jobs on same host?

reuti reuti at staff.uni-marburg.de
Tue Nov 11 17:07:04 GMT 2008


Am 11.11.2008 um 16:51 schrieb udo waechter:

> Hello again.
> I have many jobs which are split up in 2 steps where step2 is
> dependant on step1.
> I submit those with
> 	qsub -N jstep1_N ...
> and
> 	qsub -N jstep2_N -hold_jid jstep1_N ....
> where N is a unique int. Thus, each of these tuples is unique.
> Are those two jobs executed on the same execution host?

not necessarily.

> If not, is it
> possible to somehow ensure that this happens?
> Is this possible to do without any parsing of qstat output or so?

There is no flag to specify that both should run on the same machine  
- unless you limit yourself also for the first job to run on exactly  
one machine. One approach could be, to submit the second job from the  
first running jobscript, as you know by a simple `hostname` on which  
machine you are running and could request it "-l h=`hostname`". For  
this to work, the execution node must also be a submit host.

This would mean a longer time to wait for the second job, as it's  
only submitted when the first job run already. A similar approach  
would be to use "qalter" in the first job to change the hard resource  
request for "-q *@`hostname`". If you would use the resource requests  
here, you would need to specify all other requests which were already  
set, as the qalter will replace the compelte resource request list -  
changing the queue is easier. Problem here of course: the first job  
would need to know the number/name of the second job. As you have a  
unique jobnames you can use "jstep2${JOB_NAME#jstep1}" to reference  
it (the term will remove jstep1 from the variable's content and  
prepend it with jstep2).

BTW: With the way you like to work you can't use the $TMPDIR as it's  
removed after the first jobstep. Maybe it would be an idea to have a  
prolog and epilog for this queue. In the prolog you could check  
whether this is jstep1 and create a directory on the node "mkdir /tmp/ 
`basename $TMPDIR`_persistent" and if it's jstep2 remove it.

-- Reuti


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

More information about the gridengine-users mailing list