[GE users] qalter-ing consumable resource requests

reuti reuti at staff.uni-marburg.de
Mon Nov 30 23:13:27 GMT 2009


Hi,

Am 30.11.2009 um 21:06 schrieb jerry37:

> I was wondering, why its not possible to qalter resource request of  
> currently running job - i.e. add consumable resource to it. The  
> expected behavior wouldn't be for it to be accounted for  
> immediately but after requeue/migration.
> The application of this would be in creating a 'estimation' job,  
> that reads user data and computes resource requests for solution -  
> memory/scratch/licensing etc. Then these values should go into  
> request of the real worker job and scheduler can plan it accurately  
> to its needs. Currently I am using two jobs for this - one  
> 'estimating' and then the 'worker', which has job-interdependency  
> set up to the estimation. Withing the 'estimating' job, resource  
> requests for the worker are altered accordingly. This does work -  
> even though qalter says 'denied: resource request on consumable  
> "someconsumable" of running job was not contained former resource  
> request' it does alter the resource requests.
> What I wanted to do, was creating a single job, that would  
> recognize it is run the first time, do the estimation part, then  
> change its own resource requests and then exit 99 (requeue). Then  
> it would run on appropriate hardware. It is not really possible to  
> do the estimation before, because its not a trivial task and can  
> take some time (not to mention that software to do it is not  
> accessible on the submit machines).
> Since there are already lot of job options of a running job that  
> can be changed with qalter, I'd like to know, why the above is  
> supposed to be impossible. Anyone?

this would mean to have two entries for the complex: the actual one  
(from which something will be subtracted) and the new one for the  
next run. Then also both should be visible in "qstat -j <jobid>", so  
that someone can check the behavior of the job. Otherwise you might  
get confused when you look at "qhost -F" and discover some  
inconsistency.

What about a different approach: instead of submitting two jobs and  
changing the resource requests of the second one, use the first job  
to create the "header" for the real job, i.e. construct something like:

#$/bin/sh
# Do some computations
# $1 = name
# $2 = memory
MYJOB="$1-modified"
let MEMORY=$2+500
cat > myjobscript.$$.sh <<EOF
#!/bin/sh
#$ -N $MYJOB
#$ -l h_vmem=${MEMORY}G
#$ -l h=node164
echo \$HOSTNAME
EOF
cat >> myjobscript.$$.sh < $userfile
ssh headnode qsub myjobscript.$$.sh
# Remove the script for -b n, otherwise it must stay there
rm myjobscript.$$.sh

The $$ will use the process-id to avoid race conditions in case you  
use the same name more than once (to be really safe also the hostname  
would be necessary, but using the pid should do it in most of the  
cases).

Then append the real job script and submit. If you create this with  
"cat" and a here-document, the variables inside will be resolved at  
creation time of the script. If you want to put a variable to be used  
at runtime there, you have to escape the $ (luckily $ alone needn't  
to be escaped).

(ssh to the headnode with hostbased authtication can help to avoid  
that every machine is also a submit-host, you could also submit local  
on the node of course.)

-- Reuti


> user at machine:~# man qalter | grep -c "allows changing this option  
> even while the job executes."
> Reformatting qalter(1), please wait...
> 11
>
> Thanks,
>     Jerry
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do? 
> dsForumId=38&dsMessageId=230535
>
> To unsubscribe from this discussion, e-mail: [users- 
> unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=230562

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list