[GE users] qalter-ing consumable resource requests

jerry37 jerry37 at seznam.cz
Tue Dec 1 09:20:11 GMT 2009


> ------------ Original message ------------
> From: reuti <reuti at staff.uni-marburg.de>
> Subject: Re: [GE users] qalter-ing consumable resource requests
> Date: 01.12.2009 00:50:49
> ----------------------------------------
> Hi,
> Am 30.11.2009 um 21:06 schrieb jerry37:
> > I was wondering, why its not possible to qalter resource request of  
> > currently running job - i.e. add consumable resource to it. The  
> > expected behavior wouldn't be for it to be accounted for  
> > immediately but after requeue/migration.
> > The application of this would be in creating a 'estimation' job,  
> > that reads user data and computes resource requests for solution -  
> > memory/scratch/licensing etc. Then these values should go into  
> > request of the real worker job and scheduler can plan it accurately  
> > to its needs. Currently I am using two jobs for this - one  
> > 'estimating' and then the 'worker', which has job-interdependency  
> > set up to the estimation. Withing the 'estimating' job, resource  
> > requests for the worker are altered accordingly. This does work -  
> > even though qalter says 'denied: resource request on consumable  
> > "someconsumable" of running job was not contained former resource  
> > request' it does alter the resource requests.
> > What I wanted to do, was creating a single job, that would  
> > recognize it is run the first time, do the estimation part, then  
> > change its own resource requests and then exit 99 (requeue). Then  
> > it would run on appropriate hardware. It is not really possible to  
> > do the estimation before, because its not a trivial task and can  
> > take some time (not to mention that software to do it is not  
> > accessible on the submit machines).
> > Since there are already lot of job options of a running job that  
> > can be changed with qalter, I'd like to know, why the above is  
> > supposed to be impossible. Anyone?
> this would mean to have two entries for the complex: the actual one  
> (from which something will be subtracted) and the new one for the  
> next run. Then also both should be visible in "qstat -j <jobid>", so  
> that someone can check the behavior of the job. Otherwise you might  
> get confused when you look at "qhost -F" and discover some  
> inconsistency.

I don't really understand. I mean - being able to change any other option (that can be changed) of a running job, you introduce inconsistency into qstat output anyway, so whats the big deal?

> What about a different approach: instead of submitting two jobs and  
> changing the resource requests of the second one, use the first job  
> to create the "header" for the real job, i.e. construct something like:
> #$/bin/sh
> # Do some computations
> # $1 = name
> # $2 = memory
> MYJOB="$1-modified"
> let MEMORY=$2+500
> cat > myjobscript.$$.sh <<EOF
> #!/bin/sh
> #$ -N $MYJOB
> #$ -l h_vmem=${MEMORY}G
> #$ -l h=node164
> echo \$HOSTNAME
> cat >> myjobscript.$$.sh < $userfile
> ssh headnode qsub myjobscript.$$.sh
> # Remove the script for -b n, otherwise it must stay there
> rm myjobscript.$$.sh
> The $$ will use the process-id to avoid race conditions in case you  
> use the same name more than once (to be really safe also the hostname  
> would be necessary, but using the pid should do it in most of the  
> cases).
> Then append the real job script and submit. If you create this with  
> "cat" and a here-document, the variables inside will be resolved at  
> creation time of the script. If you want to put a variable to be used  
> at runtime there, you have to escape the $ (luckily $ alone needn't  
> to be escaped).
> (ssh to the headnode with hostbased authtication can help to avoid  
> that every machine is also a submit-host, you could also submit local  
> on the node of course.)

I know what you mean, but we chose not to allow job submission from within the job. Right now, all computations, no matter of how many interdependant job they consist of, are static set of jobs bound to one DRMAA session that share certain configurations. Such set is not altered very easily so thats why the choice.


> -- Reuti
> > user at machine:~# man qalter | grep -c "allows changing this option  
> > even while the job executes."
> > Reformatting qalter(1), please wait...
> > 11
> >
> > Thanks,
> >     Jerry


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

More information about the gridengine-users mailing list