[GE users] Reservation of resources in duration of checkpoing so that no other job can be able to use those resources.

reuti reuti at staff.uni-marburg.de
Wed Aug 18 08:36:01 BST 2010


Hi,

Am 18.08.2010 um 08:55 schrieb sgerns:

> Thanks for your prompt reply & Concern.
> I am looking for your help in this.
>  
> Please see the inline reply for understanding the scenario.
> 
>  
> On Tue, Aug 17, 2010 at 5:47 PM, reuti <reuti at staff.uni-marburg.de> wrote:
> Hi,
> 
> Am 17.08.2010 um 16:57 schrieb sgerns:
> 
>  
>  
>  
> > I have a scenario here, I am trying to explain this in steps.
> >
> > 1. I have some jobs running in the cluster, based on the priority of the queues (Few queues have higher priority than others & so on).
> >
> > 2. I am also having a VIP.queue which is having the highest priority among all the queues. So ofcourse jobs which has been submitted through vip queue will definitely go up in queue, as I have shown below.
> > 100    0.50500        job_name1      usr1       r              normal.q at exehost             32
> >
> > 101     0.50500      job_name2       usr1       r             normal.q at exehost              16
> >
> > 102     0.50500       job_name2      usr1       r             normal.q at exehost              32
> >
> > 103     0.50500        job_name3      usr2      r             normal.q at exehost              128
> >
> > 104     0.50973         job_name4      usr3     qw              VIP.q at exehost.              64
> >
> > 105     0.50973         job_name5      usr3      qw           normal.q at exehost         32
> >
> > 106     0.51514         job_name6        usr4     qw            normal.q at exehost         16
> >
> >
> > 3. Now These VIP Jobs are very important jobs & I want these jobs to run ASAP, Hence I will checkpoint the lower priority jobs which are running right now.
> 
> how are you checkpointing these jobs, i.e. which checkpointing environment did you set up in SGE?
> 
> --->> I am automaticaly checkpointing the jobs here (I am writing some scripts (Wrapper of job scripts)

this is not exactly what I meant. You can trigger a checkpointing mechanism by SGE on its own, when your application already supports it. So: you didn't define any checkpointing environment in SGE, but doing it "by hand" from SGE's point of view, i.e. outside of SGE (`man sge_sckpt`and `man checkpoint`).


> which will decide which job to checkpoint based on the priority & then we will send the checkpointing signal to those jobs automatically) : Nothing is manual we are not doing it by hand.
> If these automating scripts (Wrapper of the job scripts) decides which job to checkpoint as shown below e.g. job 102 & job 100
> 
> > 4. Suppose job id 100 & 102 I have selected for checkpointing & send checkpointing signal to those jobs, & It has taken 2 hours to chekpoint these jobs.
> >    I do not want any other job to run on these resources during this duration of 2 hours.
> >    Because there is a possibilty that small jobs can get the resources & start running
> 
> You mean, you e.g. suspend these jobs, which will be checkpointed as an result? As SGE thinks the queue is free again, it might be used by other jobs. Why not suspend the normal.q at exehostXY

I mentioned "suspend" here, as instead of suspending the job, it will get a signal when run with a checkpointing environment with proper setting.


> --->>  Now we will send the checkpointing signals to these jobs (i.e. job id 100 & 102 ) suppose these jobs are very long jobs & running from 2 days so probably it can take say 2 hours to get checkpointed & release the resources
> Now  my problem is during this duration I don want normal jobs to eat up these resources,

As long as the jobs are in SGE's `qstat` list, the resources are still reserved, and can't be used by other jobs.


> (It would be the possibility that few resources are free earlier & any small job will take those resource which we had freed for VIP job.)
>  
> --->> Yes suspending the normal.q

Disable only a queue instance `qmod -d normal.q at exehostXY`.

The problem is anyway only the short time after the checkpoint is done, and the VIP.q job starts.

-- Reuti


> is the way we had thought about but There are many side effects of that as you are knowing.
> if jobs will take more time to get checkpointed (say 5 hours) then all my queues are blocked & wont be able to do any processing in this duration, even if some other small resources are available for those jobs (e.g. 8 cores are freed by any other job which has finished these queues wont be able to run the jobs)
>  
> ---->>>Is there any other way we can do this so that we will be able to reserve the resources for the duration of checkpointing so that no other job can be able to take & use that.
>  
> kindly give a thought on this.
>  
>  Regards
>  Rajan
> 
> 
> PS: If you are doing all by hand w/o a checkpointing environment, the queue instances normal.q at exehostXY just need to be disabled with `qmod -d normal.q at exehostXY` AFAICS.
> -- Reuti
> 
> > Kindly help me How can I be able to reserve the resources for the duration of checkpointing so that any other un important job can not able to start.
> >
> >
> >
> >
> 
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=274953
> 
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=275131

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list