[GE users] Learning to set up queues

Chris Dagdigian dag at sonsorol.org
Fri Sep 26 21:38:01 BST 2008


Hi Margaret,

{ It's been a very long day so I'm apologizing in advance if I miss  
anything obvious or give bad advice due to brain overload -- if so  
someone from the list will correct me ... }

This is a reasonable approach, as you get used to the system you may  
end up refining your methods, with SGE there are many ways to take  
"what you want to do" and turn it into SGE configuration and policy  
directives.

Some comments:

- Normally I'd recommend against having so many queues as you could  
just implement priority schemes with SGE Functional policies and  
resource quotas if necessary all within a nice single "all.q"  but  
your requirement for having different hard runtime limits means that  
you will have to have multiple queues

- All of your "priority" stuff is happening via Unix nice values, you  
are not using any of the SGE policy mechanisms or resource quotas at  
all. This means that I could login on monday morning and fill 100% of  
your available 2h.q slots with my own jobs. This may be "unfair" to  
other people that morning who also want a shot at the 2hr queue but  
arrive later on in the morning. They'd have to wait for my 2hr jobs to  
drain before they got a shot.

- All your queues are active at all times which can lead to nodes  
being oversubscribed. The example here would be a job in the reg.q  
queue that is running at a reduced nice value but is still consuming  
tons of physical memory and causing all of the higher priority jobs to  
swap out or otherwise slow down - this can have a major effect on your  
entire cluster. People solve this with queue subordination or resource  
quotas.

- You don't have a mechanism in place for blocking people with short,  
unimportant jobs from consuming slots in the "high priority" queues.  
In fact if I was an end user on your cluster I'd pretty much always  
bypass your "reg.q" and always submit to the 24 hour queue. Heck if I  
broke my workflow up in to 2-hour chunks I could totally consume your  
2hr highest priority queue pretty much all the time. This is not a  
show stopper problem but may be one of "user education" if you are in  
an environment where people try to game the system - watch out for  
people who think they are more important than others and try to game  
the system.

- Your comment about "the submission wait time and the requirements of  
the jobs also weigh in on the run priority..." is only true if you  
configure the SGE policies that deal with entitlements for requested  
resources (part of the Urgency policy mechanism) and to actually care  
about wait time (wait_wait_time is a sub-policy of the Urgency  
mechanism). It does not appear that you have enabled these things.



Long story short, only you can configure the system the way you need  
it to behave.

What you are proposing is totally reasonable - if this is a new  
cluster though I would treat this policy as "phase I" and let you and  
your users use it for a few weeks to see how it does. I always tell  
people that it is impossible to get an SGE queue and policy  
configuration correct the first time around -- you always have to let  
real world users and work bang away at it for a while so you can learn  
what is working or not-working for the local workflows.


-Chris








On Sep 26, 2008, at 3:34 PM, Margaret Doll wrote:

> I have 32 nodes.   I want to have a queue where jobs that need high  
> priority and a maximum of 2 hours of CPU can be submitted and a  
> queue where jobs that need high priority and will use a maximum of  
> 24 hours of CPU can be submitted.
>
> I put all 32 nodes into 3 queues; ie., reg.q, 2hour.q and 24hour.q.   
> All queues show up with 32 nodes.
>
> Through qmon, I limited 2hour.q to 2 hours CPU and set nice to -4.
> 				I limited 24 hour.q to 24 hours of CPU and set nice to -2
> 			I set nice on reg.q to +2
>
> Is this the best way to set this up?
>
> Jobs submitted to 2hour.q will get the highest priority.  Jobs  
> submitted to 24hour.q will be next on the schedule.  Jobs submitted  
> to reg.q will get last billing.  I understand that the submission  
> wait time and the requirements of the jobs also weigh in on the run  
> priority.
>
> I have an SGI XE 1300 cluster running Rocks 5 and RedHat,  Linux  
> version 2.6.18-53.1.14.el5
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list