[GE users] Per-node group priorities?

craffi dag at sonsorol.org
Thu Dec 11 00:10:32 GMT 2008


On Dec 10, 2008, at 5:20 PM, Jim Zajkowski wrote:

> On 12/10/08 5:14 PM, "craffi" <dag at sonsorol.org> wrote:
>
>> On Dec 10, 2008, at 3:12 PM, Jim Zajkowski wrote:
>>
>>> Is it possible to configure SGE so that jobs submitted by members of
>>> a group have a preference for a specific set of nodes?

It is possible to to do this but it sounds like you may be more  
interested in letting the 2 PIs use any server in the cluster as long  
as the total allocation mix is 50-50 when there is activity from both  
groups.

>>>
>
>> - Do you have a specific set of 20 machines you want to use for these
>> 2 groups or do you want to pull any 20 nodes out of a much larger  
>> farm?
>
> I am indifferent - we simply ordered 20 identical machines (Sun Blade
> x6250), and they're all racked together and share the same SAN, etc.
>
>> - Would any other users or groups active on these 20 servers?
>
> No.
>

Then it sounds like a simple Department or Project based functional or  
sharetree policy with a simple 50% entitlement spread among users from  
the two labs would work fine. Resource quotas can further constrain  
things or can be used as an emergency threshold to prevent one PI from  
grabbing 100% of all available slots on an idle cluster.

The big thing to note (as I think you already picked up) is that SGE  
by default is not going to mess with running jobs. All SGE policies  
are enforced by the SGE scheduler constantly adjusting the jobs  
waiting in the pending list. This means that one of your PIs could  
swamp the cluster for a bit (getting a 100% mix) and you would go down  
to the desired 50-50 mix as active jobs drain out. Since you mentioned  
having fast running jobs this is likely not going to be a big deal for  
you.

I wrote up a blog post covering the difference between Share Tree and  
Functional policies. The main difference is if you want your 50-50 mix  
to be calculated via the current cluster usage (functional policy) or  
if you want the scheduler to look back over a period of time to  
account for prior usage when doing calculations (share tree)

The url is here:
http://gridengine.info/2005/09/30/pretty-pictures-explain-functional-vs-sharetree-scheduling

My experience suggest that while Share Trees are attractive to an  
administrator and operator they tend to be viewed suspiciously by end- 
users. When your PI comes to you demanding to know why they are not  
getting their 10 node allocation it's hard to go back and "prove" to  
them that the scheduler is just adjusting a little bit for past usage  
in which they benefited from much more than their official allocation.


>>> the other being that the tools being run already expect an SGE  
>>> environment.
>
>> Out of curiosity, what are the tools?
>
> One of the PIs has brought along the scripts and procedures from his
> previous location.  They are biologists and not computer scientists,  
> I'd
> prefer little disruption as possible.  That being said I'm open to  
> non-SGE
> solutions.
>

Clustering for Biology/life-science is my primary area of  
concentration as well. SGE is quite popular with our crowd.


> Thanks,
>

-Chris

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=92134

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list