[GE users] Functional policy in heterogeneous environment

Reuti reuti at staff.uni-marburg.de
Mon Oct 6 21:55:52 BST 2008


Hi Bon,

Am 06.10.2008 um 21:12 schrieb Robert Olson:

> We've recently added a new cluster here of 8-core intel CPUs; this  
> is going in the pool with the cluster of dual PPC G5 machines. I  
> want to put a department-based functional share to split the usage  
> between the two main users of the cluster (two bioinformatics  
> processing pipelines). Each slot in the new cluster is  
> significantly faster than the slots in the old cluster; is there a  
> way to make the functional policy take that into account? There are  
> roughly the same number of slots in each cluster, but it would be  
> unfair for a 50/50 share to have one app have the slow slots and  
> the other app have the fast slots.

you might want to defina a "usage_scaling" in the exechost  
definition, i.e. "qconf -me <hostname>" (man host_conf)

In case of parallel jobs it's seems also necesary to have them  
completely in one type of machines:

http://gridengine.info/2006/02/14/grouping-jobs-to-nodes-via-wildcard- 
pes

> I was planning on setting up a queue for each cluster

This is not necessary. You can also apply different sequence numbers  
for hosts or hostgroups in the queuedefintion, like also saving the  
above mentioned queues to separate parallel apps. Suppose you defined  
hostgroups @new and @old (or @xeon5450 and @g5):

$ qconf -sq all.q
...
seq_no                0,[@new=100],[@old=150]
...
pe_list               NONE,[@new=smp mpich1],[@old=smp mpich2]
...
slots                 1,[@new=8],[@old=2]
...

> , and set queue_sort_method to seqno with a lower number on the  
> queue with the fast nodes, so they're used first.

This is fine to fill first the new nodes.

-- Reuti

PS: Additional hint: If you have a directory structure of:

/opt/bioinformatics/lx24-amd64/bin/myapp
/opt/bioinformatics/darwin/bin/myapp

you can have a jobscript to run:

/opt/bioinformatics/$ARC/bin/myapp

and you will always get the right binaries while having only one  
jobscript - whereever you end up.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list