[GE users] Parallel job starvation

Reuti reuti at staff.uni-marburg.de
Fri Dec 21 22:21:07 GMT 2007


Hi,

Am 21.12.2007 um 17:07 schrieb Daire Byrne:

> We have a workload that includes equal amounts of single slot/cpu  
> jobs and multi-threaded (>1 slot) jobs. Now I'm aware that  
> "reservation" is designed to ensure that multi-cpu jobs can still  
> get enough slots to run but it seems to me this only works  
> efficiently if you know the approximate run times of jobs. It is  
> almost impossible in our environment to make accurate predictions  
> of run times so I fear that reservation will either leave slots  
> free for longer than necessary (how can you backfill if you don't  
> know what job can "fit" in) or you end up waiting for longer than  
> you thought for "reserved" slots on a host to free up.
>
> In such an environment I'm thinking it is better to split the  
> cluster into distinct "no. of slot" groups (using Qs?). So I would  
> dispatch my 4 thread jobs only to machines which do 4 thread jobs.  
> This way I am essentially turning a 4 thread job into a single slot  
> job on these machines. When a job finishes it frees up exactly the  
> 4 slots required for the next 4 thread/cpu job. Does anyone have  
> any experience with this kind of setup? It seems that dynamically  
> altering the number hosts accepting single thread or 4 thread jobs  
> would be very difficult to manage but it may be more efficient in  
> an environment where you can't predict the run times? We only  
> currently run jobs that have 1, 2 and 4 x threads so it would  
> potentially mean splitting the cluster into 3 distinct Q's.
>
> It would be nice to have a hybrid solution - prefer the 4-thread  
> machines/Q but if not available do a reservation elsewhere.

but this would mean a different setup for your applications. I mean,  
a PE with $pe_slots as allocation rule can and will only put all  
slots on one and the same machine. So also without any chance of  
backfilling, you will get the next free node. If you collect slots  
from more than one machine, you can't use threads in an OpenMP style  
way anyway.

If you have 4-thread machines, putting there 2-threads applications  
should still leave two slots free. You don't need three queues for  
it, as the slots can be configured differently in one queue per node  
or hostgroup. Using a sequence number for the machines/hostgroups,  
you could also implement your wish to prefer nodes with 4 cores.

If you have jobs which uses more than one thread, the best is IMO  
still to tell this to SGE by submitting a parallel job. But there are  
other opinions in the mailing list archive - personal taste ;-)

-- Reuti

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list