[GE users] Task scheduling

Toby Sargeant sargeant at wehi.edu.au
Tue Feb 15 01:56:21 GMT 2005


Is it possible to configure SGE to execute the tasks or two or
more jobs submitted to the same queue in an interleaved fashion?

The situation I'm trying to cope with with is this: the majority
of jobs that our cluster runs are long running array jobs with
large numbers of tasks in which each individual task is short
lived. In the case where we have two or more such jobs ready to
run, it is best for us if they both execute simultaneously.

At the moment our solution is to define 3 queues: all.q which
covers the whole cluster, and upper.q and lower.q which cover
half each. A job is initially started on all.q, and if a second
job needs to run, it is migrated to one of lower.q or upper.q and
the new job is started on the other.

This has a few problems:

1) If the queue isn't specified when a job is started, it
   executes tasks in all queues, which seems to cause 2 tasks to
   execute per CPU (although I'm not sure why this happens).

2) Migrating the longer running job back to all.q after the other
   job finishes has to be done by hand, and until it's done we
   waste half of our CPUs.

3) While we've only had two jobs at once thus far as soon as we
   have 3 or more this solution doesn't work.

I've looked through the documentation, including the 5.3
users/administrators guide and mailing list archives, but haven't
found anything.

Any help or pointers would be much appreciated.

Toby Sargeant.
Division of Genetics and Bioinformatics
The Walter and Eliza Hall Institute of Medical Research
Parkville 3050 VIC, Australia
Ph: +613 9345 2324   Fax: +613 9347 0852   Web: http://www.wehi.edu.au/

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list