[GE users] Scheduling Questions

Charu Chaubal Charu.Chaubal at Sun.COM
Thu Dec 15 17:50:35 GMT 2005


    [ The following text is in the "windows-1252" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hi Raymond,



Raymond Chan wrote On 12/15/05 09:02,:
> Hi all,
> 
>  
> 
> I know this has been asked before in some way or another, and I?m sorry
> if this is a repeat.  I?m not sure where to begin because as I look
> through the list there are things that may or may not apply, so I?ll be
> more direct and just ask.  Thanks to anyone in advance who can help, and
> again I apologize if this was answered recently:
> 
>  
> 
> I have a total of 6 dual cpu nodes that I am submitting parallel mpich &
> pvm jobs (qsub ?pe 6) as well as regular non-parallel jobs to.   As you
> know, some jobs take longer than others to complete, so if job #1 is a
> job that will take 5 hours to complete, and job #2 will take only 5
> minutes, is there a way to automatically in SGE make certain jobs jump
> in queue over jobs that are running a long time so these smaller jobs
> can finish (sort of in a round robin sort of way where each job maybe
> gets a certain amount of time before switching)?  More clearly, a large
> job #1 holds up the queue and everyone behind it is stuck, so what?s the
> best approach at solving this sort of scheduling problem?  
> 

This depends on your application's system utilization profile.  If it's
allowable for long-running and short-running jobs to run simultaneously
on the same processor for a temporary time, then you could simply have
two clusters queues, "short" and "long", on all hosts, and make sure
jobs are submitted with the appropriate specification.  Short jobs would
simply run alongside long jobs until they complete.

If your application allows for suspension of the long running jobs, then
you could make the "long" queue a subordinate of the "short" queue.
However, parallel jobs are usually not amenable to suspension, at least
not in a clean way.  If they are checkpointable, though, then you can
configure queues so that, instead of being suspended, the long jobs get
checkpointed and resubmitted to run somewhere else.

Finally, if you do not want long and short jobs to occupy the same
processor in any situation, then you basically have to dedicate some
percentage of hosts to run only short-running jobs.  A number of sites
that I've seen use this policy.  With GE, you can configure this using a
special hostgroup for short-running jobs --- this at least gives you an
easy way to change the allocation of hosts in a single place.  If you
want to get *really* fancy, you can even have a simple cron job which
automatically does this, depending upon the number of long jobs pending.

Regards,
	Charu


>  
> 
> I?m trying to run a department website at a university where professors
> will be able to submit SGE jobs via the web for bioinformatics apps that
> can take a long time.  It?d be nice if I had a good way of allowing each
> professor to get a fair share of the cluster instead of one guy holding
> up the whole thing.  Would it also be possible to allow SGE to run two
> jobs at once on the queue rather than one (or would this not be a good
> or possible approach)?
> 
>  
> 
> Thanks again to anyone who can offer advice w/ these newbie questions.
>  I?ve currently just been able to do jobs one at a time nicely on my web
> system with everyone waiting behind that one job to finish, but I need a
> nice scheduling method, and I have no idea how to configure this in SGE.
> 
>  
> 
> -Ray
> 
>  
> 
>  
> 

-- 
####################################################################
# Charu V. Chaubal              # Phone: (650) 786-7672 (x87672)   #
# Grid Computing Technologist   # Fax:   (650) 786-4591            #
# Sun Microsystems, Inc.        # Email: charu.chaubal at sun.com     #
####################################################################


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list