[GE users] Resource allocation question

Robert Olson olson at mcs.anl.gov
Wed Dec 12 17:52:22 GMT 2007

On Dec 12, 2007, at 11:36 AM, Robert Olson wrote:

> One issue that I don't yet know how to solve is the following: The  
> jobs in the application are pipelines with a number of stages. the  
> pipelines are driven by a periodically run pipeline manager that  
> updates the status of each job, submitting new tasks into the  
> cluster as previous tasks finish. A characteristic of the current  
> setup is that if a lot of jobs are in flight, a later stage of a  
> particular user job may end up in the scheduler queue behind early  
> stages of newer jobs. In some sense I'd like the next stage of a  
> job to get in line ahead of earlier stages of newer jobs. Hm -  
> maybe that is part of the solution - assign an explicit priority to  
> each stage of the pipeline, and factor that priority into the  
> scheduling within a project..

It occurs to me that perhaps I can use deadline scheduling for this;  
the user job submission code can estimate a completion time to be  
submitted as the deadline.


More information about the gridengine-users mailing list