[GE users] question about job queue interval

Stephan Grell - Sun Germany - SSG - Software Engineer stephan.grell at sun.com
Thu Apr 21 19:44:06 BST 2005


    [ The following text is in the "windows-1252" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hi,

I added some comments.

Chris Duke wrote:

> I have a question about the job queue interval. I apologize in advance 
> if the terms I use are not exactly correct, I?m fairly new to this. 
> I?m using SGE 5.3p6. We use SGE for a regression test. We run about 
> 2000 small jobs in a full regression test. The average test is about 
> 10-20 seconds (wall clock).
>
> What I?ve noticed is that the queuing process on the master seems to 
> run about every 15 seconds. What this occasionally causes is long gaps 
> when all the queues are empty, and they don?t get populated with new 
> jobs for several seconds. This delay is not a huge problem, but when 
> you calculate the ratio of the delay to the execution time of the job, 
> it is rather large. By reducing that delay, I am hoping to finish the 
> regression test run much more quickly. I realize that there would be 
> additional overhead associated with shorter intervals (because of 
> increased polling), but I would like to be able to experiment and 
> optimize.
>
> Solutions that seem like they would work, but I don?t know how to 
> manipulate the system to do:
>
>     * Control the poll interval of the master and all execute hosts.
>       Reduce and optimize.
>
there is no poll interval in the grid engine. The qmaster triggers the 
scheduler every scheduling interval. This can be set via qconf -msconf.
if the scheduler is busy, it gets not triggered. In the worst case, it 
can lead to 100% cpu utilization by the scheduler. You might want to
take a look into the performance tuning howto on the open source side.

>     * Our typical execute host has 2 CPU?s. Control the master so that
>       it might put 3 jobs into a particular execute hosts queue, but
>       the execute host would only be allowed to execute 2 jobs at a
>       time. That way the execute host always has something to do,
>       instead of periods of idle time.
>
you can set the number ob slots per queue. SGE does not care for the 
CPUs on a machine, but for the slots. Though that allows you to controll the
number of jobs on the hosts.

Stephan

> Thanks in advance for any help and ideas.
>
> --chris
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list