[GE users] job distribution in every execution host issue

Daniel Templeton Dan.Templeton at Sun.COM
Thu Aug 16 19:59:41 BST 2007


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

And I would go back and look at your scheduler settings, as was 
suggested earlier.  If you accepted a default scheduler interval of 15 
seconds, it's going to be a little pokey.  I usually set the scheduler 
interval to 2 minutes and the flush_submit_sec and flush_finish_sec to a 
couple of seconds.  That is, by the way, what the Max scheduler 
installation setting sets.  (Max also turns off some messages, so I 
usually choose Normal and just twiddle the scheduler settings.)

The Grid Engine scheduler is actually very fast.  In a lightly loaded 
system, the scheduler should be able to complete a run in much less than 
100ms.  You're probably running into configuration issues.  Something 
else to consider is that a lot of the overhead for "scheduling" a job is 
actually in transferring the script and forking and execing the shell.  
Individual jobs in your grid may be faster when they are all routed to a 
single machine because that machine is the fastest at starting your jobs.

Daniel

John Hearns wrote:
> On Tue, 2007-08-14 at 01:02 +0800, Benson Fung wrote:
>   
>> Hi,
>>
>> We have setup a grid engine infrastructure with 4 execution hosts in
>> place.  And we have submitted 200 jobs into the grid engine.  
>>     
>
> I back up what Reuti says.
> I would also look at the I/O performance of the jobs.
> Do your jobs read a lot of data from an NFS shared drive on the master
> node of the cluster?
>
> I would start simple - arrange one job slot per host and time your jobs,
> running on the master node and on the execution hosts.
> Then two slots etc.   You may find that jobs are being starved of I/O
> due to a bottleneck on the master node.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list