[GE users] Spooling benchmarks

jlb jlb at salilab.org
Thu Apr 15 00:50:14 BST 2010


On Fri, 9 Apr 2010 at 8:41pm, rayson wrote

> On 4/9/10, jlb <jlb at salilab.org> wrote:
>>  o I find the total throughput to be pretty low given the number of
>>    slots involved -- should it be higher?  Any hints as to where to look
>>    to improve things?
>
> How busy was the qmaster during the test?? Did it spend most of the
> time waiting for I/O or was it CPU bound??
>
> And did you play with any scheduler tuning parameters??
> "Scheduling-on-demand" should have an effect on the throughput if the
> qmaster machine can handle it:
>
> http://gridengine.sunsource.net/howto/tuning.html

It seems to me to be CPU bound.  All my initial testing was done with the 
following scheduler properties (which reflect the settings of my 
production cluster):

schedule_interval                 0:0:15
flush_submit_sec                  0
flush_finish_sec                  0

Doing a run of Test 2 (submitting and running simultaneously) with 32768 
jobs leads to the following graph of CPU usage:

http://www.duke.edu/~jlb17/subrun1500.pdf

(Note that, in the graph, 100% is 100% usage of a single core -- the 
server has 8 real cores and has hyperthreading on so that the OS sees 16).

I played around with all three parameters above and say only small 
throughput improvements.  The (roughly) best was simply:

schedule_interval                 0:0:15
flush_submit_sec                  1
flush_finish_sec                  1

This ran in 878 seconds (vs. 1076 seconds for the initial settings) and 
showed the following CPU usage:

http://www.duke.edu/~jlb17/subrun1511.pdf

So SGE is almost maxing out 2 cores.

-- 
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=253451

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list