[GE users] Re: array job accounting or scheduling problem

Andreas.Haas at Sun.COM Andreas.Haas at Sun.COM
Mon Jul 2 16:28:49 BST 2007


Hi Pascal,

after running

---> 100 times: qsub -P A -b y /bin/sleep 5
---> 100 times: qsub -P B -b y /bin/sleep 5
---> 1 times: qsub -t 1-100 -P C -b y /bin/sleep 5

with SHARETREE_RESERVED_USAGE=true be set in global cluster 
configuration sge_conf(5) I get a combined resource usage 
which sometimes is really surprisingly unbalanced. I played 
around with different arrangements to get a clue on this 
phenomenon:

Project| Comb. Usage  | Sum. Acct. Usage
-------------------------------------------
A      | 1136.78      | 1085
B      | 1161.77      | 1100
C      | 1292.73      | 1189        (array)
-------------------------------------------
A      | 1294.78      | 1222        (array)
B      | 1159.82      | 1080
C      | 1154.82      | 1097
-------------------------------------------
A      | 1052.86      |  997
B      | 1047.86      |  991
C      | 1224.82      | 1137        (array)
-------------------------------------------
A      |  782.36      |  655        (array)
B      |  646.80      |  590
C      |  645.80      |  586
-------------------------------------------
A      |  635.88      |  568
B      |  634.88      |  570
C      |  647.88      |  569        (array)
-------------------------------------------
A      |  700.77      |  640        (array)
B      |  697.77      |  633
C      |  670.77      |  605
-------------------------------------------
A      |  656.83      |  585        (array)
B      |  629.84      |  570
C      |  640.84      |  581
-------------------------------------------

this shows the accounted usage of array jobs is constantly higher 
than of of sequential jobs! I investigated it to the 
point that I can say: For some mysterious reason array tasks on 
average take longer time from fork() until /bin/sleep actually
starts. Interestingly use of "-shell no" submit option finally 
gave me a very much balanced distribution of per project accounting,
but I still can not explain why array jobs should be affected from
the overhead than sequential jobs ... :-o

With regards to the share tree behaviour I recommend use of a by 
far lower compensation factor than the default. With the compensation
factor one can control how much projects with higher usage shall
be penalized. When I used a 1 or 2 as compensation factor I got
quite good results despite of the unbalanced accounting.

HTH,
Andreas

On Fri, 29 Jun 2007, Pascal Wassam wrote:

> Hello Andreas,
>
> These are all of the weight_* values for configurations where I have seen 
> this problem.
> (I also updated these in the ticket)
>
> ----------------------------------------------------
> weight_ticket                     1.000000
> weight_waiting_time               0.000000
> weight_deadline                   0.000000
> weight_urgency                    0.000000
> weight_priority                   0.000000
> ----------------------------------------------------
> weight_ticket                     0.900000
> weight_waiting_time               0.000000
> weight_deadline                   0.000000
> weight_urgency                    0.000000
> weight_priority                   0.100000
> ----------------------------------------------------
> weight_ticket                     0.010000
> weight_waiting_time               0.000000
> weight_deadline                   3600000.000000
> weight_urgency                    0.100000
> weight_priority                   1.000000
> ----------------------------------------------------
>
> I haven't played around with these values, I will do so on my test rig and 
> see if I can change the outcome.
>
> As a side note, a totally unfounded guess as to something this could be:
>
> Each slot that is running an array job member has its usage counted as the 
> sum of usage of all members of that array,
> rather then the usage of that specific subarray job.
>
> The strangest thing about all this to me is that though the measures I'm 
> looking at, some part of sge thinks that the user running these array jobs is 
> all
> of a sudden using 80% of the cluster, when the reality is they're using more 
> like 5% of it. This persists immediately after a -clearusage.
>
> Thanks for looking into this,
> -Pascal
>
> andreas at sunsource.net wrote:
>> http://gridengine.sunsource.net/issues/show_bug.cgi?id=2298
>> 
>> 
>> 
>> 
>> 
>> 
>> ------- Additional comments from andreas at sunsource.net Fri Jun 29 02:22:39 
>> -0700 2007 -------
>> What are you using for
>>
>>    weight_urgency
>>    weight_ticket
>>    weight_waiting_time
>> 
>> in sched_conf(5). If your waiting time weight is non-zero this could
>> cause the phenomenon you observe. Reason is that waiting time contributes
>> to job urgency and urgency has higher weight than the ticket policy.
>>

http://gridengine.info/

Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten
Amtsgericht Muenchen: HRB 161028
Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
Vorsitzender des Aufsichtsrates: Martin Haering

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list