[GE users] Array jobs and Usage Accounting.

isakrejda isakrejda at lbl.gov
Tue Mar 17 15:46:35 GMT 2009


HI,

I am running 6.0u11 on my production cluster and we noticed that users 
who run array jobs are grossly
overcharged for the running time of their jobs. Looks to me like each 
instance carries the load of all
running instances.

We are planning to upgrade in about 2 weeks and I already cloned my SGE 
configuration and installed
6.2u1, so we do not plan to fix 6.0u11, instead I proceeded to test the 
new install to make sure
array jobs are properly accounted for there.

The picture in 6.2u1 is for sure different but nevertheless disturbing. 
I picked 2 users
and submitted 3 identical jobs from one of them and 1 3-item array job 
for the other.
Here is the record:



I cleared the usage and querried the system:

pc2603 176% /sge/ge6.2u1/utilbin/lx24-x86/sge_share_mon -h -c 1 -t -u -x 
-f curr_time,node_name,user_name,usage,cpu,actual_share | egrep 
"user1|user2"|egrep "star|admin"
Mon Mar 16 18:04:47 2009        /admin/user1  user1      0.000000        
0.000000        0.000000
Mon Mar 16 18:04:47 2009        /star1/user2  user2      0.000000        
0.000000        0.000000

Submitted jobs. Both are running an identical executable that ads and 
subtracts 1 until it is killed.
The nodes are identical and each job runs on its own node.

pc2609 130% qstat -u user2
job-ID  prior   name       user         state submit/start at       
queue           slots ja-task-ID
------------------------------------------------------------------------------------------------------
3041504 338.62334 runit.csh  user2        r     03/16/2009 18:06:20 
test.q at pc2605            1      
3041506 338.62334 runit.csh  user2        r     03/16/2009 18:06:20 
test.q at pc2604            1      
3041507 338.62334 runit.csh  user2        r     03/16/2009 18:06:20 
test.q at pc2607            1      



pc2609 131% qstat -u  user1
job-ID  prior   name       user         state submit/start at     
queue             slots ja-task-ID
-----------------------------------------------------------------------------------------------------
3041505 10000.00000 runit.csh  user1     r     03/16/2009 18:06:20 
test.q at pc2609           1 1
3041505 10000.00000 runit.csh  user1     r     03/16/2009 18:06:20 
test.q at pc2606           1 2
3041505 10000.00000 runit.csh  user1     r     03/16/2009 18:06:20 
test.q at pc2608           1 3
pc2609 132%


Then I look at the usage

pc2603 182% /sge/ge6.2u1/utilbin/lx24-x86/sge_share_mon -h -c 1 -t -u -x 
-f curr_time,node_name,user_name,usage,cpu,actual_share | egrep 
"user1|user2" | egrep "star|admin"
Mon Mar 16 18:12:04 2009        /admin/user1  user1  1095.593609     
1095.593609     0.302032
Mon Mar 16 18:12:04 2009        /star1/user2  user2  2531.819393     
2531.819393     0.697968

and a few minutes later

pc2603 192% /sge/ge6.2u1/utilbin/lx24-x86/sge_share_mon -h -c 1 -t -u -x 
-f curr_time,node_name,user_name,usage,cpu,actual_share | egrep 
"user1|user2" | egrep "star|admin"
Mon Mar 16 18:15:31 2009        /admin/user1  user1  2027.948912     
2027.948912     0.299805
Mon Mar 16 18:15:31 2009        /star1/user2  user2  4736.275286     
4736.275286     0.700195

user2(normal jobs) was accumulating usage twice as fast as user1 (array)


I killed all the jobs and querried again...

pc2603 198% /sge/ge6.2u1/utilbin/lx24-x86/sge_share_mon -h -c 1 -t -u -x 
-f curr_time,node_name,user_name,usage,cpu,actual_share | egrep 
"user1|user2" | egrep "star|admin"
Mon Mar 16 18:18:31 2009        /admin/user1  user1  6831.272093     
6831.272093     0.516416
Mon Mar 16 18:18:31 2009        /star1/user2  user2  6396.951513     
6396.951513     0.483584

Now the usage is really close, maybe they were not killed at exactly 
same time. Jobs ran for 12 min, 3 for each user
and the CPU factor for each node is 3.06 (3x60x3.06x12=6609.6). Looks 
like user1 (array jobs) was underreporting
until it got killed and then the usage was evened out....


We want and need to use array jobs, but we also need and want fair share 
scheduling and
we have jobs running for long periods of time (24h) so we cannot afford 
such inaccuracies of reporting
for running jobs as it skews shares.

Thank You,

Iwona

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=134398

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list