[GE users] multiplied resource limits with pe

SLIM H.A. h.a.slim at durham.ac.uk
Thu Aug 24 19:24:34 BST 2006


I am puzzled by a similar problem with a consumable resource, a 800GB
disk attached to a 8 way node.
This is the setup:

qconf -sc = the consumable:
#name               shortcut       type        relop requestable
consumable default  urgency 
#-----------------------------------------------------------------------
---------------------
bigtmpoctet        bigtmpoctet    MEMORY      <=    FORCED      YES
0G       0

qconf -sq octet.q = the queue
qname                 octet.q
hostlist              @octets
...
pe_list               mpich openmp
...
complex_values        bigtmpoctet=800G
...

qconf -sp openmp = PE to use
pe_name           openmp
slots             16
...

Now if I submit a job requesting 100G and 4 slots from the openmp PE,

qsub -q octet.q -pe openmp 4 -l bigtmpoctet=100G script

it runs but when checking the amount of free consumable diskspace before
and after start of the job I get:

qstat -q octet.q -F bigtmpoctet = before
queuename                      qtype used/tot. load_avg arch
states
------------------------------------------------------------------------
----
octet.q at octet01.hamilton.dur.a BP    8/16      5.12     lx26-amd64    
        hc:bigtmpoctet=792.549G

qstat -q octet.q -F bigtmpoctet = after
queuename                      qtype used/tot. load_avg arch
states
------------------------------------------------------------------------
----
octet.q at octet01.hamilton.dur.a BP    12/16     4.87     lx26-amd64    
        hc:bigtmpoctet=392.549G

The consumable has been decreased by 4 x 100GB, which is not right, so I
advise users to divide the actual amount of disk they want by the number
of slots (usually 4 as in this case) to get the correct bookkeeping.

Any suggestions what parameters I should change to avoid the fiddling
with division by number of slots?

Thanks

Henk

> -----Original Message-----
> From: Reuti [mailto:reuti at staff.uni-marburg.de] 
> Sent: 24 August 2006 17:27
> To: users at gridengine.sunsource.net
> Subject: Re: [GE users] multiplied resource limits with pe
> 
> Am 24.08.2006 um 17:15 schrieb Gerd Marquardt:
> 
> > Hello,
> > we have a cluster with 4-way-nodes.
> > Running a simple Job which use a parallel environment and restrict 
> > some resource limits, the limits are multiplied.
> > For example the Job:
> > #$ -pe mpi 4
> > #$ -l h_vmem=128M
> > #$ -l h_stack=32M
> > echo ulimit -d
> > ulimit -d
> > echo ulimit -s
> > ulimit -s
> >
> > Produce this output:
> > ulimit -d
> > 524288
> > ulimit -s
> > 131072
> >
> > The limits are multiplied by 4.
> > How can I avoid this behavior?
> >
> > We have Red Hat EL4, we have defined a pe for our shared memory 
> > programs ( OpenMP). Here also a defined time limit (h_cpu) is 
> > multiplied. The limit have no affect on the process but on the 
> > threads. So mostly the process run 4 times too long.
> 
> AFAIK:
> 
> Using Forks:
> 
> You are right, that each fork will get the multiplied limit, 
> but SGE will take care of the accumulated consumption and 
> kill the job.
> 
> Using Threads:
> 
> All CPU time is accumulated by the main thread, so the SGE 
> and kernel limits should be the same, and the job will also 
> be killed. This you could test by running the program 
> interactively and watch in top the used up CPU time, which 
> should run faster than your watch.
> 
> -- Reuti 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list