[GE users] Memory hard limit seems to be ignored on sge 5.3p5 version

Korambath, Prakashan ppk at ats.ucla.edu
Tue Sep 14 15:20:05 BST 2004


Thanks Shannon.  Setting h_vmem did set the limit and killed the job
through a segmentation fault error. Also as John pointed out, once the
job is started what ever change we make will have no affect on already
running jobs.  The changes are applied only for the new jobs. 

Prakashan



-----Original Message-----
From: Shannon V. Davidson [mailto:svdavidson at swbell.net] 
Sent: Monday, September 13, 2004 2:52 PM
To: users at gridengine.sunsource.net
Subject: Re: [GE users] Memory hard limit seems to be ignored on sge
5.3p5 version


I'm not sure why your job is not being killed, but keep in mind that
h_data gets set as a process limit.  If you have multiple processes in
your job and each process is under the limit, your job can still use
more 1.5GB of memory.  You might try using h_vmem, instead.  This is
implemented by Grid Engine as a job limit.  The execution daemon will
kill the job if the sum of the virtual memory being used by the
processes in the job exceeds the specified h_vmem limit.  For more
details about job limits, take a look at the queue_conf(5) man page
under RESOURCE LIMITS.  BTW, using h_vmem will only work on
architectures upon which the execution daemon dynamically collects
process usage, such as Linux, FreeBSD, Solaris, IRIX, UNICOS, Darwin,
and HP-UX.

Shannon


Korambath, Prakashan wrote:

Thanks Reuti.  Now the output looks like below.  The running job which
uses 1.9 GB is still continuing on the compute nodes.  

Prakashan



core file size        (blocks, -c) 1536000
data seg size         (kbytes, -d) 1536000
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7168
virtual memory        (kbytes, -v) unlimited

core file size        (blocks, -c) 1536000
data seg size         (kbytes, -d) 1536000
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7168
virtual memory        (kbytes, -v) unlimited

output of qconf -sq s04H command

s_rt                 INFINITY
h_rt                 INFINITY
s_cpu                INFINITY
h_cpu                INFINITY
s_fsize              INFINITY
h_fsize              INFINITY
s_data               1500M
h_data               1500M
s_stack              INFINITY
h_stack              INFINITY
s_core               INFINITY
h_core               1500M
s_rss                INFINITY
h_rss                INFINITY
s_vmem               INFINITY
h_vmem               INFINITY

Prakashan


-----Original Message-----
From:	Reuti [mailto:reuti at staff.uni-marburg.de]
Sent:	Mon 9/13/2004 11:27 AM
To:	users at gridengine.sunsource.net
Cc:	
Subject:	RE: [GE users] Memory hard limit seems to be ignored on
sge 5.3p5 version 
Please try 1500M in the queue definition instead of G. - Reuti

Quoting "Korambath, Prakashan" <ppk at ats.ucla.edu>:

  
Below is the output of what I am getting from the script
#!/bin/sh
ulimit -aH
echo
ulimit -aS

---------------------

core file size        (blocks, -c) 1048576
data seg size         (kbytes, -d) 1048576
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7168
virtual memory        (kbytes, -v) unlimited

core file size        (blocks, -c) 1048576
data seg size         (kbytes, -d) 1048576
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7168
virtual memory        (kbytes, -v) unlimited

ouput of the  command qconf -sq s04H on the node where it ran.

s_rt                 INFINITY
h_rt                 INFINITY
s_cpu                INFINITY
h_cpu                INFINITY
s_fsize              INFINITY
h_fsize              INFINITY
s_data               1.5G
h_data               1.5G
s_stack              INFINITY
h_stack              INFINITY
s_core               INFINITY
h_core               1.5G
s_rss                INFINITY
h_rss                INFINITY
s_vmem               INFINITY
h_vmem               INFINITY


Prakashan


-----Original Message-----
From:	Ron Chen [mailto:ron_chen_123 at yahoo.com]
Sent:	Mon 9/13/2004 10:43 AM
To:	users at gridengine.sunsource.net
Cc:	
Subject:	RE: [GE users] Memory hard limit seems to be ignored on
sge 5.3p5
version 
Can you run the job that prints the resource limits
again with the new setting?

 -Ron

--- "Korambath, Prakashan" <ppk at ats.ucla.edu> wrote:
    
I just set the soft limit s_data to 1.5G as well.  


s_rt                 INFINITY
h_rt                 INFINITY
s_cpu                INFINITY
h_cpu                INFINITY
s_fsize              INFINITY
h_fsize              1.5G
s_data               1.5G
h_data               1.5G
s_stack              INFINITY
h_stack              INFINITY
s_core               1.5G
h_core               1.5G
s_rss                INFINITY
h_rss                INFINITY
s_vmem               INFINITY
h_vmem               INFINITY

Prakashan


-----Original Message-----
From:	Rayson Ho [mailto:raysonho at eseenet.com]
Sent:	Mon 9/13/2004 11:04 AM
To:	users at gridengine.sunsource.net
Cc:	
Subject:	RE: [GE users] Memory hard limit seems to
be ignored on sge 5.3p5 version 
You have:

s_data               INFINITY
h_data               1.5G

However, in the shepherd code:

 /* hard limit must be greater or equal to soft
limit */
 if (rlp->rlim_max < rlp->rlim_cur)
    rlp->rlim_cur = rlp->rlim_max;

Can you set the s_data to 1.5G instead of
"INFINITY"??

Rayson



      
Thanks. The cluster is running Red Hat Linux 8.0
        
with 
2.4.20-30.8.legacysmp kernel.  The output of the
limit command is as
follows
      
cputime         unlimited
filesize        1048576 kbytes
datasize        1048576 kbytes
stacksize       unlimited
coredumpsize    1048576 kbytes
memoryuse       unlimited
vmemoryuse      unlimited
descriptors     1024
memorylocked    unlimited
maxproc         7168
        
---------------------------------------------------------
    
Get your FREE E-mail account at
http://www.eseenet.com !


      
---------------------------------------------------------------------
    
To unsubscribe, e-mail:
users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail:
users-help at gridengine.sunsource.net






      
---------------------------------------------------------------------
    
To unsubscribe, e-mail:
users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail:
users-help at gridengine.sunsource.net


      

		
__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net





---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net


    



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net





---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net


  



-- 
___________________________________________

Shannon V. Davidson <svdavidson at swbell.net>
Senior Software Engineer           Raytheon
636-479-7465 office        443-383-0331 fax
___________________________________________

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list