[GE users] h_vmem, virtual_free

Heywood, Todd heywood at cshl.edu
Fri Feb 23 20:04:57 GMT 2007


Thanks, Sean. That is helpful.

I am still left wondering why this vmatch program runs fine outside of
SGE with an unlimited stack size on a 2GB node, but requires "setting"
the stack size ulimit to 4G via h_vmem (or h_stack) on that 2GB node, in
order for it to run as an SGE job there. 

Aside: Trying "ulimit -Hs 4194304" in the submit script, rather than
using h_vmem=4G, gives: "ulimit: stack size: cannot modify limit:
Invalid argument", although that command successfully changes the stack
ulimit when run outside of SGE.

Todd

-----Original Message-----
From: Sean Dilda [mailto:sean at duke.edu] 
Sent: Friday, February 23, 2007 2:37 PM
To: users at gridengine.sunsource.net
Subject: Re: [GE users] h_vmem, virtual_free

Heywood, Todd wrote:
> Sorry, shot that off too soon. To answer my own question: because
h_vmem
> is set to "INFINITE" for my queues? If h_vmem is a queue attribute,
and
> my queues contain nodes of varying amounts of memory, this would mean
I
> can't use h_vmem for requesting memory.
> 

Yeah, I wouldn't recommend using h_vmem for that.

We use 'mem_free'.   First I set the mem_free complex to requestable and

consumable (I don't remember if it was already set that way or not). 
Then for each host I run this script:

MEMFREE=`qhost -F mem_total -h $1|tail -n 1|cut -d: -f3|sed -e 
s/total/free/`
qconf -mattr exechost complex_values $MEMFREE $1


Now, when a user requests '-l mem_free=2G' SGE will take the mem_free I 
set in the host config and subtract 2G from it.  It will also read the 
actual amount of memory free.  It will then use the lower of those two 
numbers when scheduling jobs.   That way a job that starts using memory 
slowly can still request its future amount of memory and SGE will act 
based off that instead of its temporarily low memory usage.


Also note that the load check for mem_free only looks at the amount of 
real memory that's free.  It doesn't look at swap like virtual_free
does.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list