[GE users] h_vmem limits in 6.2

abrookfield a.brookfield at sheffield.ac.uk
Thu Jun 4 10:59:01 BST 2009


Hi,

I'm trying to setup a queue which will kill jobs which exceed the physical memory in my nodes.  

The configuration for the queue looks like:
s_rt                  INFINITY
h_rt                  168:0:0
s_cpu                 INFINITY
h_cpu                 INFINITY
s_fsize               INFINITY
h_fsize               INFINITY
s_data                INFINITY
h_data                INFINITY
s_stack               INFINITY
h_stack               INFINITY
s_core                INFINITY
h_core                INFINITY
s_rss                 INFINITY
h_rss                 INFINITY
s_vmem                INFINITY
h_vmem                16G

However if I run an interactive session in this queue, ulimit -a reports:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 16777216
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 266240
max locked memory       (kbytes, -l) 33554432
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 16777216
cpu time               (seconds, -t) unlimited
max user processes              (-u) 500
virtual memory          (kbytes, -v) 16777216
file locks                      (-x) unlimited

So SGE has placed additional limits on the stack size and the data seg size.

(Running ulimit -a with:
h_vmem                INFINITY 
gives:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 266240
max locked memory       (kbytes, -l) 33554432
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 500
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited)

This may be a sensible thing to do, but setting limits on both virtual memory and stack size causes various applications to crash on startup... whilst if I manually use ulimit -v to set a limit on the virtual memory size everything runs fine.

e.g. IDL:
% Internal threading error: Unable to create thread (IDL_ThreadCreate()
  calling pthread_create()).
  Cannot allocate memory

and MATLAB:
getrlimit(RLIMIT_STACK, {rlim_cur=16777216*1024, rlim_max=16777216*1024}) = 0
mmap(NULL, 17179873280, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = -1 ENOMEM (Cannot allocate memory)

Is there any way to get SGE to just set the virtual memory limit, rather than setting the stack size as well?

Thanks,

Anthony.

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=200836

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list