[GE users] OpenMPI RLIMIT_MEMLOCK problem

Prentice Bisbal prentice at ias.edu
Wed Dec 3 20:55:08 GMT 2008


I'm using SGE 6.2 with OpenMPI 1.2.8.

I just setup OpenMPI tight integration following the instructions here:

http://www.open-mpi.org/faq/?category=running#run-n1ge-or-sge

I then defined my exec_params to set H_MEMLOCK=unlimited as described here:

http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=72405

Unfortunately, when I submit an MPI job, I still get MEMLOCK errors:

libibverbs: Warning: RLIMIT_MEMLOCK is 11162 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 11199 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 11181 bytes.
    This will severely limit memory registrations.

To make sure my parameters were took effect, I even stopped and
restarted sge_execd on every compute node with no luck.

Any ideas? Relevant configuration information is below. Please note that
in qconf I've tried

1. execd_params H_MEMORYLOCKED=ulimited
2. execd_params S_MEMORYLOCKED=ulimited
3. execd_params S_MEMORYLOCKED=ulimited H_MEMORYLOCKED=ulimited

All 3 had the same result.

Let me know if additional/complete config information would be helpful.
I don't want to flood the list with unnecessary config information.

# qconf -sp orte
pe_name            orte
slots              512
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $fill_up
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE

# qconf -sq all.q  | grep pe_list
pe_list               make orte

# qconf -sconf | grep execd_params
execd_params                 H_MEMORYLOCKED=unlimited
S_MEMORYLOCKED=unlimited

$ more xhpl.sh
#$ -S /bin/bash
#$ -N xhpl
#$ -pe orte 512
#$ -cwd
#$ -V

MPI=/usr/local/openmpi/gcc-4.1.2/x86_64/
PATH=${MPI}/bin:$i{PATH}
LD_LIBRARY_PATH=${MPI}/lib

mpirun ./xhpl


-- 
Prentice

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=90952

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list