[GE users] OpenMPI RLIMIT_MEMLOCK problem

Prentice Bisbal prentice at ias.edu
Thu Dec 4 14:34:56 GMT 2008


Harald,

Thanks for the response. As you may have figured out, "ulimited" in my
original post was a typo, I was actually using "unlimited" in my config.

I did change the value to "infinity" as you advised, and all is working
now.

Is the use of the value "infinity" documented anywhere? If so, I
couldn't find it. I was working from documentation in the Grid Engine
website (see original posting) which specifically said to use
"unlimited". I searched the documentation, and couldn't find any
coverage of execd_params.

--
Prentice


pollinger wrote:
> Hi Prentice,
> 
> "ulimited" is not supported, please use "infinity" instead.
> 
> The configuration parser interprets "ulimited" as "0".
> 
> Regards,
> Harald
> 
> Prentice Bisbal wrote:
>> I'm using SGE 6.2 with OpenMPI 1.2.8.
>>
>> I just setup OpenMPI tight integration following the instructions here:
>>
>> http://www.open-mpi.org/faq/?category=running#run-n1ge-or-sge
>>
>> I then defined my exec_params to set H_MEMLOCK=unlimited as described here:
>>
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=72405
>>
>> Unfortunately, when I submit an MPI job, I still get MEMLOCK errors:
>>
>> libibverbs: Warning: RLIMIT_MEMLOCK is 11162 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 11199 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 11181 bytes.
>>     This will severely limit memory registrations.
>>
>> To make sure my parameters were took effect, I even stopped and
>> restarted sge_execd on every compute node with no luck.
>>
>> Any ideas? Relevant configuration information is below. Please note that
>> in qconf I've tried
>>
>> 1. execd_params H_MEMORYLOCKED=ulimited
>> 2. execd_params S_MEMORYLOCKED=ulimited
>> 3. execd_params S_MEMORYLOCKED=ulimited H_MEMORYLOCKED=ulimited
>>
>> All 3 had the same result.
>>
>> Let me know if additional/complete config information would be helpful.
>> I don't want to flood the list with unnecessary config information.
>>
>> # qconf -sp orte
>> pe_name            orte
>> slots              512
>> user_lists         NONE
>> xuser_lists        NONE
>> start_proc_args    /bin/true
>> stop_proc_args     /bin/true
>> allocation_rule    $fill_up
>> control_slaves     TRUE
>> job_is_first_task  FALSE
>> urgency_slots      min
>> accounting_summary FALSE
>>
>> # qconf -sq all.q  | grep pe_list
>> pe_list               make orte
>>
>> # qconf -sconf | grep execd_params
>> execd_params                 H_MEMORYLOCKED=unlimited
>> S_MEMORYLOCKED=unlimited
>>
>> $ more xhpl.sh
>> #$ -S /bin/bash
>> #$ -N xhpl
>> #$ -pe orte 512
>> #$ -cwd
>> #$ -V
>>
>> MPI=/usr/local/openmpi/gcc-4.1.2/x86_64/
>> PATH=${MPI}/bin:$i{PATH}
>> LD_LIBRARY_PATH=${MPI}/lib
>>
>> mpirun ./xhpl
> 
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=91143

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list