[GE dev] Core binding suggestion...

dagru daniel.x.gruber at oracle.com
Thu Oct 7 08:19:51 BST 2010


When using "pe" (adding final column into the hostfile), following 
assumption is behind the (execution daemon/shepherd) decision:
- all hosts have the same architecture
- the exclusive host feature is used
- the amount of slots on each host is equal

This is because the final decision (transforming linear:1 into
the core,socket number is done on the execution host). In case
of "pe" just one execution host is involved.

In other words: The decision is made on the master tasks hosts
and is simply the same for each client. This could be a drawback.

When using "-binding linear:1" for such an 8 way tightly parallel job
all tasks are bound automatically. *Each* execution should get the more 
abstract "linear:1" information which is then transformed on all hosts
into the right core (in case another core is already bound or
the architecture is different). In other words *each* "qrsh -inherit"
call (which is done internal in case of tight integration) should get
a unique core. The exclusive host feature therefore is not needed
and the same architecture requirement should be relaxed (AFAIK).

The more interesting thing in this case ("-binding linear:1") is the
correct handling of 2 slots on the same host. AFAIK when
doing two "qrsh -inherit" calls (for each slot one) the binding
should be correct since two times the original request ("binding 
linear:1") was requested internally on the execution host.
Therefore two cores on the exec3 should be bound. In case of just one
"qrsh -inherit" call just one core is used.

Cheers

Daniel



On 10/06/10 17:42, jewellc wrote:
> On 6 Oct 2010, at 15:50, Daniel Templeton wrote:
> 
>> I forgot to mention that you can use "-binding pe linear:1", which tells OGE not to do the processor binding itself, but instead to pass along the info to the job's MPI via env vars.
> 
> 
> Interestingly, I see that when I a "qsub -pe mpi 8 -binding pe linear:1  myScript.com" with my mpi parallel environment (alloc rule $round_robin), my pe_hostfile looks like this:
> 
> exec3 2 batch.q at exec3.cluster.stats.local 0,0
> exec2 1 batch.q at exec2.cluster.stats.local 0,0
> exec7 1 batch.q at exec7.cluster.stats.local 0,0
> exec5 1 batch.q at exec5.cluster.stats.local 0,0
> exec1 1 batch.q at exec1.cluster.stats.local 0,0
> exec4 1 batch.q at exec4.cluster.stats.local 0,0
> exec6 1 batch.q at exec6.cluster.stats.local 0,0
> 
> In other words, for my 8 processor MPI job, exec3 is assigned 2 slots, but if my MPI subsystem (OpenMPI, FYI) were aware of the suggested binding, it might result in both processes being bound to socket 0, core 0.  Is this expected behaviour?  Shouldn't the binding on exec3 be "0,0:0,1" ?
> 
> Chris
> 
> (PS Is it useful to be asking this on the dev list, or shall I jump to the user list?)
> --
> Dr Chris Jewell
> Department of Statistics
> University of Warwick
> Coventry
> CV4 7AL
> UK
> Tel: +44 (0)24 7615 0778
> 
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=39&dsMessageId=286200
> 
> To unsubscribe from this discussion, e-mail: [dev-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=39&dsMessageId=286388

To unsubscribe from this discussion, e-mail: [dev-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list