[GE users] Parallel environments for multithreaded programs

Reuti reuti at staff.uni-marburg.de
Mon Jun 30 08:11:18 BST 2008


Am 30.06.2008 um 00:42 schrieb Sean Davis:

> On Sun, Jun 29, 2008 at 4:07 PM, Reuti <reuti at staff.uni-marburg.de>  
> wrote:
>> Hi,
>> Am 29.06.2008 um 21:13 schrieb Sean Davis:
>>> I have been asking quite a few questions recently and have been
>>> learning a ton.  I have yet another one.
>>> I have three machines, each being an SMP machine.  I would like  
>>> to set
>>> up a parallel environment (using OpenMPI 1.2.6) for submission of
>> Open MPI will also support communication between nodes. Do you  
>> mean OpenMP
>> instead?
> Thanks, Reuti, for the reply.  I meant OpenMPI.  I have it installed
> and it runs fine for parallel jobs.
>>> multithreaded programs to run on a single machine using a set number
>>> of threads.  Each machine actually has a different number of  
>>> physical
>>> processors.  I see how to use $pe_slots to fill each machine in  
>>> turn,
>>> but is there a way to specify that no more than the number of
>>> available slots on an available single machine get allocated for  
>>> a job
>>> so that the job runs on a single machine only?
>> If you allow your job to run with other jobs on a node: you could  
>> specify a
>> range of slots in qsub (and the PEs have $pe_slots).
> So, I would set up a different PE for each machine, each set to
> $pe_slots?

yes/no. In principle one PE is enough for just this setup to get all  
slots from one machine by setting $pe_slots. $pe_slots implies, that  
it stays on one and the same host and don't collect slots from other  

> Then, submitting would be to a given PE, each of which is
> assigned to a queue instance on one machine?  Or is there a way to
> submit to a single PE and have it run exclusively on a single machine?

Then it's more convoluted and you are right, you will indeed need  
more PEs: you will need one PE for each type of machine and assign  
the number of slots on this machine as allocation_role in these PEs  
with a similar name smp2 (allocation_rule 2), smp4 and smp8. Then you  
can attach all these to one and the same queue:

pe_list NONE,[@my2corenodes=smp2],[@my4corenodes=smp4], 

(also set the slot count for each type of machine accordingly)

and request -"pe smp8 8". Although it would be more flexible to  
request "-pe smp* 8", you could end up with getting 2 hosts with 4  
slots each.

Important to notice is, that once a PE is elected by SGE, all slots  
must come from this PE.

Hence, if you have 3 nodes with 4 cores, you could setup 3 PEs -  
smp4a, smp4b and smp4c and so on - one for each host. And avoid so  
the above side effect. Then you can submit "-pe smp* -99" and will  
always get one complete node.

-- Reuti

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list