[GE users] how to use Python multiprocessing script with SGE

reuti reuti at staff.uni-marburg.de
Sat May 15 01:13:04 BST 2010


Hi,

Am 14.05.2010 um 20:59 schrieb cguilbert69:

> Thanks Sean for the answer,
>
>
> I don't really want to use mpi for python (yet) but python  
> multiprocessing.
> The real question for me is how you pass the CPU you reserved with  
> SGE queuing to python multiprocessing
>
>
> For instance if I run the python script:
>
> import multiprocessing as mp
> print mp.cpu_count()
>
> I will have 4 because its a quad core.
>
> Now if I run the same script inside a qsub script with the option:
> #$ -pe mpi 10

usually MPI is setup to run between nodes. A PE smp with allocation  
rule $pe_slots might be better suited for Python's threads.


> I still have 4 and not 10 as I request in my qsub script.

Having 10 slots on 4 cores won't improve speed. But anyway: is  
mp.cpu_count() used to start processes? If it's just a limit which you  
have to access when starting the threads in a loop, you can access the  
environment variable $NSLOTS inside Python instead. It's set up by SGE  
with the number of granted slots.

-- Reuti


> the real question for me is how python multiprocessing can acquired  
> the "knowledge" and the use of the CPU you reserved with qsub ?
>
>
> Thanks
>
>
>
> On Thu, Apr 29, 2010 at 2:40 PM, seandavi <seandavi at gmail.com> wrote:
> On Thu, Apr 29, 2010 at 3:45 PM, cguilbert69 <cguilbert69 at gmail.com>  
> wrote:
> > Hi,
> >
> > I'd like to write a python multiprocessing script which will be  
> send using
> > SGE qsub.
> > this idea is that I am going to required/reserve  10 cores for  
> instance with
> > qsub and use/manage those cores to calculate various tasks in  
> parallel
> > within python.
> >
> > I am not sure I am using the proper terms here but here what I  
> would like to
> > do
> > I would need to create some sort of "pipes" in which I can send  
> different
> > job in parallel , I would  need to check the status of the job  
> sent in the
> > pipe for synchronization with the other jobs so I can proceed my  
> calculation
> > further when done, I would need to be able to control,  close the  
> pipes when
> > the job is done and start a new one (with a new job).
> >
> > I spent a week online trying to find the best method to achieve my  
> goal ,
> > the net result is that I am totally confuse now.
> > there's so many way of doing it.
> >
> > DRMAA-python seems to be easy to implement,  however I have no  
> idea how to
> > run it with a SGE qsub ?  should I send the python script with  
> qsub or the
> > python script is sending the job directly (without qsub) ?
> >
> > Python multiprocessing (including in the Python standard library)  
> seems to
> > be the way to go, but I could not find any example how I could  
> make it work
> > with SGE ( like reserve 10 nodes and open 10 pipes, etc ....)
> >
> > There's also many mpi Python implemenatation which are cool  
> again,  but
> > again, I don't know how to use them with SGE.
> >
> > Does any one can guide me here. if you guys could send me any  
> scripts (even
> > uncommented) so I can study them , It would be great
>
> Hi, Chris.  SGE is not a parallel programming framework, but more for
> scheduling and controlling jobs.  If a job happens to run on multiple
> cores, then SGE can be "told" that and schedule the job on the
> appropriate hardware.  However, you still need to write the parallel
> program and SGE will not really help directly with that.
>
> As for writing parallel programs in python, you will probably want to
> look at multiprocessing and threads.  MPI is another way to go, but it
> is a level of complexity that is not really necessary on a multicore
> machine.
>
> Sean
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=255480
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net 
> ].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=257342

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list