[GE users] how to use Python multiprocessing script with SGE

rayson rayrayson at gmail.com
Sat May 15 01:10:01 BST 2010


Not sure if python multiprocessing supports distributed memory
programming, but the information about the hosts & CPUs is listed in
"pe_hostfile".

http://blogs.sun.com/templedf/entry/configuring_a_new_parallel_environment

Rayson



On 5/14/10, cguilbert69 <cguilbert69 at gmail.com> wrote:
> Thanks Sean for the answer,
>
> I don't really want to use mpi for python (yet) but python multiprocessing.
> The real question for me is how you pass the CPU you reserved with SGE
> queuing to python multiprocessing
>
>
> For instance if I run the python script:
>
> import multiprocessing as mp
> print mp.cpu_count()
>
> I will have 4 because its a quad core.
>
> Now if I run the same script inside a qsub script with the option:
> #$ -pe mpi 10
>
> I still have 4 and not 10 as I request in my qsub script.
>
> the real question for me is how python multiprocessing can acquired the
> "knowledge" and the use of the CPU you reserved with qsub ?
>
>
> Thanks
>
>
>
>
> On Thu, Apr 29, 2010 at 2:40 PM, seandavi <seandavi at gmail.com> wrote:
> >
> >
> >
> > On Thu, Apr 29, 2010 at 3:45 PM, cguilbert69 <cguilbert69 at gmail.com>
> wrote:
> > > Hi,
> > >
> > > I'd like to write a python multiprocessing script which will be send
> using
> > > SGE qsub.
> > > this idea is that I am going to required/reserve  10 cores for instance
> with
> > > qsub and use/manage those cores to calculate various tasks in parallel
> > > within python.
> > >
> > > I am not sure I am using the proper terms here but here what I would
> like to
> > > do
> > > I would need to create some sort of "pipes" in which I can send
> different
> > > job in parallel , I would  need to check the status of the job sent in
> the
> > > pipe for synchronization with the other jobs so I can proceed my
> calculation
> > > further when done, I would need to be able to control,  close the pipes
> when
> > > the job is done and start a new one (with a new job).
> > >
> > > I spent a week online trying to find the best method to achieve my goal
> ,
> > > the net result is that I am totally confuse now.
> > > there's so many way of doing it.
> > >
> > > DRMAA-python seems to be easy to implement,  however I have no idea how
> to
> > > run it with a SGE qsub ?  should I send the python script with qsub or
> the
> > > python script is sending the job directly (without qsub) ?
> > >
> > > Python multiprocessing (including in the Python standard library) seems
> to
> > > be the way to go, but I could not find any example how I could make it
> work
> > > with SGE ( like reserve 10 nodes and open 10 pipes, etc ....)
> > >
> > > There's also many mpi Python implemenatation which are cool again,  but
> > > again, I don't know how to use them with SGE.
> > >
> > > Does any one can guide me here. if you guys could send me any scripts
> (even
> > > uncommented) so I can study them , It would be great
> >
> > Hi, Chris.  SGE is not a parallel programming framework, but more for
> > scheduling and controlling jobs.  If a job happens to run on multiple
> > cores, then SGE can be "told" that and schedule the job on the
> > appropriate hardware.  However, you still need to write the parallel
> > program and SGE will not really help directly with that.
> >
> > As for writing parallel programs in python, you will probably want to
> > look at multiprocessing and threads.  MPI is another way to go, but it
> > is a level of complexity that is not really necessary on a multicore
> > machine.
> >
> > Sean
> >
> > ------------------------------------------------------
> >
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=255480
> >
> > To unsubscribe from this discussion, e-mail:
> [users-unsubscribe at gridengine.sunsource.net].
> >
>
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=257340

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list