[GE users] how to use Python multiprocessing script with SGE

cguilbert69 cguilbert69 at gmail.com
Fri May 14 19:59:56 BST 2010


    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Thanks Sean for the answer,


I don't really want to use mpi for python (yet) but python multiprocessing.
The real question for me is how you pass the CPU you reserved with SGE queuing to python multiprocessing


For instance if I run the python script:

import multiprocessing as mp
print mp.cpu_count()

I will have 4 because its a quad core.

Now if I run the same script inside a qsub script with the option:
#$ -pe mpi 10

I still have 4 and not 10 as I request in my qsub script.

the real question for me is how python multiprocessing can acquired the "knowledge" and the use of the CPU you reserved with qsub ?


Thanks



On Thu, Apr 29, 2010 at 2:40 PM, seandavi <seandavi at gmail.com<mailto:seandavi at gmail.com>> wrote:
On Thu, Apr 29, 2010 at 3:45 PM, cguilbert69 <cguilbert69 at gmail.com<mailto:cguilbert69 at gmail.com>> wrote:
> Hi,
>
> I'd like to write a python multiprocessing script which will be send using
> SGE qsub.
> this idea is that I am going to required/reserve  10 cores for instance with
> qsub and use/manage those cores to calculate various tasks in parallel
> within python.
>
> I am not sure I am using the proper terms here but here what I would like to
> do
> I would need to create some sort of "pipes" in which I can send different
> job in parallel , I would  need to check the status of the job sent in the
> pipe for synchronization with the other jobs so I can proceed my calculation
> further when done, I would need to be able to control,  close the pipes when
> the job is done and start a new one (with a new job).
>
> I spent a week online trying to find the best method to achieve my goal ,
> the net result is that I am totally confuse now.
> there's so many way of doing it.
>
> DRMAA-python seems to be easy to implement,  however I have no idea how to
> run it with a SGE qsub ?  should I send the python script with qsub or the
> python script is sending the job directly (without qsub) ?
>
> Python multiprocessing (including in the Python standard library) seems to
> be the way to go, but I could not find any example how I could make it work
> with SGE ( like reserve 10 nodes and open 10 pipes, etc ....)
>
> There's also many mpi Python implemenatation which are cool again,  but
> again, I don't know how to use them with SGE.
>
> Does any one can guide me here. if you guys could send me any scripts (even
> uncommented) so I can study them , It would be great

Hi, Chris.  SGE is not a parallel programming framework, but more for
scheduling and controlling jobs.  If a job happens to run on multiple
cores, then SGE can be "told" that and schedule the job on the
appropriate hardware.  However, you still need to write the parallel
program and SGE will not really help directly with that.

As for writing parallel programs in python, you will probably want to
look at multiprocessing and threads.  MPI is another way to go, but it
is a level of complexity that is not really necessary on a multicore
machine.

Sean

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=255480

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].




More information about the gridengine-users mailing list