[GE users] how to use Python multiprocessing script with SGE

seandavi seandavi at gmail.com
Thu Apr 29 22:40:49 BST 2010

    [ The following text is in the "utf-8" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]

On Thu, Apr 29, 2010 at 3:45 PM, cguilbert69 <cguilbert69 at gmail.com> wrote:
> Hi,
> I'd like to write a python multiprocessing script which will be send using
> SGE qsub.
> this idea is that I am going to required/reserve  10 cores for instance with
> qsub and use/manage those cores to calculate various tasks in parallel
> within python.
> I am not sure I am using the proper terms here but here what I would like to
> do
> I would need to create some sort of "pipes" in which I can send different
> job in parallel , I would  need to check the status of the job sent in the
> pipe for synchronization with the other jobs so I can proceed my calculation
> further when done, I would need to be able to control,  close the pipes when
> the job is done and start a new one (with a new job).
> I spent a week online trying to find the best method to achieve my goal ,
> the net result is that I am totally confuse now.
> there's so many way of doing it.
> DRMAA-python seems to be easy to implement,  however I have no idea how to
> run it with a SGE qsub ?  should I send the python script with qsub or the
> python script is sending the job directly (without qsub) ?
> Python multiprocessing (including in the Python standard library) seems to
> be the way to go, but I could not find any example how I could make it work
> with SGE ( like reserve 10 nodes and open 10 pipes, etc ....)
> There's also many mpi Python implemenatation which are cool again,  but
> again, I don't know how to use them with SGE.
> Does any one can guide me here. if you guys could send me any scripts (even
> uncommented) so I can study them , It would be great

Hi, Chris.  SGE is not a parallel programming framework, but more for
scheduling and controlling jobs.  If a job happens to run on multiple
cores, then SGE can be "told" that and schedule the job on the
appropriate hardware.  However, you still need to write the parallel
program and SGE will not really help directly with that.

As for writing parallel programs in python, you will probably want to
look at multiprocessing and threads.  MPI is another way to go, but it
is a level of complexity that is not really necessary on a multicore



To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

More information about the gridengine-users mailing list