[GE users] how to use Python multiprocessing script with SGE
reuti at staff.uni-marburg.de
Sat May 15 01:13:04 BST 2010
Am 14.05.2010 um 20:59 schrieb cguilbert69:
> Thanks Sean for the answer,
> I don't really want to use mpi for python (yet) but python
> The real question for me is how you pass the CPU you reserved with
> SGE queuing to python multiprocessing
> For instance if I run the python script:
> import multiprocessing as mp
> print mp.cpu_count()
> I will have 4 because its a quad core.
> Now if I run the same script inside a qsub script with the option:
> #$ -pe mpi 10
usually MPI is setup to run between nodes. A PE smp with allocation
rule $pe_slots might be better suited for Python's threads.
> I still have 4 and not 10 as I request in my qsub script.
Having 10 slots on 4 cores won't improve speed. But anyway: is
mp.cpu_count() used to start processes? If it's just a limit which you
have to access when starting the threads in a loop, you can access the
environment variable $NSLOTS inside Python instead. It's set up by SGE
with the number of granted slots.
> the real question for me is how python multiprocessing can acquired
> the "knowledge" and the use of the CPU you reserved with qsub ?
> On Thu, Apr 29, 2010 at 2:40 PM, seandavi <seandavi at gmail.com> wrote:
> On Thu, Apr 29, 2010 at 3:45 PM, cguilbert69 <cguilbert69 at gmail.com>
> > Hi,
> > I'd like to write a python multiprocessing script which will be
> send using
> > SGE qsub.
> > this idea is that I am going to required/reserve 10 cores for
> instance with
> > qsub and use/manage those cores to calculate various tasks in
> > within python.
> > I am not sure I am using the proper terms here but here what I
> would like to
> > do
> > I would need to create some sort of "pipes" in which I can send
> > job in parallel , I would need to check the status of the job
> sent in the
> > pipe for synchronization with the other jobs so I can proceed my
> > further when done, I would need to be able to control, close the
> pipes when
> > the job is done and start a new one (with a new job).
> > I spent a week online trying to find the best method to achieve my
> goal ,
> > the net result is that I am totally confuse now.
> > there's so many way of doing it.
> > DRMAA-python seems to be easy to implement, however I have no
> idea how to
> > run it with a SGE qsub ? should I send the python script with
> qsub or the
> > python script is sending the job directly (without qsub) ?
> > Python multiprocessing (including in the Python standard library)
> seems to
> > be the way to go, but I could not find any example how I could
> make it work
> > with SGE ( like reserve 10 nodes and open 10 pipes, etc ....)
> > There's also many mpi Python implemenatation which are cool
> again, but
> > again, I don't know how to use them with SGE.
> > Does any one can guide me here. if you guys could send me any
> scripts (even
> > uncommented) so I can study them , It would be great
> Hi, Chris. SGE is not a parallel programming framework, but more for
> scheduling and controlling jobs. If a job happens to run on multiple
> cores, then SGE can be "told" that and schedule the job on the
> appropriate hardware. However, you still need to write the parallel
> program and SGE will not really help directly with that.
> As for writing parallel programs in python, you will probably want to
> look at multiprocessing and threads. MPI is another way to go, but it
> is a level of complexity that is not really necessary on a multicore
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net
To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
More information about the gridengine-users