[GE users] Long delay when submitting large jobs

Ron Chen ron_chen_123 at yahoo.com
Tue Feb 8 08:21:03 GMT 2005

--- Reuti <reuti at staff.uni-marburg.de> wrote: 
> > Yes, but again, just offload the MPI implementation details to mpiexec. The
> > TM interface is cleaner than what we have today.
> Agreed.

BTW, if OpenPBS/PBSPro/Torque and Gridengine all use TM, then may be more MPI vendors will be
happy to provide TM enabled "parallel job starter" (mpirun?)

> Well, let me explain this way: for now, SGE will catch the rsh, start the rshd, 
> an use a 'real' rsh to start the communication. Instead of starting the rshd, 
> why not directly start the program on the node as child of the shepered? No 
> rshd in the way. Would this work with all parallel programs out of the box?

Not exactly, because SGE itself (without qrsh/rshd) doesn't forward stdio.

> I looked into the mpiexec source, and at a first glance I saw support for all 
> the communication types like ch_p4 and gm. And the question was: is this 
> necessary, because you have only PBS_MOM and no sheperd on the slave nodes with 
> PBS - so you must take care of the communication on your own?

Not because of that.

Remember, mpiexec only replaces mpirun, and the main difference is that mpiexec starts tasks using
TM APIs instead of rsh.

Honestly, I currently don't understand the mpiexec source, as I just started reading it today. But
luckily, it is less than 5000 lines of code, and with lots of comments 8-)

And I saw some discussions on the mpiexec list today:


So it seems like there's something called the "listener" which is different depending on the
communication mode, and also there are different env setups that needed to be done depending on it

> And: what will mpiexec do with the processgroups on the slaves?

BTW, don't know much about that, what is it for?

> And argh, will the TM also send the environment variables to the 
> slave nodes like -V? Okay, I will see it - give me some days.

Yes, from the tm manpage:


int tm_spawn(argc, argv, envp, where, tid, event)

So it does allow you to pass **envp to the remote node.


Do you Yahoo!? 
Read only the mail you want - Yahoo! Mail SpamGuard. 

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list