[GE users] To run same scripts on multiple hosts

reuti reuti at staff.uni-marburg.de
Fri Mar 5 14:16:17 GMT 2010


Hi,

Am 04.03.2010 um 21:36 schrieb arvindpetaru:

> I'm new to SGE. I've installed SGE 6.2u5 on sparc solaris 10. I  
> have an existing project which is similar to SGE design of Master/ 
> Slave interactions. So want to integarte existing project into SGE  
> environment.
>
> Brief overview of my existing project....
>
> At setup, user would identify the master and slave hosts and a  
> installation script would install all the required K-shell scripts 
> (around 30+) and couple of C-binaries on eah host. So, after the  
> installation process, each host(including master-host) would have  
> same directory-structure and files(scripts and binaries).

so, nothing is shared and accessible from all machines like /opt/ 
applications or the users' homes?


> After installation, each host would start its own locally available  
> scripts in a particlar order to gather its own data and pass it on  
> to Master-host. Then the master host would signal all hosts to run  
> its binaries at the same time. So the C-binary is started and  
> stopped at the same time on all hosts gathering required data-info  
> and passed it on to master-host. Master-host would process this  
> data from all hosts and completes it assigned task....
>
> Here, are my below questions:-
>
> 1> How do I ask SGE to start running same scripts on each host at  
> the same time ?

This is no foreseen - especially when you mean "at the same" time in  
a sense of a "real-time" system. SGE will grant you a set of nodes  
(better: slots) for a parallel job and start one script/binary on the  
master node of a parallel job, the rest is up to you. All kids must  
be invoked on your own by any type of MPI, PVM, Linda or other type  
of parallel library.

MPI has such support for starting tasks on the nodes as PVM does.  
When you setup these for a Tight Integration, SGE will also have  
control of the started kids by the used `qrsh -inherit ...` to start  
something on other granted nodes of this job.

If you want to use `qrsh -inherit ...`directly on your own, this is  
also possible. The small challenge here is, that there is no real  
async qrsh. But you can check the Tight Integration for PVM, where an  
async wrapper is used to start the daemons on all nodes:

http://gridengine.sunsource.net/howto/pvm-integration/pvm- 
integration.html


> 2> Can I use a newly-developed script to initiate to use locally  
> available scripts in each host to run.? If so, how to do that ?

Sure you can do that.


> 3> Few scripts internally call other scripts...Does SGE have  
> control/track these called-scripts? If not, what happens if the  
> called-script fails? Does it get notified to the master-host ?

In a Tight Integration - yes.


> 4> How to pass data from each host to master-host. I read abt  
> prolog/epilog.....but how mcuh data it can support......my data  
> gathering on each host is huge.

The is no built-in transfer of data. The prolog/epilog scripts are  
started, and you have to program it on your own.


> 5> Also, the C-binary spawns mutiple processes on each host based  
> on the data obtained on it. But, here the criteria is that....C- 
> binary is started and stopped at te same time on each host machine.  
> Can we do this in SGE env ?

No, must be implemented by the user as suggested above.

-- Reuti


> I'd appreciate your clarifications on the above questions/concerns.
>
> Thanks in advance.
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do? 
> dsForumId=38&dsMessageId=247112
>
> To unsubscribe from this discussion, e-mail: [users- 
> unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=247196

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list