[GE users] run job only on hosts where software X is installed?

reuti reuti at staff.uni-marburg.de
Mon Nov 10 18:35:44 GMT 2008

Hi Udo,

Am 10.11.2008 um 18:58 schrieb udo waechter:

> Hello sge users.
> I am pretty new to SGE and have stumbled upon a problem that I can not
> solve.
> I guess I simply do not know how to search for the problem. If it was
> already asked, please excuse my incompetence.
> I would like to execute a job only an those hosts that have a certain
> piece of software installed.
> As I understand the whole problem so far, it seems to me that
> something like a load_sensor would do the trick.
> If I want to execute mencoder only on those hosts that have /usr/bin/
> mencoder installed, I could write a sensor which checks whether or not
> it is installed. But what if I would want to have this facility
> generalized? Namely: Let jobs have checks for arbitrary software in
> order to be executed on certain hosts?

in principle it could be done by a load sensor, but it this would  
mean to have this "reporting facility" running all the time on the  
nodes - and as the value will not change so often, it's unnecessary.

"How to look for certain packages" depends on your installations. The  
load sensor could just check in some places. I think, "which myapp"  
can't be used, as the PATH is not updated for an already running load  
sensor. So it might miss newly installed software, unless links to  
the apps are put in a general location like /usr/bin.

> I do not really get the trick with resources.
> For example, http://gridengine.sunsource.net/howto/consumable.html
> describes the 'verilog' complex, where each execution host defines how
> big its value is. Although 'complexes' would be the right thing for my
> use-case, they do not seem to be appropriate.
> Back to the 'load_sensors': Would it be possible (if yes, how) to have
> a parametrized load_sensor?

Using resources is the right approach. You will just need to:

1 - create a new entry with "qconf -mc" and a line like

myapp               ma         BOOL        ==    YES          
NO         0        0

2a - then attach this either to a queue "qconf -mq my_queue" by  
hostgroup or individual host:

complex_values        NONE,[@myhostgroup=myapp=TRUE],[myhost=myapp=TRUE]

2b - or each host "qconf -me my_host":

complex_values        myapp=TRUE

3 - submit the jobs and request this resource:

qsub -l myapp my_job.sh

(for boolean resources the attribute "TRUE" is assumed by default,  
but you could also use "-l myapp=TRUE"). Whether you implement 2a or  
2b is for this intended setup personal taste and depends more of the  
overall queue and host organization, i.e. which is less work to set  
it up would be to attach it to a queue; OTOH more logical would be to  
attach it to each host, as this will also work if there are several  
queues on each machine and it's a feature of a particular machine. In  
case of 2b you could also use a cron job to update the flag once a  
day after a similar search like you intend with the load sensor.

-- Reuti


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

More information about the gridengine-users mailing list