[GE users] Load average problem again

Reuti reuti at staff.uni-marburg.de
Mon Aug 16 20:39:51 BST 2004


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Anand,

>Exactly, so here is how the whole process is setup now. We generate lots of
>data files continuously that is pushed to the cluster. As soons as these
>data files are received they are submitted to the SGE. And each SGE job will
>in turn use these perl scripts that will do different kinds of processing on
>these data files. And it spawns a tree of processes 1.pl opens the data file
>that generates 10000's of small files now script 2.pl is executed on
>different nodes each with their share of small files and one instance of
>2.pl will open one small file and perform some computation and call 3.pl and
>will wait on 3.pl to finish and 3.pl is linked with 4.pl. Due to the nature
>of the scripts they actually run for a very short time but the initiation

how short is short? You have of course an overhead with SGE. But also without 
any queuing system you will lose time, if the jobs are too short. Do you see a 
chance to rewrite the scripts and/or combine them? E.g. making a loop in the 
2.pl scripts over a bigger part of the 10,000 files instead of running them so 
often, and incorporate 3.pl and 4.pl in them?

Is 2.pl also submitted with qsub, or is the whole job one big parallel job?

As you already discovered, it's not easy to get a good load average for such 
type of jobs.

Cheers - Reuti

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list