[GE users] Using gridengine to administer the cluster

Joe Landman landman at scalableinformatics.com
Wed Aug 30 01:22:34 BST 2006

    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Michael James wrote:
> But if I start using external commands,
> how do I schedule them into gridengine?
> ie: On a fairly full local data partition I want to run this:
> biofile>  rsync -a --delete /nfs/master/data/ /data/
> "biofile"  is the owner of the biological databases.
> At present I have to wait for a quiet moment,
>  (or make one) and then run the job.
> Otherwise I risk SGE sending a blast on nr while it's being updated.

If the major issue is data motion, this is a very solvable problem.  You
can overlap communication and computation by careful naming of the
databases, and some path management.   Aside from this, rsync is a very
processor intensive tool, and there are better ones for data motion of
the type you are talking about (syncing big genomic/protein db's to
compute nodes).  Better to talk offline about that.

> The attraction of setting a consumable and slots
>  is that I can schedule the update and it waits
>  until it can monopolise each node and do its job undisturbed.
> Clever scheduling would even mean I could set a limit
>  that only a third of my cluster are allowed
>  to take themselves offline for maintenance at any one time.

If the major issue is the data motion, as indicated there are ways to
process this with a low overhead while a current computation is in
flight, without disturbing the current computation.

> michaelj

Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452 or +1 866 888 3112
cell : +1 734 612 4615

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list