[GE users] Using gridengine to administer the cluster
landman at scalableinformatics.com
Wed Aug 30 01:22:34 BST 2006
[ The following text is in the "ISO-8859-1" character set. ]
[ Your display is set for the "ISO-8859-10" character set. ]
[ Some special characters may be displayed incorrectly. ]
Michael James wrote:
> But if I start using external commands,
> how do I schedule them into gridengine?
> ie: On a fairly full local data partition I want to run this:
> biofile> rsync -a --delete /nfs/master/data/ /data/
> "biofile" is the owner of the biological databases.
> At present I have to wait for a quiet moment,
> (or make one) and then run the job.
> Otherwise I risk SGE sending a blast on nr while it's being updated.
If the major issue is data motion, this is a very solvable problem. You
can overlap communication and computation by careful naming of the
databases, and some path management. Aside from this, rsync is a very
processor intensive tool, and there are better ones for data motion of
the type you are talking about (syncing big genomic/protein db's to
compute nodes). Better to talk offline about that.
> The attraction of setting a consumable and slots
> is that I can schedule the update and it waits
> until it can monopolise each node and do its job undisturbed.
> Clever scheduling would even mean I could set a limit
> that only a third of my cluster are allowed
> to take themselves offline for maintenance at any one time.
If the major issue is the data motion, as indicated there are ways to
process this with a low overhead while a current computation is in
flight, without disturbing the current computation.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452 or +1 866 888 3112
cell : +1 734 612 4615
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net
More information about the gridengine-users