A Prototype of a Multi-Clustering Implementation using Transfer Queues

last updated: November 27, 2002


Multi-clustering can be defined as the sharing of compute jobs between two or more independently controlled clustergrids. This could be desirable, for example, if the owners of the clusters realize that there are times when one cluster is lightly-loaded while another has a large pending list. In this case, it makes sense to make use of the idle resources, at least until the "local" demand for this cluster rises again.

We define the "local" cluster as the cluster with which a user would normally interact (qsub, qstat, etc), and "remote" cluster as the one to which jobs are dispatched (if deemed appropriate).

One way to implement multi-clustering is with transfer queues. The idea is that jobs dispatched to a transfer queue get submitted to a remote cluster; this would happen transparently to the user. To have jobs going in both directions, simply use transfer queues on both sides.


The prototype described here implements the following ideas:

The scripts used to implement the prototype can be downloaded here:

Restricting Assumptions:


To implement this prototype, create a new queue (the transfer queue) on any existing host. Configure it as shown below. Implement the load sensor on any host (it would make sense to implement it on the same host where the transfer queue lives, but that's not necessary). Configure that host as shown below.

NOTE: it is suggested that the transfer queue be implemented on the local master host, to keep things simple and to reduce communication latency.

NOTE: make sure this host is both a submit and admin host on the remote cluster.

NOTE: please modify the scripts to match your local environment. See the scripts for site-specific variables.

Test it by submitting jobs directly to the transfer queue, eg,

qsub -q transfer1 -o output.log -j y sleeper.sh 

In practice, you would not submit jobs directly to this special queue. Rather, one of the following approaches would be taken:

  1. if all jobs are "generic" enough, then it might not matter if they run in the local or remote cluster. In this case, jobs would just be submitted normally, and those which happen to end up in the transfer queue will run remotely

  2. if certain jobs cannot be run on the remote cluster for whatever reason (licensing restrictions, data/binaries inaccessible remotely), then a user-defined complex with a boolean resource (eg, "remote") should be attached to the transfer queue(s). Jobs which cannot be run remotely could be submitted with a resource request "-l remote=false". This will prevent those jobs from being dispatched to the transfer queue(s), and hence will not be dispatched remotely.

Setup of transfer queue (only relevant lines shown)

sr3-umpk-01$ qconf -sq transfer1
qname transfer1
hostname tcf-b060
load_thresholds mpk27jobs=5
qtype BATCH 
slots 40
prolog NONE
epilog NONE
starter_method /sge/TransferQueues/transfer_starter.sh
suspend_method /sge/TransferQueues/transfer_suspend.sh
resume_method /sge/TransferQueues/transfer_resume.sh
terminate_method /sge/TransferQueues/transfer_terminate.sh

Load sensor implemented on a single host (only relevant lines shown)

sr3-umpk-01$ qconf -sconf tcf-b060
load_sensor /sge/TransferQueues/clusterload.sh
load_report_time 00:00:09




There are some improvements which could be made to the current implementation of methods:

Future Directions

The implementation described here is limited in terms of transfering jobs between administrative domains (eg, different file/user namespaces, different firewalls,etc). In order to extend the concept to multiple domains, the following extensions to the MultiCluster concept would require putting additional functionality into the Grid Engine code, and most likely, integration with functionality provided by other software:

In addition, cross-domain issues tend to rely more upon standard APIs and protocols. Currently, these are not well-defined; much work towards defining these standards is occuring in standards bodies such as the Global Grid Forum (GGF).