[GE users] reserve-tank queue mini howto

Benjamin Singer bdsinger at Princeton.EDU
Thu Jun 7 16:25:54 BST 2007


i made a "reserve-tank" queue hack that maybe others might want to try.

it does the following:

- reserve some exec hosts for light users that just want to run a few  
jobs during the day
(these sorts of users were being locked out of the cluster by heavy  
users with long-running jobs)

- these exec hosts become available to heavy users at night

what i did was this:

- used this  prolog script for the large main queue:
---
#!/bin/bash

$SGE_ROOT/bin/$ARC/qconf -au $USER nosmall >> /dev/null 2>&1
---

(the nosmall ACL will maintain a list of users that have jobs in the  
main queue. these users are ineligible for running jobs in the small  
reserve queue)

- made a cronjob script on the qmaster that will garbage-collect the  
nosmall ACL:

---
#!/bin/bash

export SGE_ROOT=/common/sge

ARCH=`$SGE_ROOT/util/arch`
SGE_BIN="$SGE_ROOT/bin/$ARCH"

QCONF="$SGE_BIN/qconf"
QSTAT="$SGE_BIN/qstat"
EXCLUDE_LIST="nosmall"

# for every user not allowed in the small batch queue
# (set whenever a job of theirs runs in the main batch queue)
# see if they are still running in the main queue. If not,
# remove them from the nosmall excluded users list
for u in `$QCONF -su $EXCLUDE_LIST | grep entries | sed s/entries//g  
| sed s/,/' '/g`; do
         num_main_jobs=`$QSTAT -u $u | grep bat | wc -l`
         if [ $num_main_jobs -eq 0 ]; then
                 $QCONF -du $u $EXCLUDE_LIST
         fi
done
---

- made the reserve-tank ("small") queues with the queue configured  
with the line:

xuser_lists          nosmall

- i also made the seq_no of the small queue last, so the big queue  
would fill up first.

- put the small queues on the day calendar, and moved the existing  
queues on those exec hosts to the night calendar

this worked okay but after testing I realized a user with no jobs in  
the main queue who submits a big array job would still run in both  
the main and reserve-tank queue, presumably because the xuser_lists  
is not updated until after the job is scheduled.

yet another hack to work around that was:

- disable the reserve-tank queue for all but a couple seconds at the  
top of each minute when it will accept jobs. that way it will usually  
only accept jobs after the main queue is full. i couldn't find a way  
to make a calendar like that, so in another cronjob script:

---
#!/bin/bash

export SGE_ROOT=/common/sge
ARCH=`$SGE_ROOT/util/arch`
SGE_BIN="$SGE_ROOT/bin/$ARCH"
KEY=small
OPEN=4

small_queues=`$SGE_BIN/qconf -sql | grep $KEY`

# enable
$SGE_BIN/qmod -e $small_queues

# wait
sleep $OPEN

# disable
$SGE_BIN/qmod -d $small_queues

---

some notes:

- only running it for a couple days so far; so far in limited testing  
it seems okay
- probably someone out there has a much more elegant or better-tested  
solution; please share it if so!
- this was using SGEEE 5.3p4 ; finally upgrading to 6.1 this summer,  
so will have to redo this. not sure what the best way will be
- so far this is working but not ideal. a user can get the last job  
in the main queue and then be locked out of the small queue.
- depending on cronjobs is kind of distasteful. the second one  
especially is a pretty ugly solution!

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list