[GE users] sanity check on idea to use project value to do fairshare among users with same uid
dag at sonsorol.org
Wed Nov 10 18:05:46 GMT 2004
I've got a web portal situation where many different scientific
applications are fronted by CGIs. Behind the scenes the CGIs are talking
to Grid Engine to load balance across apple and linux clusters.
Because the jobs are coming from the web; they all run under the same
UID and Grid Engine just sees a ton of jobs being run by user "apache"
or "www" or whatever.
Grid Engine can not effectively do fairshare-by-user when all it sees is
many jobs being submitted by the webserver.
So we have the potential for one user out of many web portal users to
grab a huge percentage of cluster resources. Grid Engine has no way to
sort this out.
There is no desire to provide commandline access to the cluster so there
will never be a per-user unique userid set up.
What I've come up with was this:
1. Each web user gets a Grid Engine project created with their
portal-username. This is done automatically each time a new web portal
user is created.
2. The portal CGIs are modified to pass on "-P <username>" whenever they
do a qsub or qrsh
3. Adjust Grid Engine 5.3 or 6.0u1 policies such that "Project" is
highly weighted within the fairshare or sharetree policies
The end result is that by hijacking the Grid Engine project mechanism I
can effectively get fairshare-by-user even when all the jobs are
submitted with the same UID.
This appears to work and I can even now do per-user resource accounting
tracking with grid engine by doing:
# Get per user usage stats:
$ qacct -P <portal-username>
Just wanted to throw this out there to see if anyone sees any nasty
problems that could come back to bite me. Anyone see any problems with
Grid Engine if used in a setting where for each user there is an
associated project object?
Chris Dagdigian, <dag at sonsorol.org>
BioTeam - Independent life science IT & informatics consulting
Office: 617-665-6088, Mobile: 617-877-5498, Fax: 425-699-0193
PGP KeyID: 83D4310E iChat/AIM: bioteamdag Web: http://bioteam.net
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net
More information about the gridengine-users