[GE users] sanity check on idea to use project value to do fairshare among users with same uid

Chris Dagdigian dag at sonsorol.org
Wed Nov 10 18:05:46 GMT 2004

Hi folks,

I've got a web portal situation where many different scientific 
applications are fronted by CGIs. Behind the scenes the CGIs are talking 
to Grid Engine to load balance across apple and linux clusters.

Because the jobs are coming from the web; they all run under the same 
UID and Grid Engine just sees a ton of jobs being run by user "apache" 
or "www" or whatever.

Grid Engine can not effectively do fairshare-by-user when all it sees is 
many jobs being submitted by the webserver.

So we have the potential for one user out of many web portal users to 
grab a huge percentage of cluster resources. Grid Engine has no way to 
sort this out.

There is no desire to provide commandline access to the cluster so there 
will never be a per-user unique userid set up.

What I've come up with was this:

1. Each web user gets a Grid Engine project created with their 
portal-username. This is done automatically each time a new web portal 
user is created.

2. The portal CGIs are modified to pass on "-P <username>" whenever they 
do a qsub or qrsh

3. Adjust Grid Engine 5.3 or 6.0u1 policies such that "Project" is 
highly weighted within the fairshare or sharetree policies

The end result is that by hijacking the Grid Engine project mechanism I 
can effectively get fairshare-by-user even when all the jobs are 
submitted with the same UID.

This appears to work and I can even now do per-user resource accounting 
tracking with grid engine by doing:

# Get per user usage stats:
$ qacct -P <portal-username>

Just wanted to throw this out there to see if anyone sees any nasty 
problems that could come back to bite me. Anyone see any problems with 
Grid Engine if used in a setting where for each user there is an 
associated project object?


Chris Dagdigian, <dag at sonsorol.org>
BioTeam  - Independent life science IT & informatics consulting
Office: 617-665-6088, Mobile: 617-877-5498, Fax: 425-699-0193
PGP KeyID: 83D4310E iChat/AIM: bioteamdag  Web: http://bioteam.net

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list