[GE users] AFS and SGE

Wolfgang Friebel Wolfgang.Friebel at desy.de
Tue Feb 8 11:12:53 GMT 2005

On Tue, 8 Feb 2005, Marco Donauer wrote:

> Sean,
> Wolfgang Friebel (also on the list) is using SGE with AFS. Perhaps he can 
> help you?

Maybe yes. I was already trying to put something together when your mail 
arrived. Just to get some information to the list I can give a short overview 
what we are doing to have SGE with AFS.


1) Install the qmaster and exec hosts with the additional switch -afs
    (e.g. install_qmaster -afs or inst_sge -m -afs)

Then you should see AFS related info in the output of qconf -sconf
similar to (assuming that SGE_ROOT is /usr/SGE):

set_token_cmd             /usr/SGE/util/set_token_cmd
pag_cmd                   /usr/afsws/bin/pagsh
token_extend_time         23:30:0

2) At qsub time a script /usr/SGE/util/get_token_cmd
is called that can be used to fetch the AFS token from memory and store it
together with the job information for later use. For the procedure described 
below it is sufficient to check that the user is authenticated.

At job start time another script /usr/SGE/util/set_token_cmd is called
that will restore or create an AFS token for a given user. The procedure 
proposed originally did retrieve the AFS token stored with get_token_cmd
and then by decoding and reassembling the token with an increased lifetime
a new AFS token was stored in the process environment (memory).

The procedure currently used by us is different and is based on Kerberos5:
The SGE daemons are running as root and do have access to a Kerberos5 key file 
where a key for a special privileged kerberos principal is stored. (Another 
user that has access to that key file could be used as well)

By means of this key file a ticket for that principal is obtained and a
remote Kerberos5 aware server is contacted. That server verifies the access 
rights of the client (which is the set_token_cmd script). That is done by 
verifying the validity of the K5 ticket obtained by the client.

The server itself has access to another keytab file where the key for a k5 
principal with admin rights is stored. The server gets this way a K5 admin 
ticket and with that ticket the Kerberos5 KDC can be contacted and a key file 
for a given user is created. That key file is used to create a 
credentials cache which is transferred from the server to 
the client in a secure way. Then on the client this cache provides the 
Kerberos ticket for the user. If Heimdal-Kerberos5 is used, then an 
AFS token can be generated simultaneously, otherwise (MIT) aklog can be called 
to get an AFS token from the K5 ticket.
It is important to get the AFS token in a pagsh environment. Otherwise
nasty side effects will occur.

3) For the communication between the qmaster and the shadow masters we do use 
AFS to share common files. As there are other methods to share these files 
(Berkeley DB spooling, other Filesystems like NFS) this step is optional. We 
have been using IP based ACL's in AFS to select the proper access privileges 
from the servers to the shared files like follows:

pts creategroup sgeserver
pts createuser <ip_adr of qmaster>       #qmaster
pts createuser <ip_adr of shadow master> #shadow1
pts adduser <ip_adr of qmaster>       sgeserver
pts adduser <ip_adr of shadow master> sgeserver

Currently our get_token_cmd and set_token_cmd scripts are still highly
site specific and cannot easily be distributed. We could make available
the code for the kerberos5 aware server and client that was described above. 
>From these programs the get_token_cmd  and set_token_cmd scripts
could be easily derived. We will probably have such scripts ready for 
distribution by end of April.
The K5 client and server are pure perl implementations based on Authen::SASL. I 
have put the relevant files to our anonymous ftp server:

The files
Authen-SASL-2.08.tar.gz        (also on CPAN)
Authen-SASL-Cyrus-0.11.tar.gz  (the version on CPAN is different and does
                                 not support server side K5)

are required

If you need more information please do not hesitate to contact me again

Best regards
Wolfgang Friebel                   Deutsches Elektronen-Synchrotron DESY
Phone/Fax:  +49 33762 77372/216    Platanenallee 6
Mail: Wolfgang.Friebel AT desy.de  D-15738 Zeuthen  Germany

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list