[GE users] Why no DRMAA support for Windows?

Andy Schwierskott andy.schwierskott at sun.com
Fri Sep 28 14:32:21 BST 2007


Andreas,

I'll be answering in the dev list.

Andy


>> [this really is a developer mailing list topic now but since many
>> technically interested people are just subscribed to the users mailing list
>> but are curious to know what we are doing and what might be done I'm 
>> abusing
>> this alias to share our ideas]
>> 
>>> sounds nice forehand, but it does not answer the question how to provide a 
>>> Windows dll DRMAA library then? Though I agree Java is not unimportant, 
>>> but a traditional DRMAA library still is needed for C and C++ programs and 
>>> as base for the entire scripting lanuguages world: Perl/Python/Ruby/etc.
>> 
>> that's true - I have no good answer yet for this challenge. Either the SUA
>> environment in Windows Vista will allow to create and load Windows DLLs
>> built under a SUA development environment or we get some other great ideas.
>> Building a native Windows JAPI layer (which would include porting of tons 
>> of
>> Unix SGE code including the commlib to Windows) on which DRMAA is sitting 
>> is
>> something I don't see a realistic chance that we (i.e. the Sun developmers)
>> can and will do in the forseeable future.
>
> "Tons of Unix source code" is an undue exeggeration. Almost the entire code 
> is ANSI C. Only exceptions is threading and TCP code, but there is no doubt 
> it can be ported.
>
>> Perhaps another approach is strategically much more interesting to pursue
>> and even could attract developers from the community: with making the
>> scheduler a thread and loading a JVM we added a new internal communication
>> interface. In 6.1 and before the GDI communication (that's how we call
>> somewhat sloppily the interface/API how clients communicate with qmaster;
>> qmaster processes these requests in so called GDI threads) between qmaster
>> and scheduler was done over the network using the the communication library
>> (commlib). The scheduler thread and the JVM thread will use a new internal
>> GDI interface which bypasses the commlib to get/send data to the qmaster
>> core. In addition we create an interface which allows to start/stop qmaster
>> threads at runtime.
>> 
>> With that infrastructure in place one could go ahead a create a thread 
>> which
>> implements most part of the JAPI and send the data over the network in a
>> must less SGE dependent format, e.g. XML. On the client side it would be
>> possible to implement a much thinner DRMAA library which would mostly
>> implement the DRMAA API calls. Such a library should be very easy to port 
>> to
>> non-Unix OS'es like Windows. I think that approach would open the
>> possibility to much more easily create bindings for other programming and
>> scripting languages for DRMAA and implement a general SGE API. Likely this
>> approach also could be attracive for those who request a web services 
>> access
>> to SGE...
>
> Would you allow me two questions on this thinner DRMAA library?
>
> (1) How would you ensure it can be used concurrently by multiple threads? 
> Wouldn't this again require Windows threading library calls be used?
>
> (2) How you connect to your visualized JAPI-thread? Wouldn't it again use 
> TCP?
>
> Bottom-line: You gain nothing, but significant delay on the availability of 
> DRMAA for Windows.
>
> Regards,
> Andreas
>
>> 
>> 
>> Well, those of you who are now asking for dates when they can have a look
>> and try at the technical stuff we are currently doing, should now subscribe
>> to the "dev" mailing list and ask there.
>> 
>> Andy
>> 
>>> On Fri, 28 Sep 2007, Andy Schwierskott wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Dan is mentioning below we might have a pure Java implementation of DRMAA 
>>>> in
>>>> the future - so what's behind that vague statement?
>>>> 
>>>> For SGE 6.2 we are currently implementing a Mbean server running as a 
>>>> thread
>>>> in qmaster. This means qmaster can load a JVM when it starts up.
>>>> 
>>>> Currently most likely not known to most of you there's a unsupported and
>>>> largely undocumented libgdi.so/jgdi.jar part of SGE 6.1 which is a low 
>>>> level
>>>> access method (read, write, modify; I'm reluctant to call it API yet) to
>>>> most of the SGE data structures residing in qmaster. The big disadvantage 
>>>> of
>>>> that approach is that it requires to load the native libgdi.so into a 
>>>> Java
>>>> application - something which you would like to avoid if you want to 
>>>> contact
>>>> SGE thorugh app server. It also makes live for our Hededy engineers much
>>>> easier if the same JVM can contact multiple SGE cluster, possibly running
>>>> different versions or patch levels.
>>>> 
>>>> With the JVM running in qmaster the functional part of libgdi.so is 
>>>> moving
>>>> into qmaster and Java clients can contact qmaster without loading native
>>>> code. That's still not a true API (und the low level access will be an
>>>> unsupported and unstable part of SGE 6.2) because it's still low level 
>>>> acess
>>>> and it's still not DRMAA. However this approach opens the door to have a
>>>> pure Java DRMAA client which will base on that infrastructure.
>>> 
>>>> 
>>>> 
>>>> Andy
>>>> 
>>>> On Thu, 27 Sep 2007, Daniel Templeton wrote:
>>>> 
>>>>> From a purely structural perspective, no.  The windows bundle doesn't 
>>>>> include libdrmaa.so.  If you had a libdrmaa.so, the question would still 
>>>>> be open, as DRMAA has not been tested on Windows, hence why we don't 
>>>>> include it.  If you had a tested libdrmaa.so, I fear that the answer may 
>>>>> still be 'no', because non-SFU Windows applications cannot load SFU 
>>>>> shared libraries.  The current DRMAA implementation relies on the 
>>>>> libdrmaa.so, so I strongly suspect that the native Windows Java virtual 
>>>>> machine would be able to load the required DRMAA libraries.  (Harald, 
>>>>> can you confirm or deny?)
>>>>> 
>>>>> At some point in the future (after 6.2), we will very likely have a pure 
>>>>> Java implementation of DRMAA.  At that point, it won't matter where you 
>>>>> run it.
>>>>> 
>>>>> Daniel
>>>>> 
>>>>> Kevin Swanson wrote:
>>>>>> Thanks. Actually, I was curious whether DRMAA could work at all on
>>>>>> Windows, not specifically native Windows. So, if I did install 
>>>>>> Microsoft
>>>>>> SFU on a Windows submit host, would I then be able to use DRMAA on that
>>>>>> Windows host to submit and interact with jobs?
>>>>>> 
>>>>>> Kevin -----Original Message-----
>>>>>> From: Andreas.Haas at Sun.COM [mailto:Andreas.Haas at Sun.COM] Sent: 
>>>>>> Thursday, September 27, 2007 5:53 AM
>>>>>> To: users at gridengine.sunsource.net
>>>>>> Subject: Re: [GE users] Why no DRMAA support for Windows?
>>>>>> 
>>>>>> Hi Kevin,
>>>>>> 
>>>>>> On Wed, 26 Sep 2007, Kevin Swanson wrote:
>>>>>> 
>>>>>> 
>>>>>>> I saw in the Grid Engine 6.1 installation guide that DRMAA is not
>>>>>>> supported on Windows hosts. I had hoped to set up DRMAA (Java binding)
>>>>>>> on a Windows submit host and send jobs off to Linux execution hosts.
>>>>>>> 
>>>>>> 
>>>>>> DRMAA support for Windows were doable, if the corresponding Grid Engine 
>>>>>> client-side source code were ported to native Windows. Native Windows 
>>>>>> porting is needed, if the Java DRMAA binding shall be usable not only 
>>>>>> by
>>>>>> 
>>>>>> SFU-programs. Although Java DRMAA binding for Windows SFU programs were 
>>>>>> easier to accomplish, but I fear this is not what you envision.
>>>>>> 
>>>>>> 
>>>>>>> Does anyone know why DRMAA is not supported on Windows? Are there any
>>>>>>> plans to support it?
>>>>>>> 
>>>>>> 
>>>>>> Currently there are no unfortunately no plans to do the port. That
>>>>>> however could change, if you could help pushing and pulling as to make 
>>>>>> the
>>>>>> demand more visible and aware. If that doesn't work last resort would 
>>>>>> be to
>>>>>> pray for it.
>>>>>> 
>>>>>> Best regards,
>>>>>> Andreas
>>>>>> 
>>>>>> http://gridengine.info/
>>>>>> 
>>>>>> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551
>>>>>> Kirchheim-Heimstetten
>>>>>> Amtsgericht Muenchen: HRB 161028
>>>>>> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
>>>>>> Vorsitzender des Aufsichtsrates: Martin Haering
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>>>> 
>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>>> 
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>> 
>>>> 
>>> 
>>> http://gridengine.info/
>>> 
>>> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 
>>> Kirchheim-Heimstetten
>>> Amtsgericht Muenchen: HRB 161028
>>> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
>>> Vorsitzender des Aufsichtsrates: Martin Haering
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>> 
>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>> 
>> 
>
> http://gridengine.info/
>
> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 
> Kirchheim-Heimstetten
> Amtsgericht Muenchen: HRB 161028
> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
> Vorsitzender des Aufsichtsrates: Martin Haering
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list