[GE users] Why no DRMAA support for Windows?

Andreas.Haas at Sun.COM Andreas.Haas at Sun.COM
Fri Sep 28 14:11:26 BST 2007


Andy,

On Fri, 28 Sep 2007, Andy Schwierskott wrote:

> Andreas,
>
> [this really is a developer mailing list topic now but since many
> technically interested people are just subscribed to the users mailing list
> but are curious to know what we are doing and what might be done I'm abusing
> this alias to share our ideas]
>
>> sounds nice forehand, but it does not answer the question how to provide a 
>> Windows dll DRMAA library then? Though I agree Java is not unimportant, but 
>> a traditional DRMAA library still is needed for C and C++ programs and as 
>> base for the entire scripting lanuguages world: Perl/Python/Ruby/etc.
>
> that's true - I have no good answer yet for this challenge. Either the SUA
> environment in Windows Vista will allow to create and load Windows DLLs
> built under a SUA development environment or we get some other great ideas.
> Building a native Windows JAPI layer (which would include porting of tons of
> Unix SGE code including the commlib to Windows) on which DRMAA is sitting is
> something I don't see a realistic chance that we (i.e. the Sun developmers)
> can and will do in the forseeable future.

"Tons of Unix source code" is an undue exeggeration. Almost the entire 
code is ANSI C. Only exceptions is threading and TCP code, but there is 
no doubt it can be ported.

> Perhaps another approach is strategically much more interesting to pursue
> and even could attract developers from the community: with making the
> scheduler a thread and loading a JVM we added a new internal communication
> interface. In 6.1 and before the GDI communication (that's how we call
> somewhat sloppily the interface/API how clients communicate with qmaster;
> qmaster processes these requests in so called GDI threads) between qmaster
> and scheduler was done over the network using the the communication library
> (commlib). The scheduler thread and the JVM thread will use a new internal
> GDI interface which bypasses the commlib to get/send data to the qmaster
> core. In addition we create an interface which allows to start/stop qmaster
> threads at runtime.
>
> With that infrastructure in place one could go ahead a create a thread which
> implements most part of the JAPI and send the data over the network in a
> must less SGE dependent format, e.g. XML. On the client side it would be
> possible to implement a much thinner DRMAA library which would mostly
> implement the DRMAA API calls. Such a library should be very easy to port to
> non-Unix OS'es like Windows. I think that approach would open the
> possibility to much more easily create bindings for other programming and
> scripting languages for DRMAA and implement a general SGE API. Likely this
> approach also could be attracive for those who request a web services access
> to SGE...

Would you allow me two questions on this thinner DRMAA library?

(1) How would you ensure it can be used concurrently by multiple threads? 
Wouldn't this again require Windows threading library calls be used?

(2) How you connect to your visualized JAPI-thread? Wouldn't it again 
use TCP?

Bottom-line: You gain nothing, but significant delay on the availability 
of DRMAA for Windows.

Regards,
Andreas

>
>
> Well, those of you who are now asking for dates when they can have a look
> and try at the technical stuff we are currently doing, should now subscribe
> to the "dev" mailing list and ask there.
>
> Andy
>
>> On Fri, 28 Sep 2007, Andy Schwierskott wrote:
>> 
>>> Hi,
>>> 
>>> Dan is mentioning below we might have a pure Java implementation of DRMAA 
>>> in
>>> the future - so what's behind that vague statement?
>>> 
>>> For SGE 6.2 we are currently implementing a Mbean server running as a 
>>> thread
>>> in qmaster. This means qmaster can load a JVM when it starts up.
>>> 
>>> Currently most likely not known to most of you there's a unsupported and
>>> largely undocumented libgdi.so/jgdi.jar part of SGE 6.1 which is a low 
>>> level
>>> access method (read, write, modify; I'm reluctant to call it API yet) to
>>> most of the SGE data structures residing in qmaster. The big disadvantage 
>>> of
>>> that approach is that it requires to load the native libgdi.so into a Java
>>> application - something which you would like to avoid if you want to 
>>> contact
>>> SGE thorugh app server. It also makes live for our Hededy engineers much
>>> easier if the same JVM can contact multiple SGE cluster, possibly running
>>> different versions or patch levels.
>>> 
>>> With the JVM running in qmaster the functional part of libgdi.so is moving
>>> into qmaster and Java clients can contact qmaster without loading native
>>> code. That's still not a true API (und the low level access will be an
>>> unsupported and unstable part of SGE 6.2) because it's still low level 
>>> acess
>>> and it's still not DRMAA. However this approach opens the door to have a
>>> pure Java DRMAA client which will base on that infrastructure.
>> 
>>> 
>>> 
>>> Andy
>>> 
>>> On Thu, 27 Sep 2007, Daniel Templeton wrote:
>>> 
>>>> From a purely structural perspective, no.  The windows bundle doesn't 
>>>> include libdrmaa.so.  If you had a libdrmaa.so, the question would still 
>>>> be open, as DRMAA has not been tested on Windows, hence why we don't 
>>>> include it.  If you had a tested libdrmaa.so, I fear that the answer may 
>>>> still be 'no', because non-SFU Windows applications cannot load SFU 
>>>> shared libraries.  The current DRMAA implementation relies on the 
>>>> libdrmaa.so, so I strongly suspect that the native Windows Java virtual 
>>>> machine would be able to load the required DRMAA libraries.  (Harald, can 
>>>> you confirm or deny?)
>>>> 
>>>> At some point in the future (after 6.2), we will very likely have a pure 
>>>> Java implementation of DRMAA.  At that point, it won't matter where you 
>>>> run it.
>>>> 
>>>> Daniel
>>>> 
>>>> Kevin Swanson wrote:
>>>>> Thanks. Actually, I was curious whether DRMAA could work at all on
>>>>> Windows, not specifically native Windows. So, if I did install Microsoft
>>>>> SFU on a Windows submit host, would I then be able to use DRMAA on that
>>>>> Windows host to submit and interact with jobs?
>>>>> 
>>>>> Kevin -----Original Message-----
>>>>> From: Andreas.Haas at Sun.COM [mailto:Andreas.Haas at Sun.COM] Sent: Thursday, 
>>>>> September 27, 2007 5:53 AM
>>>>> To: users at gridengine.sunsource.net
>>>>> Subject: Re: [GE users] Why no DRMAA support for Windows?
>>>>> 
>>>>> Hi Kevin,
>>>>> 
>>>>> On Wed, 26 Sep 2007, Kevin Swanson wrote:
>>>>> 
>>>>> 
>>>>>> I saw in the Grid Engine 6.1 installation guide that DRMAA is not
>>>>>> supported on Windows hosts. I had hoped to set up DRMAA (Java binding)
>>>>>> on a Windows submit host and send jobs off to Linux execution hosts.
>>>>>> 
>>>>> 
>>>>> DRMAA support for Windows were doable, if the corresponding Grid Engine 
>>>>> client-side source code were ported to native Windows. Native Windows 
>>>>> porting is needed, if the Java DRMAA binding shall be usable not only by
>>>>> 
>>>>> SFU-programs. Although Java DRMAA binding for Windows SFU programs were 
>>>>> easier to accomplish, but I fear this is not what you envision.
>>>>> 
>>>>> 
>>>>>> Does anyone know why DRMAA is not supported on Windows? Are there any
>>>>>> plans to support it?
>>>>>> 
>>>>> 
>>>>> Currently there are no unfortunately no plans to do the port. That
>>>>> however could change, if you could help pushing and pulling as to make 
>>>>> the
>>>>> demand more visible and aware. If that doesn't work last resort would be 
>>>>> to
>>>>> pray for it.
>>>>> 
>>>>> Best regards,
>>>>> Andreas
>>>>> 
>>>>> http://gridengine.info/
>>>>> 
>>>>> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551
>>>>> Kirchheim-Heimstetten
>>>>> Amtsgericht Muenchen: HRB 161028
>>>>> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
>>>>> Vorsitzender des Aufsichtsrates: Martin Haering
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>>> 
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>> 
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>> 
>>> 
>> 
>> http://gridengine.info/
>> 
>> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 
>> Kirchheim-Heimstetten
>> Amtsgericht Muenchen: HRB 161028
>> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
>> Vorsitzender des Aufsichtsrates: Martin Haering
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>> 
>> 
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>

http://gridengine.info/

Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten
Amtsgericht Muenchen: HRB 161028
Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
Vorsitzender des Aufsichtsrates: Martin Haering

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list