[GE users] Why no DRMAA support for Windows?

Andy Schwierskott andy.schwierskott at sun.com
Fri Sep 28 12:49:39 BST 2007


Andreas,

[this really is a developer mailing list topic now but since many
technically interested people are just subscribed to the users mailing list
but are curious to know what we are doing and what might be done I'm abusing
this alias to share our ideas]

> sounds nice forehand, but it does not answer the question how to provide a 
> Windows dll DRMAA library then? Though I agree Java is not unimportant, but a 
> traditional DRMAA library still is needed for C and C++ programs and as base 
> for the entire scripting lanuguages world: Perl/Python/Ruby/etc.

that's true - I have no good answer yet for this challenge. Either the SUA
environment in Windows Vista will allow to create and load Windows DLLs
built under a SUA development environment or we get some other great ideas.
Building a native Windows JAPI layer (which would include porting of tons of
Unix SGE code including the commlib to Windows) on which DRMAA is sitting is
something I don't see a realistic chance that we (i.e. the Sun developmers)
can and will do in the forseeable future.

Perhaps another approach is strategically much more interesting to pursue
and even could attract developers from the community: with making the
scheduler a thread and loading a JVM we added a new internal communication
interface. In 6.1 and before the GDI communication (that's how we call
somewhat sloppily the interface/API how clients communicate with qmaster;
qmaster processes these requests in so called GDI threads) between qmaster
and scheduler was done over the network using the the communication library
(commlib). The scheduler thread and the JVM thread will use a new internal
GDI interface which bypasses the commlib to get/send data to the qmaster
core. In addition we create an interface which allows to start/stop qmaster
threads at runtime.

With that infrastructure in place one could go ahead a create a thread which
implements most part of the JAPI and send the data over the network in a
must less SGE dependent format, e.g. XML. On the client side it would be
possible to implement a much thinner DRMAA library which would mostly
implement the DRMAA API calls. Such a library should be very easy to port to
non-Unix OS'es like Windows. I think that approach would open the
possibility to much more easily create bindings for other programming and
scripting languages for DRMAA and implement a general SGE API. Likely this
approach also could be attracive for those who request a web services access
to SGE...


Well, those of you who are now asking for dates when they can have a look
and try at the technical stuff we are currently doing, should now subscribe
to the "dev" mailing list and ask there.

Andy

> On Fri, 28 Sep 2007, Andy Schwierskott wrote:
>
>> Hi,
>> 
>> Dan is mentioning below we might have a pure Java implementation of DRMAA 
>> in
>> the future - so what's behind that vague statement?
>> 
>> For SGE 6.2 we are currently implementing a Mbean server running as a 
>> thread
>> in qmaster. This means qmaster can load a JVM when it starts up.
>> 
>> Currently most likely not known to most of you there's a unsupported and
>> largely undocumented libgdi.so/jgdi.jar part of SGE 6.1 which is a low 
>> level
>> access method (read, write, modify; I'm reluctant to call it API yet) to
>> most of the SGE data structures residing in qmaster. The big disadvantage 
>> of
>> that approach is that it requires to load the native libgdi.so into a Java
>> application - something which you would like to avoid if you want to 
>> contact
>> SGE thorugh app server. It also makes live for our Hededy engineers much
>> easier if the same JVM can contact multiple SGE cluster, possibly running
>> different versions or patch levels.
>> 
>> With the JVM running in qmaster the functional part of libgdi.so is moving
>> into qmaster and Java clients can contact qmaster without loading native
>> code. That's still not a true API (und the low level access will be an
>> unsupported and unstable part of SGE 6.2) because it's still low level 
>> acess
>> and it's still not DRMAA. However this approach opens the door to have a
>> pure Java DRMAA client which will base on that infrastructure.
>
>> 
>> 
>> Andy
>> 
>> On Thu, 27 Sep 2007, Daniel Templeton wrote:
>> 
>>> From a purely structural perspective, no.  The windows bundle doesn't 
>>> include libdrmaa.so.  If you had a libdrmaa.so, the question would still 
>>> be open, as DRMAA has not been tested on Windows, hence why we don't 
>>> include it.  If you had a tested libdrmaa.so, I fear that the answer may 
>>> still be 'no', because non-SFU Windows applications cannot load SFU shared 
>>> libraries.  The current DRMAA implementation relies on the libdrmaa.so, so 
>>> I strongly suspect that the native Windows Java virtual machine would be 
>>> able to load the required DRMAA libraries.  (Harald, can you confirm or 
>>> deny?)
>>> 
>>> At some point in the future (after 6.2), we will very likely have a pure 
>>> Java implementation of DRMAA.  At that point, it won't matter where you 
>>> run it.
>>> 
>>> Daniel
>>> 
>>> Kevin Swanson wrote:
>>>> Thanks. Actually, I was curious whether DRMAA could work at all on
>>>> Windows, not specifically native Windows. So, if I did install Microsoft
>>>> SFU on a Windows submit host, would I then be able to use DRMAA on that
>>>> Windows host to submit and interact with jobs?
>>>> 
>>>> Kevin -----Original Message-----
>>>> From: Andreas.Haas at Sun.COM [mailto:Andreas.Haas at Sun.COM] Sent: Thursday, 
>>>> September 27, 2007 5:53 AM
>>>> To: users at gridengine.sunsource.net
>>>> Subject: Re: [GE users] Why no DRMAA support for Windows?
>>>> 
>>>> Hi Kevin,
>>>> 
>>>> On Wed, 26 Sep 2007, Kevin Swanson wrote:
>>>> 
>>>> 
>>>>> I saw in the Grid Engine 6.1 installation guide that DRMAA is not
>>>>> supported on Windows hosts. I had hoped to set up DRMAA (Java binding)
>>>>> on a Windows submit host and send jobs off to Linux execution hosts.
>>>>> 
>>>> 
>>>> DRMAA support for Windows were doable, if the corresponding Grid Engine 
>>>> client-side source code were ported to native Windows. Native Windows 
>>>> porting is needed, if the Java DRMAA binding shall be usable not only by
>>>> 
>>>> SFU-programs. Although Java DRMAA binding for Windows SFU programs were 
>>>> easier to accomplish, but I fear this is not what you envision.
>>>> 
>>>> 
>>>>> Does anyone know why DRMAA is not supported on Windows? Are there any
>>>>> plans to support it?
>>>>> 
>>>> 
>>>> Currently there are no unfortunately no plans to do the port. That
>>>> however could change, if you could help pushing and pulling as to make 
>>>> the
>>>> demand more visible and aware. If that doesn't work last resort would be 
>>>> to
>>>> pray for it.
>>>> 
>>>> Best regards,
>>>> Andreas
>>>> 
>>>> http://gridengine.info/
>>>> 
>>>> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551
>>>> Kirchheim-Heimstetten
>>>> Amtsgericht Muenchen: HRB 161028
>>>> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
>>>> Vorsitzender des Aufsichtsrates: Martin Haering
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>> 
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>> 
>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>> 
>> 
>
> http://gridengine.info/
>
> Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 
> Kirchheim-Heimstetten
> Amtsgericht Muenchen: HRB 161028
> Geschaeftsfuehrer: Marcel Schneider, Wolfgang Engels, Dr. Roland Boemer
> Vorsitzender des Aufsichtsrates: Martin Haering
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list