[GE users] A patch to run HadoopOnDemand on Grid Engine clusters

templedf dan.templeton at sun.com
Wed Nov 11 19:44:30 GMT 2009


In a nutshell, HoD allows you to allocate a Hadoop cluster in a Grid 
Engine cluster dynamically and then submit jobs against it.  Starting 
with 6.2u5, Grid Engine also natively will have the ability to support a 
Hadoop cluster as a PE.  The two solutions are complimentary, but are 
currently unaware of each other.  I am in the process of documenting the 
u5 integration, and I think it will cover most of the more general 
questions you posed.  For the HoD-specific stuff, I doubt there's much 
to document.  The idea of HoD is to abstract the user from the DRM.

Daniel

craffi wrote:
> Hi Gianluigi,
>
> Do you have any additional documentation or examples of this in action? 
> Beyond the patch it would be intesting to see a writeup of what the 
> patch does and how (in general) hadoop works on shared clusters and 
> compute farms.
>
>
> -Chris
>
>
> zag wrote:
>   
>> Hi there.
>>
>> We developed and tested a simple patch that adds Grid Engine support to
>> HOD. The patch is available as an attachment to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-6369
>>
>> After patching, HOD passes all unit tests at our site. We tested the
>> SGE driver on a production cluster consisting of about 400 nodes.
>>
>> *NOTE*: the patch is for the official hadoop-0.20.1 HOD release. It
>> should be easy, however, to generate patches for other versions.
>>
>> To patch, simply download hod_ge.patch from the above JIRA link and run:
>>
>> cd ${HADOOP_HOME}/src/contrib&&  patch -p0<hod_ge.patch
>>
>> Happy hadoop-ing,
>>
>> --gianluigi
>>
>>
>>     
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=226221
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=226223

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

    [ Part 2, "dan.templeton.vcf"  Text/X-VCARD (Name: ]
    [ "dan.templeton.vcf") ~145 bytes. ]
    [ Unable to print this part. ]



More information about the gridengine-users mailing list