[GE users] A patch to run HadoopOnDemand on Grid Engine clusters

zag gianluigi.zanetti at gmail.com
Thu Nov 12 08:42:08 GMT 2009


Hi Chris.

I subscribe to Dan's nutshell description.
At any rate, see http://hadoop.apache.org/common/docs/r0.19.1/hod.html
for instructions on how to use hod. The GridEngine patch is essentially
transparent to what is describe in the manual, apart from some
configuration variables that are grid-engine specific such as
'parallel-environment' and 'slots-per-node'. See config.txt in the
patched hod directory and the example configuration hod/conf/hodrc.sge

--gianluigi




On Wed, 2009-11-11 at 11:44 -0800, templedf wrote:
> In a nutshell, HoD allows you to allocate a Hadoop cluster in a Grid 
> Engine cluster dynamically and then submit jobs against it.  Starting 
> with 6.2u5, Grid Engine also natively will have the ability to support a 
> Hadoop cluster as a PE.  The two solutions are complimentary, but are 
> currently unaware of each other.  I am in the process of documenting the 
> u5 integration, and I think it will cover most of the more general 
> questions you posed.  For the HoD-specific stuff, I doubt there's much 
> to document.  The idea of HoD is to abstract the user from the DRM.
> 
> Daniel
> 
> craffi wrote:
> > Hi Gianluigi,
> >
> > Do you have any additional documentation or examples of this in action? 
> > Beyond the patch it would be intesting to see a writeup of what the 
> > patch does and how (in general) hadoop works on shared clusters and 
> > compute farms.
> >
> >
> > -Chris
> >
> >
> > zag wrote:
> >   
> >> Hi there.
> >>
> >> We developed and tested a simple patch that adds Grid Engine support to
> >> HOD. The patch is available as an attachment to:
> >>
> >> https://issues.apache.org/jira/browse/HADOOP-6369
> >>
> >> After patching, HOD passes all unit tests at our site. We tested the
> >> SGE driver on a production cluster consisting of about 400 nodes.
> >>
> >> *NOTE*: the patch is for the official hadoop-0.20.1 HOD release. It
> >> should be easy, however, to generate patches for other versions.
> >>
> >> To patch, simply download hod_ge.patch from the above JIRA link and run:
> >>
> >> cd ${HADOOP_HOME}/src/contrib&&  patch -p0<hod_ge.patch
> >>
> >> Happy hadoop-ing,
> >>
> >> --gianluigi
> >>
> >>
> >>     
> >
> > ------------------------------------------------------
> > http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=226221
> >
> > To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
> >
> 
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=226223
> 
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=226405

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list