[GE users] Hadoop and SGE

templedf dan.templeton at sun.com
Tue Mar 17 16:58:00 GMT 2009


I actually started working on one, but I ran out of free time.  There's 
a Hadoop on Demand project that was intended to be an integration 
between Hadoop and PBS.  Since SGE and PBS are so close in interface, it 
shouldn't be hard to add a module for SGE.  I started that work, but 
didn't get too far.

As I understand it, the HoD project is dead in the water because PBS 
wasn't able to schedule effectively around the data.  Rather than 
sending the Hadoop jobs to the machines where the data was already 
staged, it would make Hadoop restage its data every time.  I'm pretty 
sure we can overcome that in SGE with clever use of complexes, but we 
first need the integration working.

Daniel

heywood wrote:
> Has anyone tried an integration of Hadoop with SGE?
>
> Todd Heywood
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=134441
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=134445

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list