[GE users] [OT] SGE and Eucayptus clouds

craffi dag at sonsorol.org
Wed Apr 7 18:22:50 BST 2010

As far as I can tell Eucalyptus is just an open-source reimplementation 
of the public APIs that amazon exposes for their EC2 service (just like 
hadoop is an implementation of how google described map reduce in a 
technical paper ...). I would be surprised if it could do any scheduling 
at all in fact if it's main goal is to replicate EC2 APIs.

It's trivial to deploy SGE in the Amazon cloud but it's really not the 
"best practice" way of doing work on infrastructure cloud platforms. We 
use it mainly for running legacy software and pipelines where we either 
can't switch things around or we don't think it's worth the effort.

The cloud systems are different enough from an infrastructure and 
service perspective:

	- built for massive internet-scale operations
	- organized around an eventual consistency model
  	- lots of loosely coupled distributed systems
	- systems operate asynchronously and independently
	- statekeeping via message passing & queue systems
	- dedicated resources for tasks rather than shared systems

The "cloud way" to do things is quite different from the SGE model.

If you were looking to schedule virtual machines on Amazon I'd consider 
just using their already-present message passing, monitoring and 
auto-scaling tools to do so. Just make your amazon AMI images your 
"virtual machine" and use whatever metrics you want to decide when (and 
how many) of those VMs/amis to launch/run/stop at any given time.

Personally I won't touch Eucalyptus. I am not interested in using any 
IaaS provider that has less than say 200,000 CPU cores under management. 
If a vendor is doing cloud stuff on a smaller scale then I simply don't 
trust that they have seen and solved all of the engineering, technical, 
availability and management issues seen at high-scale.

More ranting:

Running an "internal cloud" in 2010 is just silly for the majority of 
public use cases I've seen. 90% of the "private cloud" stuff right now 
is just sick marketers (same people who ruined the term 'grid 
computing') rebranding the same boring stuff that people already do 
today with VMWare and Xen. It's all marketing and nothing new or clever. 
The remaining 10% of "private cloud" fluff is coming from vendors who 
want to sell you a whole new datacenter's worth of virtualized IT, 
storage and networking gear.


rayson wrote:
> Does anyone know the scheduling ability of Eucalyptus??
> I went through a few persentations on the Eucalyptus website, but
> could not find much info about the internals of Eucalyptus.
> I tried EC2 and RightScale last fall, and I think the scheduling
> capability of EC2 is rather limited. I was wondering if SGE can be
> used to schedule virtual machines, and some sort of SGE/Eucalyptus
> integration might create a powerful cloud computing platform.
> Rayson
> On 3/12/10, doraz<dobre.razvan at gmail.com>  wrote:
>> Hello
>> Well it depends on what you are going to do: a dynamic infrastructure with
>> custom made virtual machines witch configuration changes often or just a
>> mini cluster to run jobs on it.
>> If you can be more pricese on what you want to do, may be I can come with a
>> solution. Right know I'm triying to put toghether SGE + SDM + Eucalyptus for
>> my barchelot degree.
>> Razvan Dobre
>> http://www.doraz.ro
>> -----Original Message-----
>> From: seandavi [mailto:seandavi at gmail.com]
>> Sent: Friday, March 12, 2010 6:56 PM
>> To: users at gridengine.sunsource.net
>> Subject: [GE users] [OT] SGE and Eucayptus clouds
>> We are a government lab and paying for Amazon cloud time is likely to
>> be problematic.  I wondered if anyone had thoughts or experience with
>> using SGE's new abilities to add nodes dynamically to a private cloud,
>> such as a Eucalyptus cloud (http://open.eucalyptus.com/)?  We don't
>> have such a cloud in place, but I would like to know what other folks
>> are thinking about the issues of SGE and private/institutional clouds.
>> Thanks,
>> Sean
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=2
>> 48185
>> To unsubscribe from this discussion, e-mail:
>> [users-unsubscribe at gridengine.sunsource.net].
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=248210
>> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=252582
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

More information about the gridengine-users mailing list