[GE users] Best cluster management software and OS

Hai Pham bob.cratermeister at gmail.com
Sat Nov 19 17:03:14 GMT 2005

    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Sebastien is right, we've strayed way off the original poster's
question.   Whether  you use distribution A or Y isn't really that
important at the end of the day.  The most important question you
should ask yourself is what do you want to do with the cluster once
it's done.  What software are you planning to run?  How much in house
expertise do you have to build the system if you plan to do it
yourself?  How much time do you have to spend on building it?  How
much money do you have for software?

A good complete answer to this question goes far further than just
Redhat is best or Gentoo is the fastest.  And it's way beyond whatever
"advice" that I might write about in this post.  For starters, the
fact that the original poster asked what open source OS to use, not
what Linux distribution to use makes me think that he/she might not
have as much experience with clusters and operating systems in general
as we're assuming.

First of all, if you are looking for open source operating systems for
clusters, Linux is really the only game in town.  Educate yourself --
posting to a website and asking isn't going to give you a magic
answer.  Go to Google and search for "beowulf clusters" and then read.
 Yes, unfortunately, RTFM.  There are lots of good stuff out there
about how to build a cluster.  The more you know about what's
involved, the better you'll be to make an educated decision about how
to build your environment.

Next, I think the biggest mistake that a lot of people make when it
comes to software is to ask "what can I do to get the most performance
out of the hardware".  I think that's the wrong question to start with
-- very few people need the absolute best performance from their
hardware.  The question that they should really ask is "what can I do
to get the most productivity out of the hardware".  Notice that they
are not really the same question.  It's like the difference between an
F1 car and a Toyota Corolla.  If you need to get groceries, an F1 car
might get you there faster, but you're going to have to spend a lot
more time and resources into it to get it going at all.  In the
meantime, you could have gone to the grocery store, cooked dinner and
put your kids to sleep in the amount of time that you it took you to
get your F1 car started.

This brings up the point about what Linux distribution to use. 
Everyone has their favourite Linux distro, each distro has it's pro
and con.  As someone who has been using Linux since version 0.11 (yes,
that was when Linus himself built the "distribution"), I have used
just about every major Linux distribution over the years.  None of
them are perfect, so the choice of which one to use will largely
depend on what you are used to and what you are planning to install on
top of it.  The pros and cons of each distro is a totally different
topic and everyone you ask will give you a different opinion about
their favourite.

The question you should be asking yourself in this case, is what
software am I planning to run on this cluster?  For example, we
decided not to try and roll our own clustering environment, but to buy
off the shelf to save time.  After some investigation, I decided that
Bioteam's iNquiry gave us a starting place that most closely resembles
what we wanted to do with our cluster.  iNquiry works best with SUSE
9.1, so that's the distro that we went with.

The next thing to consider is what cluster management system to use. 
If one or more of your applications require SGE, then there's your
answer.  If you decide to go with an off the shelf clustering
solution, it might already come with a management system built-in. 
Also, if you know someone with experience with one of the available
systems, don't discount their knowledge.  They are an extremely
valuable asset for the times when you get stuck.  Likewise, don't
forget about the community of users that are using a particular
clustering solution or management system.  When something breaks and
you need help, having a big community of users to draw on can save you
all sorts of time and headaches.  Personally, I tried a number of
different management systems before deciding that I liked SGE the most
-- mainly because it has a very active user base and it has most of
the features that I find useful, not because I thought it was the best
or because it gave me the best performance.

Do your due dilligence, the time you spend educating yourself now will
save you tons of time later!

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list