[GE users] File server overloading

Joe Landman landman at scalableinformatics.com
Thu Jan 31 12:58:49 GMT 2008

    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Ron Chen wrote:
> Lustre is good, I heard many good feedbacks about it. And

We have heard some fairly mixed comments.  There are a few design issues 
(centralized non-replicated MDS) that are at a minimum, a significant 
concern to anyone wanting to make sure they have a live file system at 
all times.

For some workloads Lustre is fine.

> Lustre+Gigabit Eithernet should be good enough, Infiniband is
> fast and low latency but latency is mainly needed for parallel
> computing (e.g. MPI jobs with lows of small message exchange
> between tasks).

... or very high bandwidth storage.  You can get 800+ MB/s (B as in 
bytes, not "b" for bits) to a node from storage using NFSoverRDMA.

Add to this now, the inexpensive IB cards and switch, and frankly it 
gets difficult not to look at IB for storage systems.

> Also, pNFS (parallel NFS) is supposed to be better than
> traditional NFS too, anyone tried it in a compute farm
> environment?

pNFS is sadly, not yet ready for prime time (from an implementation 
view).  There are bits and CVS things floating around out there.

That said, there are some "up and coming" storage software systems: 
GlusterFS is interesting, and very fast.  It doesn't suffer the MDS 
issue of Lustre.  Our testing of it has been, well, interesting.

To the original poster, GFS is known to be slow.  If you need high 
performance parallel file systems, you are better advised to look 
elsewhere.  GFS was not designed for the high throughput/scalable file 
systems that people need more of today.  Lustre sort of is, but it has a 
few design issues that you should be wary of ... as long as you don't 
mind using it for fast temporary storage, Lustre is fine.  Permanent 
storage is another issue.  There are commercial offerings such as 
Polyview, Ibrix, etc.  Some of these scale ok, but will have a cost to 
them.  Then there are the integrated storage appliances (Isilon, DDN, 
etc...).  Those will scale and work well as safe and fast storage, and 
scale well.  But they will also cost you.

You have choices, not all of them free, but some of them are quite good.

>  -Ron

Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list