Re: [Moosefs-users] Best FileSystem for chunkers.

[Raymond J. <ray...@ca...>]

On 5/17/2012 12:26 PM, Atom Powers wrote:
> I use ZFS on FreeBSD, which is one of the main reasons I use FreeBSD on
> my chunk servers.

We also use FreeBSD here, due to the fact that ZFS works quite well at 
our scale (10-20TB zpools, 8 disks/zpool, one mfschunkserver invocation
per zpool).

Our current setup is to have cheap commodity nodes that boot FreeBSD
disklessly. We can easily provision more storage (it's about 10
commands and 30 minutes; we don't expand fast enough to automate
everything).

Admittedly, we have a small cluster (60TB), but we haven't run into
scaling issues so far.

>
> Good:
> * Compression, 1.16x in my environment
> * zraid
> * probably improved performance (I haven't done a comparison on MooseFS
> but saw better performance over UFS for "standard" file system use)
> * Easy to carve up for other uses on the same server

ZFS makes disk management a pleasure, compared to UFS. With labeled GPT
partitions and FreeBSD's built in GPT name support, it makes
administration a breeze.

Unfortunately most of our computing applications require a homogenous
Linux computing environment, so our storage nodes don't see other use.

> Bad:
> * high RAM requirement

RAM is cheap, and without deduplication/compression ZFS only takes as
much RAM as you tell it to (all of it by default, but it normally lies
unused on our storage nodes anyway).

> Ugly:
> * FreeBSD is tricky to build with bootable ZFS
> * Linux ZFS is FUSE.

We tried the Linux LLNL port when it came out back at the beginning of
last year, but the number of metadata lookups vs. data lookups was
killing ZFS performance and was causing instability. This was on a
smaller cluster of about 30TB, so we didn't scale up.

Combined with the better documentation and administrative features
of FreeBSD, ZFS was a no-brainer for us.

Raymond Jimenez