Re: [Moosefs-users] MooseFS vs GlusterFS

[Reinis R. <r...@ro...>]

> I’ve been asked to compare GlusterFS with MooseFS, please feel free to comment if you agree or have more info to add!

- One of the main drawbacks (or feature - depends how you look at it) in my opinion  is the way how each of the products achieve 
file system consistency.

GlusterFS - has the the "lazy replication" approach - e.g. it actually doesn't actively (at least till 3.2.x) check if the file has 
enough copies/replicas until you access that particular file or directory.
In case of rarely accessed files it can lead to missing data if one (or more) storage nodes fail but the files on the "mirror" nodes 
are not accessed therefore no spare copies are made to another bricks. So if the node having the single copy fails the file is gone.
While the targeted self heal "solution" exists ( 
http://community.gluster.org/a/howto-targeted-self-heal-repairing-less-than-the-whole-volume/ ) if the filesystem is huge it can be 
problematic.

MooseFS on the other hand keeps online data about each chunk and tries to ensure enough copies exist at all times.

The only tricky thing (at least for me) was to adjust the replication speed so it doesn't take too long (replicating million files 
over a month in case of 2 copies can lead to the same result as with glusterfs) but isn't also too aggressive to cripple the normal 
operations.


- Second drawback (or feature again) is the centralised Moosefs meta server.

While for file meta operations it gives superior speeds compared to gluster (for each stat() (not cached) needs to physically access 
each file on backends) it doesn't scale very well (no way to split) - if your file system starts to grow over 100 million files the 
metadata file and the memory consumption can go beyond what a single server instance can handle.

This approach also makes the moosefs component startup quite slow (both meta and the storage nodes) - the chunkservers have to 
traverse and checksum each chunk versus gluster where the cluster (nodes) can be onlined/downed pretty much instantly.


My 2 cents.

rr