Re: [Moosefs-users] Anybody experienced MooseFS as a mail server's storage

[WK <wk...@bn...>]

We have several MooseFS clusters each with 20,000-50,000 users for IMAP 
Maildir storage for a few months now.

We have been very happy with the results to date, compared to our old 
solutions.

Comments:

Each cluster has at least 4 chunkservers, some of which are the 
metaloggers. Pay attention to the RAM requirements, as a chunkserver 
that goes into swap is going to slow things down and MooseFS chunks eat 
RAM. The chunkservers aren't very powerful CPU wise, generally older 
Opterons that were repurposed and SATA drives.

We use dual power supply ECC RAM servers for the Masters, with lots of 
RAM. You don't want to lose your Master, though the one time we did have 
an issue, we were able to recover from the metalogger with very very 
minimal loss, which mostly amounted to the files that were 'on the fly' 
at the time of the failure. You'd see the same thing with an NFS server.

We use a Goal of 3 because if a chunk server dies it can take weeks for 
it to rebalance. Knowing there are still two copies during the rebalance 
gives one peace of mind.

We set the trashtime to offset 12 hours because if you delete a large 
account with hundreds of thousands of files in their folders, when the 
trashtime expires, the mount performance is really affected as the 
server tries to delete all those chunks and their "goal" copies. With an 
offset of 12 hours that generally happens at a less busy time (Note: I 
filed a bug describing the problem on SourceForge. They really need some 
sort of way to 'nice' those large deletions).

MooseFS is not efficient with small files in regards to storage size. 
You still get the same size chunk no matter the file size. So between 
the goal setting and the chunk size expect a 5x-7x increase in your 
total storage requirement over a jbod disk setup. Our attitude is that 
SATA hard disks are cheap and due to the goal replication of three its 
not that big a deal if one dies. Certainly not worth paying 5x for 
Enterprise drives in this situation.

We use multiple MooseFS Clusters because we are paranoid and didn't want 
to put all our eggs in one basket, however, we would be comfortable 
expanding the original clusters as the need occurs by adding 
additional/bigger drives and or chunkservers.

-WK


On 7/26/11 2:26 AM, Mostafa Rokooie wrote:
> I'm going to use MooseFS as my mail server's backend storage, I did 
> many research to find a suitable fault tolerant distributed file 
> system, finally I think that MooseFS can be a good choice for me, 
> We're going to have large amount of users (100,000+).
> Is here anybody who experienced MooseFS as a mail server's storage?
>
>
> ------------------------------------------------------------------------------
> Magic Quadrant for Content-Aware Data Loss Prevention
> Research study explores the data loss prevention market. Includes in-depth
> analysis on the changes within the DLP market, and the criteria used to
> evaluate the strengths and weaknesses of these DLP solutions.
> http://www.accelacomm.com/jaw/sfnl/114/51385063/
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users