Re: [Moosefs-users] Small files real size

[Michal B. <mic...@ge...>]

Hi Samuel!

Please also have a look at this answer:
http://www.moosefs.org/moosefs-faq.html#source_code


Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01




-----Original Message-----
From: wk...@bn... [mailto:wk...@bn...] 
Sent: Monday, May 30, 2011 2:08 AM
To: moo...@li...
Subject: Re: [Moosefs-users] Small files real size

On 5/29/11 7:05 AM, Samuel Hassine, Olympe Network wrote:
> Hi all,
>
> I know that MooseFS chunks take more disk space than the realsize of the
> files. But for us, it become very critical :
>
> root@on-003:~# mfsdirinfo -h /dns
> /dns:
>    inodes:         15Mi
>     directories:  1.9Mi
>     files:         13Mi
>    chunks:         13Mi
>    length:       247GiB
>    size:         1.0TiB
>    realsize:     2.1TiB
>
> 1TB for 247GB of real files... and with a goal of 2, we are using 2TB...
>

You have 13 Million files each of which has a minimum 64K fixed block 
size in MooseFS (plus some overhead). The devs feel that is the optimum 
block for their purposes and are unlikely to change that.

Here is a typical mount for one our dedicated customer imap servers.

  inodes:        1.2Mi
   directories:   17Ki
   files:        1.2Mi
  chunks:        1.2Mi
  length:       111GiB
  size:         183GiB
  realsize:     550GiB

Note: We are in the process of increasing all our mount goals from 2 to 3

So taking the storage size increase hit is part of the tradeoff using MFS.

Fortunately, MFS makes it easy to increase the pool size, either by 
adding chunkservers and/or adding drives to a chunkserver.  In my part 
of the world 1TB drives can be had for US$59  and 2TB are usually below 
US$100 (though we don't quite trust those yet after having some horrific 
experiences a year or so ago), so we happily trade off getting an extra 
drive or so for the ease of maintenance and speed of the MFS cluster vs 
the other high availability solutions we have tried (DRBD, GFS2/OCFS 
clusters, Gluster, etc).

-bill



----------------------------------------------------------------------------
--
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery, 
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now. 
http://p.sf.net/sfu/quest-d2dcopy1
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users