Re: [Moosefs-users] Small file performance is an order of magnitude worse than large files

[Robert S. <rsa...@ne...>]

It is not really an issue of MFS or not. You will find the same 
performance issues on EXT2/3/4 and XFS.

Hard drives and most file systems are made for streaming large files. 
Not dealing with large numbers of small files. It helps to have enough 
memory on the machines mounting the file systems to keep the folder 
structure cached in memory. In that way MFS is good as it forces the 
whole folder structure to be in memory through mfsmaster.

But, there seems to be other limitations that makes it hard to deal with 
large numbers of small files.

One of the largest limitations of working with small files is the IOPS 
(Input/Output Operations Per Second) limits of the individual drives and 
the controllers. A 7200 RPM drive can do at most 100 IOPS per second. An 
IOPS is a seek and read or a write. To open a file and read/write it can 
be several operations to follow the folder structure and to identify the 
location on the disk and then to actually read/write it. This implies 
that if you are very optimistic you can create 100 files per disk per 
second. If a single file is only 10k then it means you can write at 
around 1 MB/s to disk on a single disk as a theoretical maximum. In the 
real world it will probably be less than 50% of that. RAID5 and RAID6 
will have a significantly negative effect on IOPS for an array of disks.

A 15 kRPM drive is limited to around 210 IOPS.

This obviously ignores the effect of caching (OS, controller, disk) and 
reordering of operations. But if the total size of the data set you are 
working with is larger than the amount of RAM available for caching then 
caching will not have much of an effect. Reordering of reads and writes 
can have a positive influence on the effective IOPS so it may be useful 
to invest in a better quality controller card.

This all assumes that you avoid some of the obvious pitfalls when 
dealing with large numbers of files. One of them is not to store too 
many files per folder. A lot of operating systems/file systems slow down 
significantly once they find more than a few thousand files in a single 
folder.

If you find a way around these types of limits I would love to know 
about it. I have looked at using SSD drives due to their much higher 
IOPS but then you have to be careful of the IOPS limits on the 
controller cards. That and the per GB cost of the SSD drives is a bit 
problematic at the moment. I have also looked at using MySQL for storing 
files of this size and that has shown some potential but it did not 
really outperform anything else I looked at. It was also somewhat 
problematic to scale.

Robert

On 8/2/11 2:41 PM, Kristofer Pettijohn wrote:
> MFS is designed for large files, as it was written as a clone to 
> Google File System.  It will perform very poorly with small files.  I 
> don't think there is much you can do.
>
> ------------------------------------------------------------------------
> *From: *"Vineet Jain" <vin...@gm...>
> *To: *moo...@li...
> *Sent: *Tuesday, August 2, 2011 10:03:26 AM
> *Subject: *[Moosefs-users] Small file performance is an order of 
> magnitude        worse than large files
>
> I have a test instance up with a 5 drives one master and one chunk 
> server. Large file performance is fine. However when testing with 
> 100,000's of small files (10k - 100k) I'm getting about 200-300k write 
> performance per drive from looking at the output of iostat. Is that 
> sound right? Any way to speed up the writes. It is painfully slow.
>
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos & much more. Register early & save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos&  much more. Register early&  save!
> http://p.sf.net/sfu/rim-blackberry-1
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users