Menu
▾
▴
Re: [Moosefs-users] Small file performance is an order of magnitude worse than large files
|
From: Robert S. <rsa...@ne...> - 2011-08-03 11:57:36
|
I think the original question was about single disk IOPS. The moment you add multiple disks then effective IOPS will increase. But there is always a penalty. For MooseFS it is the network I/O. For RAID5/6 it is the calculation of the error correcting codes and the extra data that has to be read and written for every write. But even assuming you have a 100 x 7200 RPM spindles and the files are perfectly distributed with no overhead then with 10 kB files you will only get a 100 MB/s write speed under perfect conditions. In reality you will see much lower numbers. Adding more spindles and/or faster spindles is the only way to improve the speed of dealing with many small files but there are other limits/overhead which limits the gain you will get. Robert On 8/3/11 6:11 AM, Dennis Jacobfeuerborn wrote: > I would think that the iops issue is somewhat mitigated by the parallelism > of MooseFS since iops don't have to be processed in strictly sequential order. > On the negative side there are a lot of round-trips between the servers > involved for writes so if you have a lot of them they are going to be > slower than on local storage. > > Regards, > Dennis > > On 08/03/2011 01:39 AM, Robert Sandilands wrote: >> It is not really an issue of MFS or not. You will find the same performance >> issues on EXT2/3/4 and XFS. >> >> Hard drives and most file systems are made for streaming large files. Not >> dealing with large numbers of small files. It helps to have enough memory >> on the machines mounting the file systems to keep the folder structure >> cached in memory. In that way MFS is good as it forces the whole folder >> structure to be in memory through mfsmaster. >> >> But, there seems to be other limitations that makes it hard to deal with >> large numbers of small files. >> >> One of the largest limitations of working with small files is the IOPS >> (Input/Output Operations Per Second) limits of the individual drives and >> the controllers. A 7200 RPM drive can do at most 100 IOPS per second. An >> IOPS is a seek and read or a write. To open a file and read/write it can be >> several operations to follow the folder structure and to identify the >> location on the disk and then to actually read/write it. This implies that >> if you are very optimistic you can create 100 files per disk per second. If >> a single file is only 10k then it means you can write at around 1 MB/s to >> disk on a single disk as a theoretical maximum. In the real world it will >> probably be less than 50% of that. RAID5 and RAID6 will have a >> significantly negative effect on IOPS for an array of disks. >> >> A 15 kRPM drive is limited to around 210 IOPS. >> >> This obviously ignores the effect of caching (OS, controller, disk) and >> reordering of operations. But if the total size of the data set you are >> working with is larger than the amount of RAM available for caching then >> caching will not have much of an effect. Reordering of reads and writes can >> have a positive influence on the effective IOPS so it may be useful to >> invest in a better quality controller card. >> >> This all assumes that you avoid some of the obvious pitfalls when dealing >> with large numbers of files. One of them is not to store too many files per >> folder. A lot of operating systems/file systems slow down significantly >> once they find more than a few thousand files in a single folder. >> >> If you find a way around these types of limits I would love to know about >> it. I have looked at using SSD drives due to their much higher IOPS but >> then you have to be careful of the IOPS limits on the controller cards. >> That and the per GB cost of the SSD drives is a bit problematic at the >> moment. I have also looked at using MySQL for storing files of this size >> and that has shown some potential but it did not really outperform anything >> else I looked at. It was also somewhat problematic to scale. >> >> Robert >> >> On 8/2/11 2:41 PM, Kristofer Pettijohn wrote: >>> MFS is designed for large files, as it was written as a clone to Google >>> File System. It will perform very poorly with small files. I don't think >>> there is much you can do. >>> >>> --------------------------------------------------------------------------- >>> *From: *"Vineet Jain"<vin...@gm...> >>> *To: *moo...@li... >>> *Sent: *Tuesday, August 2, 2011 10:03:26 AM >>> *Subject: *[Moosefs-users] Small file performance is an order of >>> magnitude worse than large files >>> >>> I have a test instance up with a 5 drives one master and one chunk >>> server. Large file performance is fine. However when testing with >>> 100,000's of small files (10k - 100k) I'm getting about 200-300k write >>> performance per drive from looking at the output of iostat. Is that sound >>> right? Any way to speed up the writes. It is painfully slow. >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >>> The must-attend event for mobile developers. Connect with experts. >>> Get tools for creating Super Apps. See the latest technologies. >>> Sessions, hands-on labs, demos& much more. Register early& save! >>> http://p.sf.net/sfu/rim-blackberry-1 >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >>> The must-attend event for mobile developers. Connect with experts. >>> Get tools for creating Super Apps. See the latest technologies. >>> Sessions, hands-on labs, demos& much more. Register early& save! >>> http://p.sf.net/sfu/rim-blackberry-1 >>> >>> >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> >> ------------------------------------------------------------------------------ >> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >> The must-attend event for mobile developers. Connect with experts. >> Get tools for creating Super Apps. See the latest technologies. >> Sessions, hands-on labs, demos& much more. Register early& save! >> http://p.sf.net/sfu/rim-blackberry-1 >> >> >> >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > ------------------------------------------------------------------------------ > BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA > The must-attend event for mobile developers. Connect with experts. > Get tools for creating Super Apps. See the latest technologies. > Sessions, hands-on labs, demos& much more. Register early& save! > http://p.sf.net/sfu/rim-blackberry-1 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
