Re: [MooseFS-Users] Performance suggestions for millions of small files

[Wilson, S. M <st...@pu...>]

?Thanks for the suggestion!  I had thought about that also but a lot of our files need to be accessed simultaneously from different clients and, as you mentioned, this approach doesn't support that.


Steve



________________________________
From: Alexander AKHOBADZE <ale...@op...>
Sent: Friday, October 19, 2018 1:55 AM
To: Wilson, Steven M; MooseFS-Users
Subject: RE: Performance suggestions for millions of small files


Hi Steve!

You know... I'v made a conclution not store millions of small files on MooseFS ;--)
In such case I use a big file stored on MooseFS as container, format it as EXT4 or XFS and then mount it on client side.
Yes. I know that in such case small files are not clustered anymore ... it's a pity but here you are.

Wbr
Alexander (Anri) Akhobadze, ale...@op...
System administrator, DATAVISION NN

From: Wilson, Steven M [mailto:st...@pu...]
Sent: Thursday, October 18, 2018 11:47 PM
To: MooseFS-Users
Subject: [MooseFS-Users] Performance suggestions for millions of small files


Hi,



We have ten different MooseFS installations in our research group and one, in particular, is struggling with poor I/O performance.  This installation currently has 315 million files occupying 170TB of disk space (goal = 2).  If anyone else has a similar installation, I would like to hear what you have done to maintain performance at a reasonable level.



Here are some metrics to give a basic idea of the performance characteristics.  I'll include in parentheses the range of measurements from other MFS installations with far fewer files for comparison.

    * tar xf linux-4.9-rc3.tar: 1185 secs (220 - 296 secs)

    * smallfile test, create MB/s: 0.8 (2.3 - 4.8)  <== Ouch!

    * smallfile test, read MB/s: 10.7 (12.8 - 15.4)

    * smallfile test, append MB/s: 6.1 (3.0 - 7.7)



It looks file creation is where I'm losing most of my performance compared to the other installations.  My master server has a Xeon E5-1630v3 3.7GHz CPU with 256GB of DDR4 2133MHz memory.



I tried several mfsmount options but the only one that showed any significant improvement was the mfsfsyncmintime option ("mfsfsyncmintime=5").  As to be expected, the improvement gained was during the write/append operation.  Here are the results using the same tests as above:
    * tar xf linux-4.9-rc3.tar: 683 secs
    * smallfile test, create MB/s: 1.2
    * smallfile test, read MB/s: 11.7
    * smallfile test, append MB/s: 11.4  <== Dramatic improvement over 6.1 MB/s



The smallfile benchmark test I used is ?from https://github.com/distributed-system-analysis/smallfile.



Thanks for any suggestions you might have!



Regards,

Steve