[MooseFS-Users] Performance suggestions for millions of small files

[Wilson, S. M <st...@pu...>]

Hi,


We have ten different MooseFS installations in our research group and one, in particular, is struggling with poor I/O performance.  This installation currently has 315 million files occupying 170TB of disk space (goal = 2).  If anyone else has a similar installation, I would like to hear what you have done to maintain performance at a reasonable level.


Here are some metrics to give a basic idea of the performance characteristics.  I'll include in parentheses the range of measurements from other MFS installations with far fewer files for comparison.

    * tar xf linux-4.9-rc3.tar: 1185 secs (220 - 296 secs)

    * smallfile test, create MB/s: 0.8 (2.3 - 4.8)  <== Ouch!

    * smallfile test, read MB/s: 10.7 (12.8 - 15.4)

    * smallfile test, append MB/s: 6.1 (3.0 - 7.7)


It looks file creation is where I'm losing most of my performance compared to the other installations.  My master server has a Xeon E5-1630v3 3.7GHz CPU with 256GB of DDR4 2133MHz memory.


I tried several mfsmount options but the only one that showed any significant improvement was the mfsfsyncmintime option ("mfsfsyncmintime=5").  As to be expected, the improvement gained was during the write/append operation.  Here are the results using the same tests as above:

    * tar xf linux-4.9-rc3.tar: 683 secs
    * smallfile test, create MB/s: 1.2
    * smallfile test, read MB/s: 11.7
    * smallfile test, append MB/s: 11.4  <== Dramatic improvement over 6.1 MB/s


The smallfile benchmark test I used is ?from https://github.com/distributed-system-analysis/smallfile.


Thanks for any suggestions you might have!


Regards,

Steve