From: Alexander A. <ale...@op...> - 2018-10-19 06:11:26
|
Hi Steve! You know… I’v made a conclution not store millions of small files on MooseFS ;--) In such case I use a big file stored on MooseFS as container, format it as EXT4 or XFS and then mount it on client side. Yes. I know that in such case small files are not clustered anymore … it's a pity but here you are. Wbr Alexander (Anri) Akhobadze, ale...@op... System administrator, DATAVISION NN From: Wilson, Steven M [mailto:st...@pu...] Sent: Thursday, October 18, 2018 11:47 PM To: MooseFS-Users Subject: [MooseFS-Users] Performance suggestions for millions of small files Hi, We have ten different MooseFS installations in our research group and one, in particular, is struggling with poor I/O performance. This installation currently has 315 million files occupying 170TB of disk space (goal = 2). If anyone else has a similar installation, I would like to hear what you have done to maintain performance at a reasonable level. Here are some metrics to give a basic idea of the performance characteristics. I'll include in parentheses the range of measurements from other MFS installations with far fewer files for comparison. * tar xf linux-4.9-rc3.tar: 1185 secs (220 - 296 secs) * smallfile test, create MB/s: 0.8 (2.3 - 4.8) <== Ouch! * smallfile test, read MB/s: 10.7 (12.8 - 15.4) * smallfile test, append MB/s: 6.1 (3.0 - 7.7) It looks file creation is where I'm losing most of my performance compared to the other installations. My master server has a Xeon E5-1630v3 3.7GHz CPU with 256GB of DDR4 2133MHz memory. I tried several mfsmount options but the only one that showed any significant improvement was the mfsfsyncmintime option ("mfsfsyncmintime=5"). As to be expected, the improvement gained was during the write/append operation. Here are the results using the same tests as above: * tar xf linux-4.9-rc3.tar: 683 secs * smallfile test, create MB/s: 1.2 * smallfile test, read MB/s: 11.7 * smallfile test, append MB/s: 11.4 <== Dramatic improvement over 6.1 MB/s The smallfile benchmark test I used is from https://github.com/distributed-system-analysis/smallfile. Thanks for any suggestions you might have! Regards, Steve |