Menu
▾
▴
Re: [MooseFS-Users] Performance suggestions for millions of small files
|
From: Piotr R. K. <pio...@ge...> - 2018-10-19 17:14:33
|
Hi Steve, what is the latency: Client <----> Master Server Client <----> Chunkservers? When we consider small files operations, latency becomes the most crucial parameter. Could you please provide us with some ping test results? Also, what are results (of single tests and summed up) if you run e.g. two, three, more such tests at the same time? Can you plz paste also some Master charts from the time of tests are being ran? Thank you, Best regards, Peter Piotr Robert Konopelko | m: +48 601 476 440 | e: pio...@mo... <mailto:pio...@mo...> Business & Technical Support Manager MooseFS Client Support Team WWW <http://moosefs.com/> | GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs> > On 19 Oct 2018, at 6:36 PM, Marco Milano <mar...@gm...> wrote: > > Steve, > > My wild guess is that somehow the master server is not very efficient > to handle that many files. > (Obviously if this is the case, it is very bad.) > > I will do some tests with the 4.x series and 0.5 billion files > and let you know.(it will take me several weeks to create that test > environment.) > > In the meantime, you can split the namespace into two which may help > on the same hardware. (Basically you can run as many namespaces > as you want on the same hardware setup, however this will require > a lot of work to setup and migrate, in this case, there will be > two master server processes running on your master server hardware > just at different ports) > > Or, just hope that the performance of the master server is better > with version 4.x > > -- Marco > > On 10/19/18 11:10 AM, Wilson, Steven M wrote: >> Hi Diego, >> I appreciate you taking the time to run these tests on your own setup! My parameters to the smallfile benchmark were a little different (I took them from some GlusterFS benchmarking documentation): >> smallfile_cli.py --top smallfile-tests --threads 4 --file-size 4 >> --files 10000 --response-times Y >> >> Steve >> ------------------------------------------------------------------------ >> *From:* Remolina, Diego J <dij...@ae...> >> *Sent:* Friday, October 19, 2018 7:19 AM >> *To:* MooseFS-Users >> *Subject:* Re: [MooseFS-Users] Performance suggestions for millions of small files >> Hi Steve, >> I have by no means a similar amount of files and space, as I am just testing, but this is what I see with MooseFS 4.6.0 and goal=3 on a pretty new (in testing phase, no load) 3-way server setup: >> time tar -xf linux-4.9-rc3.tar >> real 4m0.332s >> user 0m1.668s >> sys 0m9.517s >> python /tmp/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 1 --files 2048 --top /nethome/dijuremo/test >> total threads = 8 >> total files = 15948 >> total IOPS = 15948 >> total data = 0.015 GiB >> 97.34% of requested files processed, minimum is 90.00 >> elapsed time = 11.608 >> files/sec = 1373.870032 >> IOPS = 1373.870032 >> MiB/sec = 1.341670 >> python /tmp/smallfile/smallfile_cli.py --operation read --threads 8 --file-size 1 --files 2048 --top /nethome/dijuremo/test >> total threads = 8 >> total files = 16384 >> total IOPS = 16384 >> total data = 0.016 GiB >> 100.00% of requested files processed, minimum is 90.00 >> elapsed time = 2.553 >> files/sec = 6416.909838 >> IOPS = 6416.909838 >> MiB/sec = 6.266514 >> python /tmp/smallfile/smallfile_cli.py --operation append --threads 8 --file-size 1 --files 2048 --top /nethome/dijuremo/test >> total threads = 8 >> total files = 15348 >> total IOPS = 15348 >> total data = 0.015 GiB >> 93.68% of requested files processed, minimum is 90.00 >> elapsed time = 8.018 >> files/sec = 1914.272783 >> IOPS = 1914.272783 >> MiB/sec = 1.869407 >> I will be happy to adjust the smallfile test settings if any of my tests are useful to you and re-run them for comparison. >> Diego >> ------------------------------------------------------------------------ >> *From:* Wilson, Steven M <st...@pu...> >> *Sent:* Thursday, October 18, 2018 4:47:14 PM >> *To:* MooseFS-Users >> *Subject:* [MooseFS-Users] Performance suggestions for millions of small files >> Hi, >> We have ten different MooseFS installations in our research group and one, in particular, is struggling with poor I/O performance. This installation currently has 315 million files occupying 170TB of disk space (goal = 2). If anyone else has a similar installation, I would like to hear what you have done to maintain performance at a reasonable level. >> Here are some metrics to give a basic idea of the performance characteristics. I'll include in parentheses the range of measurements from other MFS installations with far fewer files for comparison. >> * tar xf linux-4.9-rc3.tar: 1185 secs (220 - 296 secs) >> * smallfile test, create MB/s: 0.8 (2.3 - 4.8) <== Ouch! >> * smallfile test, read MB/s: 10.7 (12.8 - 15.4) >> * smallfile test, append MB/s: 6.1 (3.0 - 7.7) >> It looks file creation is where I'm losing most of my performance compared to the other installations. My master server has a Xeon E5-1630v3 3.7GHz CPU with 256GB of DDR4 2133MHz memory. >> I tried several mfsmount options but the only one that showed any significant improvement was the mfsfsyncmintime option ("mfsfsyncmintime=5"). As to be expected, the improvement gained was during the write/append operation. Here are the results using the same tests as above: >> * tar xf linux-4.9-rc3.tar: 683 secs >> * smallfile test, create MB/s: 1.2 >> * smallfile test, read MB/s: 11.7 >> * smallfile test, append MB/s: 11.4 <== Dramatic improvement over 6.1 MB/s >> The smallfile benchmark test I used is from https://github.com/distributed-system-analysis/smallfile. >> Thanks for any suggestions you might have! >> Regards, >> Steve >> _________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
