Menu
▾
▴
Re: [MooseFS-Users] Performance suggestions for millions of small fi!!!les
|
From: Wilson, S. M <st...@pu...> - 2018-10-19 19:26:36
|
Hi Peter,
Let me get some of this information to you.
Ping results:
Client to Master Server (which is also Chunkserver #1)
25 packets transmitted, 25 received, 0% packet loss, time 24581ms
rtt min/avg/max/mdev = 0.082/0.108/0.147/0.023 ms
Client to Chunkserver #2
25 packets transmitted, 25 received, 0% packet loss, time 24567ms
rtt min/avg/max/mdev = 0.102/0.118/0.135/0.016 ms
Client to Chunkserver #3
25 packets transmitted, 25 received, 0% packet loss, time 24556ms
rtt min/avg/max/mdev = 0.104/0.126/0.153/0.014 ms
Client to Chunkserver #4
25 packets transmitted, 25 received, 0% packet loss, time 24557ms
rtt min/avg/max/mdev = 0.067/0.073/0.081/0.012 ms
Three tests run at the same time from different clients (using the results from client #1):
Create IOPS: 174
Create MB/s: 0.68
Read IOPS: 3166
Read MB/s: 12.37
Append IOPS: 1221
Append MB/s: 4.77
Rename files/s: 1158
tar -xf linux-4.9-rc3.tar: 824 secs
Single test run on client #1:
Create IOPS: 281
Create MB/s: 1.1
Read IOPS: 3351
Read MB/s: 13.09
Append IOPS: 1678
Append MB/s: 6.55
Rename files/s: 1319
tar -xf linux-4.9-rc3.tar: 937 secs (not sure why this took longer... perhaps due to an increase in other activity)
These tests took place between 14:00 and 15:15 so you can see the related activity on the attached Master Charts images.
Thanks!
Steve
________________________________
From: Piotr Robert Konopelko <pio...@ge...>
Sent: Friday, October 19, 2018 12:58 PM
To: Wilson, Steven M
Cc: moo...@li...
Subject: Re: [MooseFS-Users] Performance suggestions for millions of small files
Hi Steve,
what is the latency:
Client <----> Master Server
Client <----> Chunkservers?
When we consider small files operations, latency becomes the most crucial parameter.
Could you please provide us with some ping test results?
Also, what are results (of single tests and summed up) if you run e.g. two, three, more such tests at the same time?
Can you plz paste also some Master charts from the time of tests are being ran?
Thank you,
Best regards,
Peter
Piotr Robert Konopelko | m: +48 601 476 440 | e: pio...@mo...<mailto:pio...@mo...>
Business & Technical Support Manager
MooseFS Client Support Team
WWW<http://moosefs.com/> | GitHub<https://github.com/moosefs/moosefs> | Twitter<https://twitter.com/moosefs> | Facebook<https://www.facebook.com/moosefs> | LinkedIn<https://www.linkedin.com/company/moosefs>
On 19 Oct 2018, at 6:36 PM, Marco Milano <mar...@gm...<mailto:mar...@gm...>> wrote:
Steve,
My wild guess is that somehow the master server is not very efficient
to handle that many files.
(Obviously if this is the case, it is very bad.)
I will do some tests with the 4.x series and 0.5 billion files
and let you know.(it will take me several weeks to create that test
environment.)
In the meantime, you can split the namespace into two which may help
on the same hardware. (Basically you can run as many namespaces
as you want on the same hardware setup, however this will require
a lot of work to setup and migrate, in this case, there will be
two master server processes running on your master server hardware
just at different ports)
Or, just hope that the performance of the master server is better
with version 4.x
-- Marco
On 10/19/18 11:10 AM, Wilson, Steven M wrote:
Hi Diego,
I appreciate you taking the time to run these tests on your own setup! My parameters to the smallfile benchmark were a little different (I took them from some GlusterFS benchmarking documentation):
smallfile_cli.py --top smallfile-tests --threads 4 --file-size 4
--files 10000 --response-times Y
Steve
------------------------------------------------------------------------
*From:* Remolina, Diego J <dij...@ae...<mailto:dij...@ae...>>
*Sent:* Friday, October 19, 2018 7:19 AM
*To:* MooseFS-Users
*Subject:* Re: [MooseFS-Users] Performance suggestions for millions of small files
Hi Steve,
I have by no means a similar amount of files and space, as I am just testing, but this is what I see with MooseFS 4.6.0 and goal=3 on a pretty new (in testing phase, no load) 3-way server setup:
time tar -xf linux-4.9-rc3.tar
real 4m0.332s
user 0m1.668s
sys 0m9.517s
python /tmp/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 1 --files 2048 --top /nethome/dijuremo/test
total threads = 8
total files = 15948
total IOPS = 15948
total data = 0.015 GiB
97.34% of requested files processed, minimum is 90.00
elapsed time = 11.608
files/sec = 1373.870032
IOPS = 1373.870032
MiB/sec = 1.341670
python /tmp/smallfile/smallfile_cli.py --operation read --threads 8 --file-size 1 --files 2048 --top /nethome/dijuremo/test
total threads = 8
total files = 16384
total IOPS = 16384
total data = 0.016 GiB
100.00% of requested files processed, minimum is 90.00
elapsed time = 2.553
files/sec = 6416.909838
IOPS = 6416.909838
MiB/sec = 6.266514
python /tmp/smallfile/smallfile_cli.py --operation append --threads 8 --file-size 1 --files 2048 --top /nethome/dijuremo/test
total threads = 8
total files = 15348
total IOPS = 15348
total data = 0.015 GiB
93.68% of requested files processed, minimum is 90.00
elapsed time = 8.018
files/sec = 1914.272783
IOPS = 1914.272783
MiB/sec = 1.869407
I will be happy to adjust the smallfile test settings if any of my tests are useful to you and re-run them for comparison.
Diego
------------------------------------------------------------------------
*From:* Wilson, Steven M <st...@pu...<mailto:st...@pu...>>
*Sent:* Thursday, October 18, 2018 4:47:14 PM
*To:* MooseFS-Users
*Subject:* [MooseFS-Users] Performance suggestions for millions of small files
Hi,
We have ten different MooseFS installations in our research group and one, in particular, is struggling with poor I/O performance. This installation currently has 315 million files occupying 170TB of disk space (goal = 2). If anyone else has a similar installation, I would like to hear what you have done to maintain performance at a reasonable level.
Here are some metrics to give a basic idea of the performance characteristics. I'll include in parentheses the range of measurements from other MFS installations with far fewer files for comparison.
* tar xf linux-4.9-rc3.tar: 1185 secs (220 - 296 secs)
* smallfile test, create MB/s: 0.8 (2.3 - 4.8) <== Ouch!
* smallfile test, read MB/s: 10.7 (12.8 - 15.4)
* smallfile test, append MB/s: 6.1 (3.0 - 7.7)
It looks file creation is where I'm losing most of my performance compared to the other installations. My master server has a Xeon E5-1630v3 3.7GHz CPU with 256GB of DDR4 2133MHz memory.
I tried several mfsmount options but the only one that showed any significant improvement was the mfsfsyncmintime option ("mfsfsyncmintime=5"). As to be expected, the improvement gained was during the write/append operation. Here are the results using the same tests as above:
* tar xf linux-4.9-rc3.tar: 683 secs
* smallfile test, create MB/s: 1.2
* smallfile test, read MB/s: 11.7
* smallfile test, append MB/s: 11.4 <== Dramatic improvement over 6.1 MB/s
The smallfile benchmark test I used is from https://github.com/distributed-system-analysis/smallfile.
Thanks for any suggestions you might have!
Regards,
Steve
_________________________________________
moosefs-users mailing list
moo...@li...<mailto:moo...@li...>
https://lists.sourceforge.net/lists/listinfo/moosefs-users
_________________________________________
moosefs-users mailing list
moo...@li...<mailto:moo...@li...>
https://lists.sourceforge.net/lists/listinfo/moosefs-users
|
