[MooseFS-Users] New to MooseFS, question about performance of bonnie++ test

[Remolina, D. J <dij...@ae...>]

Hi everyone,


I have been playing with MooseFS 3.x for the past week and a half. It has been pretty nice to work with it and looking forward to the 4.x release with HA.


As part of the testing I am doing (comparing drbd, glusterfs and MooseFS), I have discovered one issue when running bonnie++ tests. The rewrite portion of the bonnie++ test, just takes forever. I am not sure that I will hit this particular use in real life since this will be mostly a file server, likely exporting files via native MooseFS to other *nix machines and likely using a samba server (which mounts moosefs as a client) to export to windows machines.


>From the bonnie++ docs: https://www.coker.com.au/bonnie++/readme.html


3. Rewrite. Each BUFSIZ of the file is read with read(2), dirtied, and rewritten with write(2), requiring an lseek(2). Since no space allocation is done, and the I/O is well-localized, this should test the effectiveness of the filesystem cache and the speed of data transfer.


MooseFS Setup:


3 Similar DELL R730xv on 3 separate datacenters located in separate buildings, but on same campus.

Servers have Dual Xeon 2680v4 (1 and 2) and 2640v4 (server 3) and 256GB RAM each.

Servers have 24 10K RPM 2.5" SAS drives. These were formatted as individual ZFS pools and a file system created on each drive:


zfs create -o logbias=trohougput -o atime=off -o xattr=sa mfspoolXX/mfsfs (XX: 00 to 23)


compression=lz4 was set on the pool, so it is inherited already by any file system created under each pool.
All servers connected to network via 10Gbps (dual links LACP). The datacenters are interconnected at least at 40Gbps to the campus network


Server 3 is the mfsmaster, chunkserver and runs cgi server (datacenter with the best redundancies and lower risk of failure). Servers 2 and 1 are meta loggers and chunk servers.


My test client is an Ubuntu machine running 16.04 and it is on a separate subnet which has to traverse a subnet firewall and each server's host based firewall. Servers have opened 9419-9422 to the subnet where my client lives. Client machine is connected via 1 gbit to the network and the building switches have at least 20gbps connections to the main campus network.


Highest ping RTT from client to servers is:


Server1 -> rtt min/avg/max/mdev = 1.220/1.318/1.409/0.081 ms

Server2 -> rtt min/avg/max/mdev = 1.214/1.357/1.420/0.064 ms

Server3 -> rtt min/avg/max/mdev = 1.399/1.458/1.606/0.072 ms

I have configured MooseFS with goal=3.


My bonnie++ test looks like (just one result, will be happy to post the other 2 on request):

Version      1.97   ------Sequential Output------ --Sequential Input- --Random-
                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ae410001_ubuntu 32G    48  40 113920  19  1346   0  2665  99 112490   3  2531  28
Latency               197ms   29525us    1606ms    8509us    1550ms     108ms
                   ------Sequential Create------ --------Random Create--------
                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
ae410001_ubuntu_ 16   954   5 +++++ +++  1886   6   934   5  1933   4  1000   4
Latency             17502us    1771us   40616us   24583us   10711us    8372us

Running the 3 tests using this command:
time bonnie++ -d /mnt/mfsserver/home/dijuremo -r 16G -s 32G -m
ae410001_ubuntu_cliene_vlan -q -x3 -u dijuremo

Took:
real    1283m6.528s
user    0m39.649s
sys     9m29.547s


Any other real time use scenarios, copying files with cp, scp and rsync and duplicating files, have been just fine. I pretty much hit the gigabit network bandwidth limitation from the client computer.

I am wondering why the rewrite tests on bonnie++ in my setup are so bad (two orders of magnitude lower). Is there something that I may be overlooking in my configuration?

I already tried one run with another server connected to the same subnet as the MooseFS servers to skip the subnet firewalls, but results are pretty similar. That other server has a gigabit network connection and the read and writes are around the maximum (~112MB/s) allowed for a gigabit network card, but the rewrite piece of the test is just around the 1MB/s mark.

Sorry for  the long e-mail, but I hope I have provided enough information to describe the setup.

Thanks,


Diego