Re: [MooseFS-Users] performance inquiry

[Aleksander W. <ale...@mo...>]

Hi.
Thank you for this information.

Can you do one more simple test.
What network bandwidth can you achieve between two FreeBSD machines?

I mean, something like:
FreeBSD 1 /dev/zero > 10Gb NIC > FreeBSD 2 /dev/null
(simple nc and dd tool will tell you a lot.)

We know that FUSE on FreeBSD systems had some problems but we need to
take close look to this issue.
We will try to repeat this scenario in our test environment and return
to you after 08.09.2015, because we are in progress of different tests
till this day.

I would like to add one more aspect.
Mfsmaster application is single-thread so we compared your cpu with our.
This are the results:
*(source: cpubenchmark.net)*

CPU Xeon E5-1620 v2 @ 3.70GHz
Avarage points: 9508
*Single Thread points**: 1920*

CPU Intel Atom C2758 @ 2.40GHz
Avarage points: 3620
*Single Thread points: 520*

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 27.08.2015 23:19, Joseph Love wrote:
> Hi,
>
> So, I did the following:
> - Moved the mfsmaster to one of the chunk servers (which has Xeons in it).
> - Installed ubuntu 14.04 on the 4th xeon server, which I was trying as
> a client in the latest tests.
>
> A single ‘dd’ instance on ubuntu 14.04 was able to write to the
> cluster at just under 500MB/s (494MB/s, 20GB from /dev/zero).
> I also ran (though it failed to generate the statistics) tiobench,
> which ran substantially faster, before generating a divide by zero error.
>
> I noticed something interesting and different when doing this.
>
> I’ve been running nload on the chunk servers, showing the bandwidth
> used by the chunk servers on my screen.  It’s a little interesting how
> this varies between when the freebsd client is writing and when the
> ubuntu client is writing (even from just dd from /dev/zero).
>
> The freebsd client neither achieves the speeds the ubuntu client does,
> nor the consistency of sending data to all the chunk servers that the
> ubuntu client achieves.
>
> http://www.getsomewhere.net/nload_freebsd_client.png
> http://www.getsomewhere.net/nload_ubuntu_client.png
>
> -Joe
>
>
>> On Aug 26, 2015, at 8:56 AM, Aleksander Wieliczko
>> <ale...@mo...
>> <mailto:ale...@mo...>> wrote:
>>
>> Thank you for this information.
>>
>> We have two ideas to test in your environment:
>> 1. Can you test mfsclient on Linux OS(like Ubuntu 14/Debian 8) with
>> FUSE >= 2.9.3 ?
>> 2. Can you switch from Atom to Xeon CPU for master during your tests?
>>
>> We are waiting for your feedback.
>>
>> Best regards
>> Aleksander Wieliczko
>> Technical Support Engineer
>> MooseFS.com <x-msg://49/moosefs.com>
>>
>> On 26.08.2015 15:23, Joseph Love wrote:
>>> Hi,
>>>
>>> Sure.
>>> All parts are running 3.0.39, on FreeBSD 10.2.
>>>
>>> Chunk servers are dual xeon X5560s, 24gb memory.
>>> Master is an atom C2758, 4gb memory
>>> Clients vary between an atom C2758 4gb memory (not the master), and
>>> a dual xeon L5650, 24gb memory.
>>>
>>> I’ve tried two different disk setups with the chunk servers:
>>> - with a pair of 1tb WD RE3s, mirrored; and
>>> - with a single Intel DC S3500 SSD.
>>>
>>> Goal is set to 1.
>>> NICs are Intel X520-DA2 (10gbe) with direct-attach SFP+ cables to a
>>> Broadcom 8000b switch.
>>>
>>> Latency from client 1 (atom C2758) - 30 packets:
>>> to Master: round-trip min/avg/max/stddev = 0.069/0.086/0.096/0.007 ms
>>> to chunk1: round-trip min/avg/max/stddev = 0.059/0.125/0.612/0.159 ms
>>> to chunk2: round-trip min/avg/max/stddev = 0.049/0.087/0.611/0.098 ms
>>> to chunk3: round-trip min/avg/max/stddev = 0.054/0.071/0.100/0.010 ms
>>>
>>> Latency from client 2 (xeon L5650) - 30 packets:
>>> to Master: round-trip min/avg/max/stddev = 0.045/0.056/0.073/0.006 ms
>>> to chunk1: round-trip min/avg/max/stddev = 0.029/0.036/0.043/0.003 ms
>>> to chunk2: round-trip min/avg/max/stddev = 0.033/0.037/0.044/0.003 ms
>>> to chunk3: 
>>>
>>> There’s no traffic shaping/QOS on this LAN, and the master & chunk
>>> servers are all on a network dedicated just to them.
>>>
>>> I’ve also tried this from client2, partially out of curiosity:
>>> > dd if=/dev/zero of=test.zero.dd bs=1m count=10000 & && dd
>>> if=/dev/zero of=test.zero-2.dd bs=1m count=10000 & && dd
>>> if=/dev/zero of=test.zero-3.dd bs=1m count=10000 &
>>>
>>> 10000+0 records in
>>> 10000+0 records out
>>> 10485760000 bytes transferred in 101.117837 secs (103698421 bytes/sec)
>>> 10000+0 records in
>>> 10000+0 records out
>>> 10485760000 bytes transferred in 101.644066 secs (103161556 bytes/sec)
>>> 10000+0 records in
>>> 10000+0 records out
>>> 10485760000 bytes transferred in 101.900907 secs (102901537 bytes/sec)
>>>
>>> Running 3 instances, I was able to write at 100MB/s per instance of
>>> dd.  Which suggests being able to write at at least 300MB/s from a
>>> single client (just not in a single process/thread).
>>>
>>> I tried the same thing with 7 instances.  I think it actually hit
>>> the limits on the chunk server’s disk speeds (approximately 220MB/s
>>> per ssd, iirc).  7 instances of dd gave me these results:
>>> 10485760000 bytes transferred in 124.690526 secs (84094280 bytes/sec)
>>> 10485760000 bytes transferred in 124.719954 secs (84074438 bytes/sec)
>>> 10485760000 bytes transferred in 126.579289 secs (82839460 bytes/sec)
>>> 10485760000 bytes transferred in 124.954106 secs (83916890 bytes/sec)
>>> 10485760000 bytes transferred in 126.181140 secs (83100850 bytes/sec)
>>> 10485760000 bytes transferred in 126.294929 secs (83025978 bytes/sec)
>>> 10485760000 bytes transferred in 103.810845 secs (101008329 bytes/sec)
>>> Add that all up, and it’s about 600MB/s, which is about
>>> 200MB/s/chunkserver - pretty close to the theoretical max for the SSDs.
>>>
>>> So, I guess a single client can reach the speeds, just not in a
>>> single process/thread.
>>>
>>> -Joe
>>>
>>>
>>>> On Aug 26, 2015, at 1:17 AM, Aleksander Wieliczko
>>>> <ale...@mo...
>>>> <mailto:ale...@mo...>> wrote:
>>>>
>>>> Hi.
>>>>
>>>> Can we get some more details about your configuration?
>>>>
>>>> - MooseFS master version?
>>>> - MooseFS chunkserver version?
>>>> - MooseFS client version?
>>>> - Kernel version ?
>>>> - CPU speed ?
>>>> - RAM size ?
>>>> - Number of disks per chunkserver?
>>>> - What GOAL you set for test folder?
>>>> - NIC interface type - Coper or Fiber?
>>>> - Network latency from client to master and chunkservers(ping
>>>> mfsmaster)?
>>>> - Do you have some traffic shaping/QOS in you LAN?
>>>>
>>>> Best regards
>>>> Aleksander Wieliczko
>>>> Technical Support Engineer
>>>> MooseFS.com <x-msg://27/moosefs.com>
>>>>
>>>>
>>>> On 25.08.2015 18:29, Joseph Love wrote:
>>>>> They’re all on 10gbe.  I did turn on Jumbo frames during my testing, it didn’t seem to make a really big difference.
>>>>>
>>>>> I just tried with SSDs in the chunk servers (Intel DC S3500 200GB), and still seeing about the same performance characteristic.
>>>>>
>>>>> I know in the middle of some other synthetic tests that I can break 200MB/s (sequential) read, 150MB/s write, but that’s a multithreaded test application.
>>>>> Actually, now that I say that, I suppose it might be a single-thread performance characteristic with FUSE on FreeBSD.
>>>>>
>>>>> Anyone have statistic from a Linux system that shows > 100MB/s per thread on a client?
>>>>>
>>>>> -Joe
>>>>>
>>>>>> On Aug 25, 2015, at 11:12 AM, Ricardo J. Barberis <ric...@do...> wrote:
>>>>>>
>>>>>> El Martes 25/08/2015, Joseph Love escribió:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I’ve been doing some tests to get an idea as to what sort of speeds I can
>>>>>>> maybe expect from moosefs, and ran into something unexpected.  From
>>>>>>> multiple clients, I can sustain 80-100MB/s per client (only tested up to 3
>>>>>>> clients) to my 3-node cluster (3 chunk servers).  From a single client
>>>>>>> (while everything else is idle) I get the same result.  It occurred to me
>>>>>>> that the write speed to a disk in each chunk server is roughly 100MB/s, and
>>>>>>> I was curious if this seems to be the likely culprit for performance
>>>>>>> limitations for a single stream from a single client.
>>>>>>>
>>>>>>> I’m about to try it again with SSDs, but I have a bit of time before that’s
>>>>>>> ready, and I figured I’d try to pose the question early.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> -Joe
>>>>>> How about network? If your clients are connected to 1 Gbps, 100 MB/s is nearly 
>>>>>> saturating the network.
>>>>>>
>>>>>> Also, using Jumbo frames might give you a few extra MB/s.
>>>>>>
>>>>>> Regards,
>>>>>> -- 
>>>>>> Ricardo J. Barberis
>>>>>> Senior SysAdmin / IT Architect
>>>>>> DonWeb
>>>>>> La Actitud Es Todo
>>>>>> www.DonWeb.com
>>>>>>  _____
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> _________________________________________
>>>>>> moosefs-users mailing list
>>>>>> moo...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>> ------------------------------------------------------------------------------
>>>>> _________________________________________
>>>>> moosefs-users mailing list
>>>>> moo...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _________________________________________
>>>> moosefs-users mailing list
>>>> moo...@li...
>>>> <mailto:moo...@li...>
>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>
>>
>
>
>
> ------------------------------------------------------------------------------
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users