moosefs-users

[Moosefs-users] architecture question

[Elliot F. <efi...@gm...>]

Does putting more spindles on a chunkserver increase performance?  If
so, to what point?

I'm specifically thinking of a SAS2 controller/SAS2 expander with 127
drives attached.  Does the chunkserver daemon have the capability to
efficiently use that many spindles?

TIA
Elliot

Re: [Moosefs-users] architecture question

[Robert S. <rsa...@ne...>]

The only time that many spindles will be remotely useful on 1 controller 
is if you are doing a large amount of small random read/write accesses. 
For anything else you are more likely to saturate the bus connecting the 
machine to the controller/expander.

Performance, as I see it, is divided into at least 2 types of performance:

1. Transfer speed
2. IOPS (I/O Operations per second)

For completely serial jobs you can get high transfer speeds with very 
few spindles. The more clients or the more random the load generated by 
your clients the more spindles becomes useful. But there is a tradeoff. 
If you are reading large numbers of large files then transfer speed may 
be your limiting factor. If you read large numbers of small files then 
IOPS may be.

We have 68 spindles distributed between 2 chunkservers and we are maxed 
out on IOPS on the one chunkserver. On the other chunkserver we are 
around 60%. But our load pattern is random reads of millions of small 
files ( < 600 kB ). The reason for the imbalance is historical and will 
go away in the next few weeks.

When you start dealing with large numbers of disks on a single 
controller/device you will be surprised what bottlenecks exist. The RAID 
controller rapidly becomes your bottleneck when it comes to transfer 
speed. In my experience it seems to be better to have either multiple 
controllers, multiple servers or multiple channels if you are talking to 
so many spindles. In the case of multiple controllers and channels you 
may still end up saturating the PCI bus or your network interfaces.

At some stage a storage company tried to convince us to buy an enclosure 
with 12 x SSD drives. I asked them to do some performance tests and the 
numbers were quite interesting. The SSD's could quickly saturate the 
bus. In transfer speed 12 x SSD was not any faster than 12 x SAS drives. 
The reason: the SAS bus got saturated in both cases. In IOPS the SSD 
drives were better until it got to a point where the bus were saturated 
and then adding more SSD drives did not provide any advantage.

Robert

On 7/21/11 2:46 PM, Elliot Finley wrote:
> Does putting more spindles on a chunkserver increase performance?  If
> so, to what point?
>
> I'm specifically thinking of a SAS2 controller/SAS2 expander with 127
> drives attached.  Does the chunkserver daemon have the capability to
> efficiently use that many spindles?
>
> TIA
> Elliot
>
> ------------------------------------------------------------------------------
> 5 Ways to Improve&  Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] architecture question

[Elliot F. <efi...@gm...>]

On Thu, Jul 21, 2011 at 8:12 PM, Robert Sandilands
<rsa...@ne...> wrote:
> The only time that many spindles will be remotely useful on 1 controller
> is if you are doing a large amount of small random read/write accesses.
> For anything else you are more likely to saturate the bus connecting the
> machine to the controller/expander.

I appreciate you taking the time to answer.  What you're saying here
is a common sense answer, but it doesn't really answer my specific
question.  Most likely because I didn't state it clear enough.  So
here is my question in a (hopefully) more clear fashion:

Is MooseFS sufficiently multithreaded/multi-process to be able to make
use of lots of spindles to handle lots of IOPS...  To the point of
saturating the 6Gbps SAS2 link?  This obviously implies a 10G ethernet
link to the chunkserver.

Also asked another way:  does having more cores == more performance?
i.e. Can MooseFS take advantage of multiple cores?  to what extent?

I ask to what extent because a multithreaded/multi-process daemon can
get hung up with lock contention.

TIA
Elliot

Re: [Moosefs-users] architecture question

[Elliot F. <efi...@gm...>]

I forgot to add that my question implies lots of very small files.
Would I be able to scale a chunkserver by using sufficent
cores/spindles to the point of saturating the 6Gbps SAS2 link with a
workload of lots of very small files?

On Sat, Jul 23, 2011 at 7:16 PM, Elliot Finley <efi...@gm...> wrote:
> On Thu, Jul 21, 2011 at 8:12 PM, Robert Sandilands
> <rsa...@ne...> wrote:
>> The only time that many spindles will be remotely useful on 1 controller
>> is if you are doing a large amount of small random read/write accesses.
>> For anything else you are more likely to saturate the bus connecting the
>> machine to the controller/expander.
>
> I appreciate you taking the time to answer.  What you're saying here
> is a common sense answer, but it doesn't really answer my specific
> question.  Most likely because I didn't state it clear enough.  So
> here is my question in a (hopefully) more clear fashion:
>
> Is MooseFS sufficiently multithreaded/multi-process to be able to make
> use of lots of spindles to handle lots of IOPS...  To the point of
> saturating the 6Gbps SAS2 link?  This obviously implies a 10G ethernet
> link to the chunkserver.
>
> Also asked another way:  does having more cores == more performance?
> i.e. Can MooseFS take advantage of multiple cores?  to what extent?
>
> I ask to what extent because a multithreaded/multi-process daemon can
> get hung up with lock contention.
>
> TIA
> Elliot
>

Re: [Moosefs-users] architecture question

[Robert S. <rsa...@ne...>]

Some of this the developers will have to answer.

What we have seen is that there does seem to be some limitation in 
MooseFS on the maximum number of operations per second on a specific 
mount. Network latencies seems to have a large influence on overall IOPS.

Given the right conditions we could max out the network connections with 
no problems. We did not have 10 GB/s to play with but multiple bound 1 
GB/s network cards. Depending on the size of the average transfer you 
are doing the bus seems to be easy to saturate. In our use case the IOPS 
of the drives and the network latency limitations seems to be saturated 
before the bus are saturated.

Will more cores be useful for MooseFS?: My guess is that depends on what 
else you are doing on the same machine. If you only use it as a 
chunkserver then I think the bus and the disk IOPS limits will be 
reached way before you will be CPU constrained. If you also mount the 
filesystem and export it using something like Samba or Apache then the 
nature of Samba and Apache may make you CPU constrained.

For example, a very non-scientific measurement is that I have seen 
around 1.75 million operations per hour on our system. This is around 
486 per second. I need to stress that this is from a random sampling of 
looking at the CGI interface and may not be the maximum possible on our 
network. This is with the load distributed between 3 mounts from the 
same mfsmaster. I suspect we will be able to raise it a little by 
distributing the load between 4 mounts. But it does not scale linearly. 
I also have to stress that this is not IOPS. This is how MooseFS define 
operations. If I look at the CPU usage of mfsmount, mfschunkserver and 
mfsmaster then it is negligible. RAM usage is significant and disk 
utilization is significant. I/O wait time is significant.

What would be interesting is to see what would have the biggest effect 
on overall IOPS: 10 GB/s ethernet or Infiniband with its lower 
throughput but also much lower latencies. A answer I did get from the 
developers implied that upgrading to 10 GB/s may not have much of an 
effect on the overall IOPS although it probably will have an effect on 
total throughput.

Saturating a 6 Gbps SAS2 link can be done with less than 12 SAS drives. 
88 MB/s x 12 = 1.056 GB/s or around 8.4 Gb/s. That is if we assume that 
the average 10k RPM SAS drive has an average transfer rate of 88 MB/s. 
This is sort of a worst case scenario but it proves my point: A single 
SAS link is relatively trivial to saturate given a modest number of 
spindles.

Robert

On 7/23/11 9:16 PM, Elliot Finley wrote:
> On Thu, Jul 21, 2011 at 8:12 PM, Robert Sandilands
> <rsa...@ne...>  wrote:
>> The only time that many spindles will be remotely useful on 1 controller
>> is if you are doing a large amount of small random read/write accesses.
>> For anything else you are more likely to saturate the bus connecting the
>> machine to the controller/expander.
> I appreciate you taking the time to answer.  What you're saying here
> is a common sense answer, but it doesn't really answer my specific
> question.  Most likely because I didn't state it clear enough.  So
> here is my question in a (hopefully) more clear fashion:
>
> Is MooseFS sufficiently multithreaded/multi-process to be able to make
> use of lots of spindles to handle lots of IOPS...  To the point of
> saturating the 6Gbps SAS2 link?  This obviously implies a 10G ethernet
> link to the chunkserver.
>
> Also asked another way:  does having more cores == more performance?
> i.e. Can MooseFS take advantage of multiple cores?  to what extent?
>
> I ask to what extent because a multithreaded/multi-process daemon can
> get hung up with lock contention.
>
> TIA
> Elliot