Re: [Moosefs-users] architecture question

[Robert S. <rsa...@ne...>]

Some of this the developers will have to answer.

What we have seen is that there does seem to be some limitation in 
MooseFS on the maximum number of operations per second on a specific 
mount. Network latencies seems to have a large influence on overall IOPS.

Given the right conditions we could max out the network connections with 
no problems. We did not have 10 GB/s to play with but multiple bound 1 
GB/s network cards. Depending on the size of the average transfer you 
are doing the bus seems to be easy to saturate. In our use case the IOPS 
of the drives and the network latency limitations seems to be saturated 
before the bus are saturated.

Will more cores be useful for MooseFS?: My guess is that depends on what 
else you are doing on the same machine. If you only use it as a 
chunkserver then I think the bus and the disk IOPS limits will be 
reached way before you will be CPU constrained. If you also mount the 
filesystem and export it using something like Samba or Apache then the 
nature of Samba and Apache may make you CPU constrained.

For example, a very non-scientific measurement is that I have seen 
around 1.75 million operations per hour on our system. This is around 
486 per second. I need to stress that this is from a random sampling of 
looking at the CGI interface and may not be the maximum possible on our 
network. This is with the load distributed between 3 mounts from the 
same mfsmaster. I suspect we will be able to raise it a little by 
distributing the load between 4 mounts. But it does not scale linearly. 
I also have to stress that this is not IOPS. This is how MooseFS define 
operations. If I look at the CPU usage of mfsmount, mfschunkserver and 
mfsmaster then it is negligible. RAM usage is significant and disk 
utilization is significant. I/O wait time is significant.

What would be interesting is to see what would have the biggest effect 
on overall IOPS: 10 GB/s ethernet or Infiniband with its lower 
throughput but also much lower latencies. A answer I did get from the 
developers implied that upgrading to 10 GB/s may not have much of an 
effect on the overall IOPS although it probably will have an effect on 
total throughput.

Saturating a 6 Gbps SAS2 link can be done with less than 12 SAS drives. 
88 MB/s x 12 = 1.056 GB/s or around 8.4 Gb/s. That is if we assume that 
the average 10k RPM SAS drive has an average transfer rate of 88 MB/s. 
This is sort of a worst case scenario but it proves my point: A single 
SAS link is relatively trivial to saturate given a modest number of 
spindles.

Robert

On 7/23/11 9:16 PM, Elliot Finley wrote:
> On Thu, Jul 21, 2011 at 8:12 PM, Robert Sandilands
> <rsa...@ne...>  wrote:
>> The only time that many spindles will be remotely useful on 1 controller
>> is if you are doing a large amount of small random read/write accesses.
>> For anything else you are more likely to saturate the bus connecting the
>> machine to the controller/expander.
> I appreciate you taking the time to answer.  What you're saying here
> is a common sense answer, but it doesn't really answer my specific
> question.  Most likely because I didn't state it clear enough.  So
> here is my question in a (hopefully) more clear fashion:
>
> Is MooseFS sufficiently multithreaded/multi-process to be able to make
> use of lots of spindles to handle lots of IOPS...  To the point of
> saturating the 6Gbps SAS2 link?  This obviously implies a 10G ethernet
> link to the chunkserver.
>
> Also asked another way:  does having more cores == more performance?
> i.e. Can MooseFS take advantage of multiple cores?  to what extent?
>
> I ask to what extent because a multithreaded/multi-process daemon can
> get hung up with lock contention.
>
> TIA
> Elliot